[ome-devel] CellProfiler on the cluster crashes OMERO

[Frederik Grüll]

Hello Chris,

Is there maybe a workaround, for example by loading the images not
through OMERO but from the cluster file system? When I import a plate I
select a HTD file and OMERO.insight uploads all the TIFFs of the plate
along with it, with one TIFF per channel, and displays them as a
multi-channel view as expected. I have four channels and would be
interested in the filesystem location of the four TIFF files that back
each image, and the information which TIFF belongs to which channel. I
could then use this information to generate a CSV file for CellProfiler
with paths instead of OMERO URLs.

I had experimented a bit with image.getFileset() to get the original
TIFFs. However, when I call the method, it takes a long time and then
returns the paths of _all_ 9600 images of the plate, not only the paths
of the four TIFFs I wanted. Is there a way to just get the paths to the
TIFFs of a view only, plus the information which TIFF backs which channel?

Cheers,

Frederik


On 13.12.2016 14:57, Chris Allan wrote:
> Hello, Frederick:
>
> There are a number of issues with the CellProfiler-OMERO integration that will be felt especially hard on any significantly sized cluster configuration. At Glencoe Software we have been working on some solutions and have be liaising with Anne Carpenter's team here:
>
> 	https://github.com/CellProfiler/CellProfiler/issues/1772
>
> We are aiming to release at least one potential solution to the community before the end of the year. Testing such solutions thoroughly is, as you can imagine, quite time consuming.
>
> As things stand I think it's fair to say that (1) if you utilise a large number of workers or are processing a large dataset CellProfiler-OMERO usage will severely impact your OMERO server performance-- it may even exhaust all resources you currently have allocated to OMERO; and (2) that even if you utilise some of our forthcoming changes you will still have to be very careful about resource allocation on your server. CellProfiler-OMERO in a cluster configuration can easily request data at a rate in excess of 1gbps for short periods of time.
>
> I realise none of the above is particularly helpful to you right this minute but I hope that it at least helps shed some light on why you're seeing what you're seeing. At the moment my only concrete suggestion to you is to keep your CellProfiler executions short-lived and your parallelisation limited.
>
> -Chris
>
> On 13 Dec 2016, at 10:49, Frederik Grüll <frederik.gruell at unibas.ch> wrote:
>
>> Dear all,
>>
>> I am using CellProfiler on our cluster to process plates for screening.
>> The images are fetched from OMERO with the CellProfiler-OMERO
>> integration. A typical job consists of a command like this:
>>
>> cellprofiler -b -p Entry-pipeline_omero.cpproj -c -r -o $OUT_DIR -t
>> $TMPDIR -f $FIRST_IMAGE_SET -l $LAST_IMAGE_SET --data-file
>> plate_303_iids.csv -d $DONE_FILE --omero-credentials
>> host=omero.biozentrum.unibas.ch,port=4064,session-id=33c6118d-f8b2-4ac2-adb2-12d48ae37a2f
>>
>> When I run about 20 jobs in parallel, performance looks good at the
>> beginning, only limited by the performance of CellProfiler and not by
>> the I/O with OMERO. The plate I am processing has 2400 sites with three
>> channels and the OMERO IDs are in the CSV file plate_303_iids.csv that I
>> generated before. A job processes 50 image sets, selected with
>> $FIRST_IMAGE_SET and $LAST_IMAGE_SET. The results of the pipeline are
>> correct.
>>
>> However, after about 4/5 of the images have been processed, OMERO
>> becomes very slow. The load on the OMERO server reaches 10, with the
>> Java process for Blitz consuming 10 cores. Eventually, my CellProfiler
>> jobs will loose connection ("JavaException:
>> Ice.ConnectionLostException"), and OMERO recovers in a few cases or
>> otherwise the CPU load falls back to normal, but OMERO needs to be
>> restarted anyway.
>>
>> If I run more than 20 jobs in parallel, I would occasional get an error
>> message "ome.conditions.OverUsageException: servantsPerSession reached
>> for 05dbc314-3030-40af-8e72-68b3688e8c94: 10000" after CellProfiler
>> processed only 1665 single-channel images, implying 6 servants per image
>> per channel.
>>
>> I have already had a look into the logs, especially Blitz-0.log, but
>> could not find a reason why OMERO would become so slow after a while.
>> Jstat indicates that all time is spend on garbage collection. Our OMERO
>> server has 250GB of RAM with omero.jvmcfg.percent.blitz=40.
>>
>> Where else could I look into to find the cause and prevent the
>> degradation in performance? I use OMERO.server 5.2.5 with OpenJDK
>> version 1.8.0_65 and CellProfiler 2.2.0 with Oracle Java 1.8.0_92.
>>
>> Cheers and thank you for your time,
>> Frederik
>>
>> -- 
>> Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
>> University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
>> (61) 207 2250 | frederik.gruell at unibas.ch | www.biozentrum.unibas.ch
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

-- 
Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
(61) 207 2250 | frederik.gruell at unibas.ch | www.biozentrum.unibas.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20161215/83719b17/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20161215/83719b17/attachment.asc>