[ome-devel] CellProfiler on the cluster crashes OMERO

[Chris Allan]

Hello, Frederick:

There are a number of issues with the CellProfiler-OMERO integration that will be felt especially hard on any significantly sized cluster configuration. At Glencoe Software we have been working on some solutions and have be liaising with Anne Carpenter's team here:

	https://github.com/CellProfiler/CellProfiler/issues/1772

We are aiming to release at least one potential solution to the community before the end of the year. Testing such solutions thoroughly is, as you can imagine, quite time consuming.

As things stand I think it's fair to say that (1) if you utilise a large number of workers or are processing a large dataset CellProfiler-OMERO usage will severely impact your OMERO server performance-- it may even exhaust all resources you currently have allocated to OMERO; and (2) that even if you utilise some of our forthcoming changes you will still have to be very careful about resource allocation on your server. CellProfiler-OMERO in a cluster configuration can easily request data at a rate in excess of 1gbps for short periods of time.

I realise none of the above is particularly helpful to you right this minute but I hope that it at least helps shed some light on why you're seeing what you're seeing. At the moment my only concrete suggestion to you is to keep your CellProfiler executions short-lived and your parallelisation limited.

-Chris

On 13 Dec 2016, at 10:49, Frederik Grüll <frederik.gruell at unibas.ch> wrote:

> Dear all,
> 
> I am using CellProfiler on our cluster to process plates for screening.
> The images are fetched from OMERO with the CellProfiler-OMERO
> integration. A typical job consists of a command like this:
> 
> cellprofiler -b -p Entry-pipeline_omero.cpproj -c -r -o $OUT_DIR -t
> $TMPDIR -f $FIRST_IMAGE_SET -l $LAST_IMAGE_SET --data-file
> plate_303_iids.csv -d $DONE_FILE --omero-credentials
> host=omero.biozentrum.unibas.ch,port=4064,session-id=33c6118d-f8b2-4ac2-adb2-12d48ae37a2f
> 
> When I run about 20 jobs in parallel, performance looks good at the
> beginning, only limited by the performance of CellProfiler and not by
> the I/O with OMERO. The plate I am processing has 2400 sites with three
> channels and the OMERO IDs are in the CSV file plate_303_iids.csv that I
> generated before. A job processes 50 image sets, selected with
> $FIRST_IMAGE_SET and $LAST_IMAGE_SET. The results of the pipeline are
> correct.
> 
> However, after about 4/5 of the images have been processed, OMERO
> becomes very slow. The load on the OMERO server reaches 10, with the
> Java process for Blitz consuming 10 cores. Eventually, my CellProfiler
> jobs will loose connection ("JavaException:
> Ice.ConnectionLostException"), and OMERO recovers in a few cases or
> otherwise the CPU load falls back to normal, but OMERO needs to be
> restarted anyway.
> 
> If I run more than 20 jobs in parallel, I would occasional get an error
> message "ome.conditions.OverUsageException: servantsPerSession reached
> for 05dbc314-3030-40af-8e72-68b3688e8c94: 10000" after CellProfiler
> processed only 1665 single-channel images, implying 6 servants per image
> per channel.
> 
> I have already had a look into the logs, especially Blitz-0.log, but
> could not find a reason why OMERO would become so slow after a while.
> Jstat indicates that all time is spend on garbage collection. Our OMERO
> server has 250GB of RAM with omero.jvmcfg.percent.blitz=40.
> 
> Where else could I look into to find the cause and prevent the
> degradation in performance? I use OMERO.server 5.2.5 with OpenJDK
> version 1.8.0_65 and CellProfiler 2.2.0 with Oracle Java 1.8.0_92.
> 
> Cheers and thank you for your time,
> Frederik
> 
> -- 
> Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
> University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
> (61) 207 2250 | frederik.gruell at unibas.ch | www.biozentrum.unibas.ch
> 
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel