[ome-devel] CellProfiler on the cluster crashes OMERO

[Frederik Grüll]

Hi Simon,

Thank you for your offer to have a look. I uploaded all the logs and the
output of the commands in a zip file "debug.zip".

I started my cluster jobs at 9:14, and OMERO was restarted at 10:49 on
Dec 14 2016. All times are CET.

Best regards

Frederik


On 13.12.2016 14:41, Simon Li wrote:
> Hi Frederik
>
> Could you give us your server configuration and diagnostics:
>
>     omero config get --hide-password
>     omero admin diagnostics
>
> It would also be helpful if we could see your logs for all OMERO
> services, not just Blitz. Would you mind uploading them to
> https://www.openmicroscopy.org/qa2/qa/upload/
> <https://www.openmicroscopy.org/qa2/qa/upload/> and giving us the
> timestamp of when the problem first arises following a restart?
>
> Best wishes
>
> Simon
>
>
> On 13 December 2016 at 10:49, Frederik Grüll
> <frederik.gruell at unibas.ch <mailto:frederik.gruell at unibas.ch>> wrote:
>
>     Dear all,
>
>     I am using CellProfiler on our cluster to process plates for
>     screening.
>     The images are fetched from OMERO with the CellProfiler-OMERO
>     integration. A typical job consists of a command like this:
>
>     cellprofiler -b -p Entry-pipeline_omero.cpproj -c -r -o $OUT_DIR -t
>     $TMPDIR -f $FIRST_IMAGE_SET -l $LAST_IMAGE_SET --data-file
>     plate_303_iids.csv -d $DONE_FILE --omero-credentials
>     host=omero.biozentrum.unibas.ch
>     <http://omero.biozentrum.unibas.ch>,port=4064,session-id=33c6118d-f8b2-4ac2-adb2-12d48ae37a2f
>
>     When I run about 20 jobs in parallel, performance looks good at the
>     beginning, only limited by the performance of CellProfiler and not by
>     the I/O with OMERO. The plate I am processing has 2400 sites with
>     three
>     channels and the OMERO IDs are in the CSV file plate_303_iids.csv
>     that I
>     generated before. A job processes 50 image sets, selected with
>     $FIRST_IMAGE_SET and $LAST_IMAGE_SET. The results of the pipeline are
>     correct.
>
>     However, after about 4/5 of the images have been processed, OMERO
>     becomes very slow. The load on the OMERO server reaches 10, with the
>     Java process for Blitz consuming 10 cores. Eventually, my CellProfiler
>     jobs will loose connection ("JavaException:
>     Ice.ConnectionLostException"), and OMERO recovers in a few cases or
>     otherwise the CPU load falls back to normal, but OMERO needs to be
>     restarted anyway.
>
>     If I run more than 20 jobs in parallel, I would occasional get an
>     error
>     message "ome.conditions.OverUsageException: servantsPerSession reached
>     for 05dbc314-3030-40af-8e72-68b3688e8c94: 10000" after CellProfiler
>     processed only 1665 single-channel images, implying 6 servants per
>     image
>     per channel.
>
>     I have already had a look into the logs, especially Blitz-0.log, but
>     could not find a reason why OMERO would become so slow after a while.
>     Jstat indicates that all time is spend on garbage collection. Our
>     OMERO
>     server has 250GB of RAM with omero.jvmcfg.percent.blitz=40.
>
>     Where else could I look into to find the cause and prevent the
>     degradation in performance? I use OMERO.server 5.2.5 with OpenJDK
>     version 1.8.0_65 and CellProfiler 2.2.0 with Oracle Java 1.8.0_92.
>
>     Cheers and thank you for your time,
>     Frederik
>
>     --
>     Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
>     University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
>     (61) 207 2250 | frederik.gruell at unibas.ch
>     <mailto:frederik.gruell at unibas.ch> | www.biozentrum.unibas.ch
>     <http://www.biozentrum.unibas.ch>
>
>
>     _______________________________________________
>     ome-devel mailing list
>     ome-devel at lists.openmicroscopy.org.uk
>     <mailto:ome-devel at lists.openmicroscopy.org.uk>
>     http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>     <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

-- 
Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
(61) 207 2250 | frederik.gruell at unibas.ch | www.biozentrum.unibas.ch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20161214/294c2480/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20161214/294c2480/attachment.asc>