<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Frederik<br>
<br>
</div>
Could you give us your server configuration and diagnostics:<br>
<br>
omero config get --hide-password<br>
omero admin diagnostics<br>
<br>
</div>
It would also be helpful if we could see your logs for all OMERO services, not just Blitz. Would you mind uploading them to
<a href="https://www.openmicroscopy.org/qa2/qa/upload/" target="_blank">https://www.openmicroscopy.<wbr>org/qa2/qa/upload/</a> and giving us the timestamp of when the problem first arises following a restart?<br>
<br>
</div>
Best wishes<br>
<br>
</div>
Simon<br>
<div>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 13 December 2016 at 10:49, Frederik Grüll <span dir="ltr">
<<a href="mailto:frederik.gruell@unibas.ch" target="_blank">frederik.gruell@unibas.ch</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear all,<br>
<br>
I am using CellProfiler on our cluster to process plates for screening.<br>
The images are fetched from OMERO with the CellProfiler-OMERO<br>
integration. A typical job consists of a command like this:<br>
<br>
cellprofiler -b -p Entry-pipeline_omero.cpproj -c -r -o $OUT_DIR -t<br>
$TMPDIR -f $FIRST_IMAGE_SET -l $LAST_IMAGE_SET --data-file<br>
plate_303_iids.csv -d $DONE_FILE --omero-credentials<br>
host=<a href="http://omero.biozentrum.unibas.ch" rel="noreferrer" target="_blank">omero.biozentrum.unibas.<wbr>ch</a>,port=4064,session-id=<wbr>33c6118d-f8b2-4ac2-adb2-<wbr>12d48ae37a2f<br>
<br>
When I run about 20 jobs in parallel, performance looks good at the<br>
beginning, only limited by the performance of CellProfiler and not by<br>
the I/O with OMERO. The plate I am processing has 2400 sites with three<br>
channels and the OMERO IDs are in the CSV file plate_303_iids.csv that I<br>
generated before. A job processes 50 image sets, selected with<br>
$FIRST_IMAGE_SET and $LAST_IMAGE_SET. The results of the pipeline are<br>
correct.<br>
<br>
However, after about 4/5 of the images have been processed, OMERO<br>
becomes very slow. The load on the OMERO server reaches 10, with the<br>
Java process for Blitz consuming 10 cores. Eventually, my CellProfiler<br>
jobs will loose connection ("JavaException:<br>
Ice.ConnectionLostException"), and OMERO recovers in a few cases or<br>
otherwise the CPU load falls back to normal, but OMERO needs to be<br>
restarted anyway.<br>
<br>
If I run more than 20 jobs in parallel, I would occasional get an error<br>
message "ome.conditions.<wbr>OverUsageException: servantsPerSession reached<br>
for 05dbc314-3030-40af-8e72-<wbr>68b3688e8c94: 10000" after CellProfiler<br>
processed only 1665 single-channel images, implying 6 servants per image<br>
per channel.<br>
<br>
I have already had a look into the logs, especially Blitz-0.log, but<br>
could not find a reason why OMERO would become so slow after a while.<br>
Jstat indicates that all time is spend on garbage collection. Our OMERO<br>
server has 250GB of RAM with omero.jvmcfg.percent.blitz=40.<br>
<br>
Where else could I look into to find the cause and prevent the<br>
degradation in performance? I use OMERO.server 5.2.5 with OpenJDK<br>
version 1.8.0_65 and CellProfiler 2.2.0 with Oracle Java 1.8.0_92.<br>
<br>
Cheers and thank you for your time,<br>
Frederik<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,<br>
University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41<br>
(61) 207 2250 | <a href="mailto:frederik.gruell@unibas.ch">frederik.gruell@unibas.ch</a> |
<a href="http://www.biozentrum.unibas.ch" rel="noreferrer" target="_blank">www.biozentrum.unibas.ch</a><br>
<br>
</font></span><br>
______________________________<wbr>_________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.<wbr>openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" rel="noreferrer" target="_blank">http://lists.openmicroscopy.<wbr>org.uk/mailman/listinfo/ome-<wbr>devel</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>