<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Hi Frederik</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
There is no direct way to retrieve the information you need by querying the OMERO API</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Mark Carroll has written a testing application <a href="https://github.com/mtbc/omero-downloader">https://github.com/mtbc/omero-downloader</a> (experimental work in progress)</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
that will probably get you where you want.</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
The first part of the following method (line 232-260)</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<a href="https://github.com/mtbc/omero-downloader/blob/dev_5_3/src/main/java/org/openmicroscopy/client/downloader/Download.java#L232">https://github.com/mtbc/omero-downloader/blob/dev_5_3/src/main/java/org/openmicroscopy/client/downloader/Download.java#L232</a></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<font face="Calibri,sans-serif">Shows how to retrieve the original files composing a given image</font></div>
<div><font face="Calibri,sans-serif">You won’t need the downloading part of the method</font></div>
<div><font face="Calibri,sans-serif"><br>
</font></div>
<div><font face="Calibri,sans-serif">The order of the id of the files “should” match the channels order. This will have to be validated with your format</font></div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
To test it, you can run the “downloader” app against your omero server with the following arguments:</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
-u username -w password -s hostname -p port -d absolutePathToLocalDirectory Image:image_id -f binary</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
This will only download the files composing the image in absolutePathToLocalDirectory</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Cheers</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
Jmarie</div>
<div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>ome-devel <<a href="mailto:ome-devel-bounces@lists.openmicroscopy.org.uk">ome-devel-bounces@lists.openmicroscopy.org.uk</a>> on behalf of Frederik Grüll <<a href="mailto:frederik.gruell@unibas.ch">frederik.gruell@unibas.ch</a>><br>
<span style="font-weight:bold">Reply-To: </span>OME External Developer List <<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, 15 December 2016 10:14<br>
<span style="font-weight:bold">To: </span>OME External Developer List <<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [ome-devel] CellProfiler on the cluster crashes OMERO<br>
</div>
<div><br>
</div>
<div>
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello Chris,</p>
<p>Is there maybe a workaround, for example by loading the images not through OMERO but from the cluster file system? When I import a plate I select a HTD file and OMERO.insight uploads all the TIFFs of the plate along with it, with one TIFF per channel, and
displays them as a multi-channel view as expected. I have four channels and would be interested in the filesystem location of the four TIFF files that back each image, and the information which TIFF belongs to which channel. I could then use this information
to generate a CSV file for CellProfiler with paths instead of OMERO URLs.<br>
</p>
<p>I had experimented a bit with image.getFileset() to get the original TIFFs. However, when I call the method, it takes a long time and then returns the paths of _all_ 9600 images of the plate, not only the paths of the four TIFFs I wanted. Is there a way
to just get the paths to the TIFFs of a view only, plus the information which TIFF backs which channel?<br>
</p>
<p>Cheers,</p>
<p>Frederik<br>
</p>
<br>
<div class="moz-cite-prefix">On 13.12.2016 14:57, Chris Allan wrote:<br>
</div>
<blockquote cite="mid:5B4AAC2E-0B1D-4E89-BE71-72E2E9A94736@glencoesoftware.com" type="cite">
<pre wrap="">Hello, Frederick:
There are a number of issues with the CellProfiler-OMERO integration that will be felt especially hard on any significantly sized cluster configuration. At Glencoe Software we have been working on some solutions and have be liaising with Anne Carpenter's team here:
<a class="moz-txt-link-freetext" href="https://github.com/CellProfiler/CellProfiler/issues/1772">https://github.com/CellProfiler/CellProfiler/issues/1772</a>
We are aiming to release at least one potential solution to the community before the end of the year. Testing such solutions thoroughly is, as you can imagine, quite time consuming.
As things stand I think it's fair to say that (1) if you utilise a large number of workers or are processing a large dataset CellProfiler-OMERO usage will severely impact your OMERO server performance-- it may even exhaust all resources you currently have allocated to OMERO; and (2) that even if you utilise some of our forthcoming changes you will still have to be very careful about resource allocation on your server. CellProfiler-OMERO in a cluster configuration can easily request data at a rate in excess of 1gbps for short periods of time.
I realise none of the above is particularly helpful to you right this minute but I hope that it at least helps shed some light on why you're seeing what you're seeing. At the moment my only concrete suggestion to you is to keep your CellProfiler executions short-lived and your parallelisation limited.
-Chris
On 13 Dec 2016, at 10:49, Frederik Grüll <a class="moz-txt-link-rfc2396E" href="mailto:frederik.gruell@unibas.ch"><frederik.gruell@unibas.ch></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Dear all,
I am using CellProfiler on our cluster to process plates for screening.
The images are fetched from OMERO with the CellProfiler-OMERO
integration. A typical job consists of a command like this:
cellprofiler -b -p Entry-pipeline_omero.cpproj -c -r -o $OUT_DIR -t
$TMPDIR -f $FIRST_IMAGE_SET -l $LAST_IMAGE_SET --data-file
plate_303_iids.csv -d $DONE_FILE --omero-credentials
host=omero.biozentrum.unibas.ch,port=4064,session-id=33c6118d-f8b2-4ac2-adb2-12d48ae37a2f
When I run about 20 jobs in parallel, performance looks good at the
beginning, only limited by the performance of CellProfiler and not by
the I/O with OMERO. The plate I am processing has 2400 sites with three
channels and the OMERO IDs are in the CSV file plate_303_iids.csv that I
generated before. A job processes 50 image sets, selected with
$FIRST_IMAGE_SET and $LAST_IMAGE_SET. The results of the pipeline are
correct.
However, after about 4/5 of the images have been processed, OMERO
becomes very slow. The load on the OMERO server reaches 10, with the
Java process for Blitz consuming 10 cores. Eventually, my CellProfiler
jobs will loose connection ("JavaException:
Ice.ConnectionLostException"), and OMERO recovers in a few cases or
otherwise the CPU load falls back to normal, but OMERO needs to be
restarted anyway.
If I run more than 20 jobs in parallel, I would occasional get an error
message "ome.conditions.OverUsageException: servantsPerSession reached
for 05dbc314-3030-40af-8e72-68b3688e8c94: 10000" after CellProfiler
processed only 1665 single-channel images, implying 6 servants per image
per channel.
I have already had a look into the logs, especially Blitz-0.log, but
could not find a reason why OMERO would become so slow after a while.
Jstat indicates that all time is spend on garbage collection. Our OMERO
server has 250GB of RAM with omero.jvmcfg.percent.blitz=40.
Where else could I look into to find the cause and prevent the
degradation in performance? I use OMERO.server 5.2.5 with OpenJDK
version 1.8.0_65 and CellProfiler 2.2.0 with Oracle Java 1.8.0_92.
Cheers and thank you for your time,
Frederik
--
Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum,
University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41
(61) 207 2250 | <a class="moz-txt-link-abbreviated" href="mailto:frederik.gruell@unibas.ch">frederik.gruell@unibas.ch</a> | <a class="moz-txt-link-abbreviated" href="http://www.biozentrum.unibas.ch">www.biozentrum.unibas.ch</a>
_______________________________________________
ome-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><a class="moz-txt-link-freetext" href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a></pre>
</blockquote>
<pre wrap="">_______________________________________________
ome-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><a class="moz-txt-link-freetext" href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a></pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
Dr. Frederik Grüll | Image Analysis Specialist | G1055, Biozentrum, University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41 (61) 207 2250 |
<a class="moz-txt-link-abbreviated" href="mailto:frederik.gruell@unibas.ch"></a><a class="moz-txt-link-abbreviated" href="mailto:frederik.gruell@unibas.ch">frederik.gruell@unibas.ch</a> |
<a class="moz-txt-link-abbreviated" href="http://www.biozentrum.unibas.ch">www.biozentrum.unibas.ch</a></div>
</div>
</div>
</span><br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>