<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Hi Paul,</div>
<div class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 19 Mar 2015, at 15:37, Paul van Schayck <<a href="mailto:paul@vanschayck.nl" class="">paul@vanschayck.nl</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Hi Sebastien,<br class="">
<br class="">
Ah thank you! Those changes look excellent. I suspect though, that<br class="">
they are initially only available for the command line importer, and<br class="">
not the Java client?<br class="">
</div>
</blockquote>
<div><br class="">
</div>
<div>Yes you are right. For OMERO 5.1.0, these changes will be available to</div>
<div>the command-line interface only.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">I guess it is technically difficult to do both the upload and checksum<br class="">
in one read, right?<br class="">
</div>
</blockquote>
<div><br class="">
</div>
<div>Actually, the client-side checksums are calculated together with the upload process</div>
<div><span style="text-rendering: optimizelegibility; line-height: 14.300000190734863px;" class=""><font color="#0033cc" face="Lucida Grande" class=""><a href="https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L477" class="">https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L477</a></font></span></div>
<div><br class="">
</div>
<div>But the checksum is also calculating server-side in order to account for either transfer</div>
<div>corruption and/or file-writing problems on the server:</div>
<div><a href="https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L482" class="">https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L482</a></div>
<div><br class="">
</div>
<div>So the checksum calculation effectively happens twice (client-side and server-side)</div>
<div>and this operation can be computationally intensive depending on the file size/number</div>
<div>and the checksum algorithm used.</div>
<div><br class="">
</div>
<div>On OMERO 5.0.x, you should already be able to assess the impact of this checksum</div>
<div>on your import time. Using the command-line interface, you can set the checksum</div>
<div>algorithm to a different method and check the effect on import speed:</div>
<div><a href="https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/in-place-import.html#getting-started" class="">https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/in-place-import.html#getting-started</a></div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">Kind regards,<br class="">
<br class="">
Paul<br class="">
<br class="">
</div>
</blockquote>
<div><br class="">
</div>
Best,</div>
<div>Sebastien</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Thu, Mar 19, 2015 at 3:52 PM, Sebastien Besson <<a href="mailto:seb.besson@gmail.com" class="">seb.besson@gmail.com</a>> wrote:<br class="">
<blockquote type="cite" class="">Hi Paul,<br class="">
<br class="">
the checksumming operation has indeed been identified as one of the<br class="">
duplicate and<br class="">
time-consuming operation. While converging towards 5.1.0, we are trying to<br class="">
put many<br class="">
solutions into place to selectively disable some consuming import steps and<br class="">
let import<br class="">
process of large datasets be much faster. See notably:<br class="">
<br class="">
<a href="https://github.com/openmicroscopy/openmicroscopy/pull/3264" class="">https://github.com/openmicroscopy/openmicroscopy/pull/3264</a><br class="">
https://github.com/openmicroscopy/openmicroscopy/pull/3580<br class="">
https://github.com/openmicroscopy/openmicroscopy/pull/3610<br class="">
https://github.com/openmicroscopy/openmicroscopy/pull/3630<br class="">
<br class="">
Some of this work is still under review and documentation will follow but I<br class="">
would expect<br class="">
these advanced import options will at least partially solve some of your<br class="">
import issues.<br class="">
<br class="">
Best,<br class="">
Sebastien<br class="">
<br class="">
On 19 Mar 2015, at 14:42, Paul van Schayck <paul@vanschayck.nl> wrote:<br class="">
<br class="">
Hi Josh and Melissa,<br class="">
<br class="">
Coming back to this issue. I've got a theory, and I wanted to run this past<br class="">
you.<br class="">
<br class="">
Could it be that the slow down during import is due to the verify<br class="">
step, in which a checksum is calculated both on the importer and the<br class="">
server? Could it be that the file is read twice on the importer side,<br class="">
once during submit and then a second time to calculate the checksum?<br class="">
This would be of less concern on a local disk, but on a network disk<br class="">
this is killing.<br class="">
<br class="">
I've also observed that a ManagedImport job may let the sever run out<br class="">
of memory by starting too many jobs that are not left to finish. Here<br class="">
is a small piece of filtered log. See both the OutOfMemory error, and<br class="">
the delay of 30 minutes.<br class="">
<br class="">
$ grep "40d58dd2" var/log/Blitz-0.log.1<br class="">
2015-03-13 12:07:56,493 DEBUG [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) User<br class="">
callContext: {omero.session.uuid=92c5a250-fd3e-450f-a881-b97b0d5d0e98,<br class="">
omero.client.uuid=b0c821a4-3b8a-4ebe-a7ee-9e485157da5c}<br class="">
2015-03-13 12:07:56,494 INFO [<br class="">
ome.services.util.ServiceHandler] (2-thread-2) Executor.doWork --<br class="">
omero.cmd.HandleI.run(92c5a250-fd3e-450f-a881-b97b0d5d0e98/IHandlec85bb139-737d-46f9-aa0b-33a51e8e6b95,<br class="">
ome.services.blitz.repo.ManagedImportRequestI@40d58dd2)<br class="">
2015-03-13 12:07:57,493 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getRequest:<br class="">
ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">
2015-03-13 12:07:57,840 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) Add callback:<br class="">
~3SJ7}(JvCFzsbw9oOJ*/00b0bd2e-4cbd-4c43-a67b-b6b72b5d7931<br class="">
2015-03-13 12:07:58,536 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getRequest:<br class="">
ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">
2015-03-13 12:07:58,536 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) Cancelled<br class="">
2015-03-13 12:07:58,537 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getResponse:<br class="">
null<br class="">
2015-03-13 12:07:58,537 DEBUG [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) Request<br class="">
cancelled by java.lang.OutOfMemoryError: Java heap space<br class="">
2015-03-13 12:07:58,897 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) notify<br class="">
cancelled: omero.cmd.ERR@2c5bf6d6/omero.cmd.Status@49aa725a<br class="">
2015-03-13 12:07:59,250 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-47) getRequest:<br class="">
ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">
2015-03-13 12:39:04,791 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-53) Closing...<br class="">
2015-03-13 12:39:04,791 INFO [<br class="">
o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-53) notify<br class="">
cancelled: omero.cmd.ERR@2c5bf6d6/omero.cmd.Status@49aa725a<br class="">
<br class="">
Thanks,<br class="">
<br class="">
Paul<br class="">
<br class="">
On Tue, Jan 6, 2015 at 9:53 AM, Paul van Schayck <paul@vanschayck.nl> wrote:<br class="">
<br class="">
Hi Josh,<br class="">
<br class="">
Thank you for your summary. Just before the holidays I did some<br class="">
additional testing but only were able to post the results now:<br class="">
<br class="">
High Latency Network drive (HL): 10Mbit connection, typical 40ms<br class="">
latency (2% packet loss)<br class="">
<br class="">
Low Latency Network drive (LL): 100Mbit connection, typical < 10ms<br class="">
latency (0% packet loss)<br class="">
<br class="">
I compared simple transfering the files to my local machine to<br class="">
importing to the OMERO server. This import to the OMERO server of<br class="">
course takes place via my local machine, hence the comparison. Note<br class="">
that the connection between my local machine and the OMERO server is<br class="">
100mbit/s and < 1ms. I did all tests in duplo and had results within a<br class="">
few percent of the previous.<br class="">
<br class="">
Copying to local machine takes:<br class="">
HL: 19m30s<br class="">
LL: 2m30s<br class="">
<br class="">
Importing to OMERO server takes:<br class="">
HL: 26m30s<br class="">
LL: 12m30s<br class="">
<br class="">
And importing from my local machine to OMERO takes 6m30s. What these<br class="">
results show while there is some overhead to importing, it's not too<br class="">
big. Funny ennough the overhead seems less for the high latency<br class="">
situation.<br class="">
<br class="">
However what was noticeable is that the importer client will appear to<br class="">
hang when the latency is higher. The import will continue, but the<br class="">
client will not show any progress any more and seem to hang. What<br class="">
would help you most to debug such an apparent hang, what will show<br class="">
where Java is waiting for?<br class="">
<br class="">
Thanks,<br class="">
<br class="">
Paul<br class="">
<br class="">
<br class="">
On Fri, Dec 12, 2014 at 4:47 PM, Josh Moore <josh@glencoesoftware.com><br class="">
wrote:<br class="">
<br class="">
Hi all,<br class="">
<br class="">
as a fairly general summary of this thread:<br class="">
<br class="">
* There will always be hidden costs for Bio-Formats and OMERO to work from<br class="">
a mounted disk[1] and as datasets grow in size, we'll probably have to come<br class="">
up with more comprehensive solutions to issues like these, but that won't be<br class="">
possible for 5.1.0<br class="">
<br class="">
* Nevertheless, there is obviously still huge room for improvement. A large<br class="">
part of the underlying problem is that we don't and probably can't regularly<br class="">
test with realistically sized datasets of each file format over various<br class="">
mounting options. This is certainly where you can help! If anyone has<br class="">
similar issues perhaps with other file formats, let us know the details.<br class="">
<br class="">
* In the meantime, we'll start working on the formats and components where<br class="">
we know there are issues and keep you posted. If anyone is interested in<br class="">
getting their hands dirty, likely the most helpful information would be Java<br class="">
hprof[2] output or similar from an example run of what you consider slow.<br class="">
<br class="">
A lovely weekend to one and all,<br class="">
~Josh<br class="">
<br class="">
[1] http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing<br class="">
[2] https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html<br class="">
<br class="">
<br class="">
On Thu, Dec 11, 2014 at 10:11 AM, Melissa Linkert<br class="">
<melissa@glencoesoftware.com> wrote:<br class="">
<br class="">
<br class="">
Hi Paul and Niko,<br class="">
<br class="">
Would this bug also explain slowness in the OMERO importer while<br class="">
importing files which are located on a network drive (in Windows)?<br class="">
<br class="">
<br class="">
It's very possible - the easiest way to know for sure is to compare the<br class="">
import time for the same file on a network drive vs. on a local drive.<br class="">
<br class="">
I doubt this is related - as I understand it, the (current) import<br class="">
process<br class="">
using OMERO.insight just copies the image file(s) onto the server<br class="">
before it<br class="">
is processing them. So the first time the BioFormats kicks in there is<br class="">
*after* the actual transfer and therefore the network drive issue<br class="">
should<br class="">
not apply here.<br class="">
<br class="">
<br class="">
That's mostly correct, but Bio-Formats is still used client-side at<br class="">
import time send metadata to the server and to determine which files<br class="">
need to be uploaded. Bio-Formats definitely would be accessing the<br class="">
files as they are on the network drive.<br class="">
<br class="">
You're probably right that the File.isHidden() in BioFormats isn't<br class="">
called<br class="">
in the OMERO importer at the client side. However, from quickly grepping<br class="">
through the source of openmicroscopy/components/insight I found several<br class="">
calls to File.isHidden(). I was wondering if these are also meant to be<br class="">
optimized as part of this issue.<br class="">
<br class="">
<br class="">
We weren't initially considering the OMERO code base, but once this<br class="">
issue is resolved in Bio-Formats it likely makes sense to use the same<br class="">
fix in OMERO.<br class="">
<br class="">
Regards,<br class="">
-Melissa<br class="">
<br class="">
P.S. Niko, you are now CC'd on the ticket as well.<br class="">
<br class="">
On Thu, Dec 11, 2014 at 09:07:41PM +0100, Paul van Schayck wrote:<br class="">
<br class="">
Hi Niko,<br class="">
<br class="">
You're probably right that the File.isHidden() in BioFormats isn't<br class="">
called<br class="">
in the OMERO importer at the client side. However, from quickly grepping<br class="">
through the source of openmicroscopy/components/insight I found several<br class="">
calls to File.isHidden(). I was wondering if these are also meant to be<br class="">
optimized as part of this issue.<br class="">
<br class="">
What we experience in the OMERO importer is for very low latency network<br class="">
drives that the import is relatively slow (compared to file copy). For<br class="">
higher latency network drives the import is slow and the importer UI<br class="">
becomes unresponsive (although the import continues). If you want me to<br class="">
quantify these experiences, I could do that.<br class="">
<br class="">
Thanks,<br class="">
<br class="">
Paul<br class="">
<br class="">
<br class="">
On Thu, Dec 11, 2014 at 8:22 PM, Niko Ehrenfeuchter <<br class="">
nikolaus.ehrenfeuchter@unibas.ch> wrote:<br class="">
<br class="">
Hi Paul,<br class="">
<br class="">
I doubt this is related - as I understand it, the (current) import<br class="">
process<br class="">
using OMERO.insight just copies the image file(s) onto the server<br class="">
before it<br class="">
is processing them. So the first time the BioFormats kicks in there is<br class="">
*after* the actual transfer and therefore the network drive issue<br class="">
should<br class="">
not apply here.<br class="">
<br class="">
I might be wrong, though...<br class="">
<br class="">
Cheers<br class="">
Niko<br class="">
<br class="">
On 11.12.2014 17:29, Paul van Schayck wrote:<br class="">
<br class="">
Hi Melissa,<br class="">
<br class="">
Would this bug also explain slowness in the OMERO importer while<br class="">
importing files which are located on a network drive (in Windows)?<br class="">
<br class="">
Thanks,<br class="">
<br class="">
Paul<br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
<br class="">
</blockquote>
_______________________________________________<br class="">
ome-users mailing list<br class="">
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</blockquote>
</div>
<br class="">
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>