<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Paul,<div class=""><br class=""></div><div class="">the checksumming operation has indeed been identified as one of the duplicate and</div><div class="">time-consuming operation. While converging towards 5.1.0, we are trying to put many</div><div class="">solutions into place to selectively disable some consuming import steps and let import</div><div class="">process of large datasets be much faster. See notably:</div><div class=""><br class=""></div><div class=""><a href="https://github.com/openmicroscopy/openmicroscopy/pull/3264" class="">https://github.com/openmicroscopy/openmicroscopy/pull/3264</a></div><div class=""><a href="https://github.com/openmicroscopy/openmicroscopy/pull/3580" class="">https://github.com/openmicroscopy/openmicroscopy/pull/3580</a></div><div class=""><a href="https://github.com/openmicroscopy/openmicroscopy/pull/3610" class="">https://github.com/openmicroscopy/openmicroscopy/pull/3610</a></div><div class=""><a href="https://github.com/openmicroscopy/openmicroscopy/pull/3630" class="">https://github.com/openmicroscopy/openmicroscopy/pull/3630</a></div><div class=""><br class=""></div><div class=""><div class="">Some of this work is still under review and documentation will follow but I would expect</div><div class="">these advanced import options will at least partially solve some of your import issues.</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Sebastien</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 19 Mar 2015, at 14:42, Paul van Schayck <<a href="mailto:paul@vanschayck.nl" class="">paul@vanschayck.nl</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">Hi Josh and Melissa,<br class=""><br class="">Coming back to this issue. I've got a theory, and I wanted to run this past you.<br class=""><br class="">Could it be that the slow down during import is due to the verify<br class="">step, in which a checksum is calculated both on the importer and the<br class="">server? Could it be that the file is read twice on the importer side,<br class="">once during submit and then a second time to calculate the checksum?<br class="">This would be of less concern on a local disk, but on a network disk<br class="">this is killing.<br class=""><br class="">I've also observed that a ManagedImport job may let the sever run out<br class="">of memory by starting too many jobs that are not left to finish. Here<br class="">is a small piece of filtered log. See both the OutOfMemory error, and<br class="">the delay of 30 minutes.<br class=""><br class="">$ grep "40d58dd2" var/log/Blitz-0.log.1<br class="">2015-03-13 12:07:56,493 DEBUG [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) User<br class="">callContext: {omero.session.uuid=92c5a250-fd3e-450f-a881-b97b0d5d0e98,<br class="">omero.client.uuid=b0c821a4-3b8a-4ebe-a7ee-9e485157da5c}<br class="">2015-03-13 12:07:56,494 INFO [<br class="">ome.services.util.ServiceHandler] (2-thread-2) Executor.doWork --<br class="">omero.cmd.HandleI.run(92c5a250-fd3e-450f-a881-b97b0d5d0e98/IHandlec85bb139-737d-46f9-aa0b-33a51e8e6b95,<br class="">ome.services.blitz.repo.ManagedImportRequestI@40d58dd2)<br class="">2015-03-13 12:07:57,493 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getRequest:<br class="">ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">2015-03-13 12:07:57,840 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) Add callback:<br class="">~3SJ7}(JvCFzsbw9oOJ*/00b0bd2e-4cbd-4c43-a67b-b6b72b5d7931<br class="">2015-03-13 12:07:58,536 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getRequest:<br class="">ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">2015-03-13 12:07:58,536 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) Cancelled<br class="">2015-03-13 12:07:58,537 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-54) getResponse:<br class="">null<br class="">2015-03-13 12:07:58,537 DEBUG [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) Request<br class="">cancelled by java.lang.OutOfMemoryError: Java heap space<br class="">2015-03-13 12:07:58,897 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (2-thread-2) notify<br class="">cancelled: omero.cmd.ERR@2c5bf6d6/omero.cmd.Status@49aa725a<br class="">2015-03-13 12:07:59,250 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-47) getRequest:<br class="">ome.services.blitz.repo.ManagedImportRequestI@40d58dd2<br class="">2015-03-13 12:39:04,791 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-53) Closing...<br class="">2015-03-13 12:39:04,791 INFO [<br class="">o.s.b.r.ManagedImportRequestI.@40d58dd2] (.Server-53) notify<br class="">cancelled: omero.cmd.ERR@2c5bf6d6/omero.cmd.Status@49aa725a<br class=""><br class="">Thanks,<br class=""><br class="">Paul<br class=""><br class="">On Tue, Jan 6, 2015 at 9:53 AM, Paul van Schayck <<a href="mailto:paul@vanschayck.nl" class="">paul@vanschayck.nl</a>> wrote:<br class=""><blockquote type="cite" class="">Hi Josh,<br class=""><br class="">Thank you for your summary. Just before the holidays I did some<br class="">additional testing but only were able to post the results now:<br class=""><br class="">High Latency Network drive (HL): 10Mbit connection, typical 40ms<br class="">latency (2% packet loss)<br class=""><br class="">Low Latency Network drive (LL): 100Mbit connection, typical < 10ms<br class="">latency (0% packet loss)<br class=""><br class="">I compared simple transfering the files to my local machine to<br class="">importing to the OMERO server. This import to the OMERO server of<br class="">course takes place via my local machine, hence the comparison. Note<br class="">that the connection between my local machine and the OMERO server is<br class="">100mbit/s and < 1ms. I did all tests in duplo and had results within a<br class="">few percent of the previous.<br class=""><br class="">Copying to local machine takes:<br class="">HL: 19m30s<br class="">LL: 2m30s<br class=""><br class="">Importing to OMERO server takes:<br class="">HL: 26m30s<br class="">LL: 12m30s<br class=""><br class="">And importing from my local machine to OMERO takes 6m30s. What these<br class="">results show while there is some overhead to importing, it's not too<br class="">big. Funny ennough the overhead seems less for the high latency<br class="">situation.<br class=""><br class="">However what was noticeable is that the importer client will appear to<br class="">hang when the latency is higher. The import will continue, but the<br class="">client will not show any progress any more and seem to hang. What<br class="">would help you most to debug such an apparent hang, what will show<br class="">where Java is waiting for?<br class=""><br class="">Thanks,<br class=""><br class="">Paul<br class=""><br class=""><br class="">On Fri, Dec 12, 2014 at 4:47 PM, Josh Moore <<a href="mailto:josh@glencoesoftware.com" class="">josh@glencoesoftware.com</a>> wrote:<br class=""><blockquote type="cite" class="">Hi all,<br class=""><br class="">as a fairly general summary of this thread:<br class=""><br class=""> * There will always be hidden costs for Bio-Formats and OMERO to work from<br class="">a mounted disk[1] and as datasets grow in size, we'll probably have to come<br class="">up with more comprehensive solutions to issues like these, but that won't be<br class="">possible for 5.1.0<br class=""><br class=""> * Nevertheless, there is obviously still huge room for improvement. A large<br class="">part of the underlying problem is that we don't and probably can't regularly<br class="">test with realistically sized datasets of each file format over various<br class="">mounting options. This is certainly where you can help! If anyone has<br class="">similar issues perhaps with other file formats, let us know the details.<br class=""><br class=""> * In the meantime, we'll start working on the formats and components where<br class="">we know there are issues and keep you posted. If anyone is interested in<br class="">getting their hands dirty, likely the most helpful information would be Java<br class="">hprof[2] output or similar from an example run of what you consider slow.<br class=""><br class="">A lovely weekend to one and all,<br class="">~Josh<br class=""><br class="">[1] <a href="http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing" class="">http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing</a><br class="">[2] <a href="https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html" class="">https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html</a><br class=""><br class=""><br class="">On Thu, Dec 11, 2014 at 10:11 AM, Melissa Linkert<br class=""><<a href="mailto:melissa@glencoesoftware.com" class="">melissa@glencoesoftware.com</a>> wrote:<br class=""><blockquote type="cite" class=""><br class="">Hi Paul and Niko,<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Would this bug also explain slowness in the OMERO importer while<br class="">importing files which are located on a network drive (in Windows)?<br class=""></blockquote></blockquote></blockquote><br class="">It's very possible - the easiest way to know for sure is to compare the<br class="">import time for the same file on a network drive vs. on a local drive.<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">I doubt this is related - as I understand it, the (current) import<br class="">process<br class="">using OMERO.insight just copies the image file(s) onto the server<br class="">before it<br class="">is processing them. So the first time the BioFormats kicks in there is<br class="">*after* the actual transfer and therefore the network drive issue<br class="">should<br class="">not apply here.<br class=""></blockquote></blockquote><br class="">That's mostly correct, but Bio-Formats is still used client-side at<br class="">import time send metadata to the server and to determine which files<br class="">need to be uploaded. Bio-Formats definitely would be accessing the<br class="">files as they are on the network drive.<br class=""><br class=""><blockquote type="cite" class="">You're probably right that the File.isHidden() in BioFormats isn't<br class="">called<br class="">in the OMERO importer at the client side. However, from quickly grepping<br class="">through the source of openmicroscopy/components/insight I found several<br class="">calls to File.isHidden(). I was wondering if these are also meant to be<br class="">optimized as part of this issue.<br class=""></blockquote><br class="">We weren't initially considering the OMERO code base, but once this<br class="">issue is resolved in Bio-Formats it likely makes sense to use the same<br class="">fix in OMERO.<br class=""><br class="">Regards,<br class="">-Melissa<br class=""><br class="">P.S. Niko, you are now CC'd on the ticket as well.<br class=""><br class="">On Thu, Dec 11, 2014 at 09:07:41PM +0100, Paul van Schayck wrote:<br class=""><blockquote type="cite" class="">Hi Niko,<br class=""><br class="">You're probably right that the File.isHidden() in BioFormats isn't<br class="">called<br class="">in the OMERO importer at the client side. However, from quickly grepping<br class="">through the source of openmicroscopy/components/insight I found several<br class="">calls to File.isHidden(). I was wondering if these are also meant to be<br class="">optimized as part of this issue.<br class=""><br class="">What we experience in the OMERO importer is for very low latency network<br class="">drives that the import is relatively slow (compared to file copy). For<br class="">higher latency network drives the import is slow and the importer UI<br class="">becomes unresponsive (although the import continues). If you want me to<br class="">quantify these experiences, I could do that.<br class=""><br class="">Thanks,<br class=""><br class="">Paul<br class=""><br class=""><br class="">On Thu, Dec 11, 2014 at 8:22 PM, Niko Ehrenfeuchter <<br class=""><a href="mailto:nikolaus.ehrenfeuchter@unibas.ch" class="">nikolaus.ehrenfeuchter@unibas.ch</a>> wrote:<br class=""><br class=""><blockquote type="cite" class="">Hi Paul,<br class=""><br class="">I doubt this is related - as I understand it, the (current) import<br class="">process<br class="">using OMERO.insight just copies the image file(s) onto the server<br class="">before it<br class="">is processing them. So the first time the BioFormats kicks in there is<br class="">*after* the actual transfer and therefore the network drive issue<br class="">should<br class="">not apply here.<br class=""><br class="">I might be wrong, though...<br class=""><br class="">Cheers<br class="">Niko<br class=""><br class="">On 11.12.2014 17:29, Paul van Schayck wrote:<br class=""><br class=""><blockquote type="cite" class="">Hi Melissa,<br class=""><br class="">Would this bug also explain slowness in the OMERO importer while<br class="">importing files which are located on a network drive (in Windows)?<br class=""><br class="">Thanks,<br class=""><br class="">Paul<br class=""><br class=""></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>_______________________________________________<br class="">ome-users mailing list<br class=""><a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><br class="">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class=""></div></blockquote></div><br class=""></div></div></body></html>