[ome-users] Network drive slowness

[Paul van Schayck]

Hi Josh,

Thank you for your summary. Just before the holidays I did some
additional testing but only were able to post the results now:

High Latency Network drive (HL): 10Mbit connection, typical 40ms
latency (2% packet loss)

Low Latency Network drive (LL): 100Mbit connection, typical < 10ms
latency (0% packet loss)

I compared simple transfering the files to my local machine to
importing to the OMERO server. This import to the OMERO server of
course takes place via my local machine, hence the comparison. Note
that the connection between my local machine and the OMERO server is
100mbit/s and < 1ms. I did all tests in duplo and had results within a
few percent of the previous.

Copying to local machine takes:
HL: 19m30s
LL: 2m30s

Importing to OMERO server takes:
HL: 26m30s
LL: 12m30s

And importing from my local machine to OMERO takes 6m30s. What these
results show while there is some overhead to importing, it's not too
big. Funny ennough the overhead seems less for the high latency
situation.

However what was noticeable is that the importer client will appear to
hang when the latency is higher. The import will continue, but the
client will not show any progress any more and seem to hang. What
would help you most to debug such an apparent hang, what will show
where Java is waiting for?

Thanks,

Paul


On Fri, Dec 12, 2014 at 4:47 PM, Josh Moore <josh at glencoesoftware.com> wrote:
> Hi all,
>
> as a fairly general summary of this thread:
>
>  * There will always be hidden costs for Bio-Formats and OMERO to work from
> a mounted disk[1] and as datasets grow in size, we'll probably have to come
> up with more comprehensive solutions to issues like these, but that won't be
> possible for 5.1.0
>
>  * Nevertheless, there is obviously still huge room for improvement. A large
> part of the underlying problem is that we don't and probably can't regularly
> test with realistically sized datasets of each file format over various
> mounting options. This is certainly where you can help! If anyone has
> similar issues perhaps with other file formats, let us know the details.
>
>  * In the meantime, we'll start working on the formats and components where
> we know there are issues and keep you posted. If anyone is interested in
> getting their hands dirty, likely the most helpful information would be Java
> hprof[2] output or similar from an example run of what you consider slow.
>
> A lovely weekend to one and all,
> ~Josh
>
> [1] http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
> [2] https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html
>
>
> On Thu, Dec 11, 2014 at 10:11 AM, Melissa Linkert
> <melissa at glencoesoftware.com> wrote:
>>
>> Hi Paul and Niko,
>>
>> > >> Would this bug also explain slowness in the OMERO importer while
>> > >> importing files which are located on a network drive (in Windows)?
>>
>> It's very possible - the easiest way to know for sure is to compare the
>> import time for the same file on a network drive vs. on a local drive.
>>
>> > > I doubt this is related - as I understand it, the (current) import
>> > > process
>> > > using OMERO.insight just copies the image file(s) onto the server
>> > > before it
>> > > is processing them. So the first time the BioFormats kicks in there is
>> > > *after* the actual transfer and therefore the network drive issue
>> > > should
>> > > not apply here.
>>
>> That's mostly correct, but Bio-Formats is still used client-side at
>> import time send metadata to the server and to determine which files
>> need to be uploaded.  Bio-Formats definitely would be accessing the
>> files as they are on the network drive.
>>
>> > You're probably right that the File.isHidden() in BioFormats isn't
>> > called
>> > in the OMERO importer at the client side. However, from quickly grepping
>> > through the source of openmicroscopy/components/insight I found several
>> > calls to File.isHidden(). I was wondering if these are also meant to be
>> > optimized as part of this issue.
>>
>> We weren't initially considering the OMERO code base, but once this
>> issue is resolved in Bio-Formats it likely makes sense to use the same
>> fix in OMERO.
>>
>> Regards,
>> -Melissa
>>
>> P.S. Niko, you are now CC'd on the ticket as well.
>>
>> On Thu, Dec 11, 2014 at 09:07:41PM +0100, Paul van Schayck wrote:
>> > Hi Niko,
>> >
>> > You're probably right that the File.isHidden() in BioFormats isn't
>> > called
>> > in the OMERO importer at the client side. However, from quickly grepping
>> > through the source of openmicroscopy/components/insight I found several
>> > calls to File.isHidden(). I was wondering if these are also meant to be
>> > optimized as part of this issue.
>> >
>> > What we experience in the OMERO importer is for very low latency network
>> > drives that the import is relatively slow (compared to file copy). For
>> > higher latency network drives the import is slow and the importer UI
>> > becomes unresponsive (although the import continues). If you want me to
>> > quantify these experiences, I could do that.
>> >
>> > Thanks,
>> >
>> > Paul
>> >
>> >
>> > On Thu, Dec 11, 2014 at 8:22 PM, Niko Ehrenfeuchter <
>> > nikolaus.ehrenfeuchter at unibas.ch> wrote:
>> >
>> > > Hi Paul,
>> > >
>> > > I doubt this is related - as I understand it, the (current) import
>> > > process
>> > > using OMERO.insight just copies the image file(s) onto the server
>> > > before it
>> > > is processing them. So the first time the BioFormats kicks in there is
>> > > *after* the actual transfer and therefore the network drive issue
>> > > should
>> > > not apply here.
>> > >
>> > > I might be wrong, though...
>> > >
>> > > Cheers
>> > > Niko
>> > >
>> > > On 11.12.2014 17:29, Paul van Schayck wrote:
>> > >
>> > >> Hi Melissa,
>> > >>
>> > >> Would this bug also explain slowness in the OMERO importer while
>> > >> importing files which are located on a network drive (in Windows)?
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Paul
>> > >>