[ome-users] Network drive slowness

[Josh Moore]

Hi all,

as a fairly general summary of this thread:

 * There will always be hidden costs for Bio-Formats and OMERO to work from
a mounted disk[1] and as datasets grow in size, we'll probably have to come
up with more comprehensive solutions to issues like these, but that won't
be possible for 5.1.0

 * Nevertheless, there is obviously still huge room for improvement. A
large part of the underlying problem is that we don't and probably can't
regularly test with realistically sized datasets of each file format over
various mounting options. This is certainly where you can help! If anyone
has similar issues perhaps with other file formats, let us know the details.

 * In the meantime, we'll start working on the formats and components where
we know there are issues and keep you posted. If anyone is interested in
getting their hands dirty, likely the most helpful information would be
Java hprof[2] output or similar from an example run of what you consider
slow.

A lovely weekend to one and all,
~Josh

[1] http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
[2] https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html


On Thu, Dec 11, 2014 at 10:11 AM, Melissa Linkert <
melissa at glencoesoftware.com> wrote:
>
> Hi Paul and Niko,
>
> > >> Would this bug also explain slowness in the OMERO importer while
> > >> importing files which are located on a network drive (in Windows)?
>
> It's very possible - the easiest way to know for sure is to compare the
> import time for the same file on a network drive vs. on a local drive.
>
> > > I doubt this is related - as I understand it, the (current) import
> process
> > > using OMERO.insight just copies the image file(s) onto the server
> before it
> > > is processing them. So the first time the BioFormats kicks in there is
> > > *after* the actual transfer and therefore the network drive issue
> should
> > > not apply here.
>
> That's mostly correct, but Bio-Formats is still used client-side at
> import time send metadata to the server and to determine which files
> need to be uploaded.  Bio-Formats definitely would be accessing the
> files as they are on the network drive.
>
> > You're probably right that the File.isHidden() in BioFormats isn't called
> > in the OMERO importer at the client side. However, from quickly grepping
> > through the source of openmicroscopy/components/insight I found several
> > calls to File.isHidden(). I was wondering if these are also meant to be
> > optimized as part of this issue.
>
> We weren't initially considering the OMERO code base, but once this
> issue is resolved in Bio-Formats it likely makes sense to use the same
> fix in OMERO.
>
> Regards,
> -Melissa
>
> P.S. Niko, you are now CC'd on the ticket as well.
>
> On Thu, Dec 11, 2014 at 09:07:41PM +0100, Paul van Schayck wrote:
> > Hi Niko,
> >
> > You're probably right that the File.isHidden() in BioFormats isn't called
> > in the OMERO importer at the client side. However, from quickly grepping
> > through the source of openmicroscopy/components/insight I found several
> > calls to File.isHidden(). I was wondering if these are also meant to be
> > optimized as part of this issue.
> >
> > What we experience in the OMERO importer is for very low latency network
> > drives that the import is relatively slow (compared to file copy). For
> > higher latency network drives the import is slow and the importer UI
> > becomes unresponsive (although the import continues). If you want me to
> > quantify these experiences, I could do that.
> >
> > Thanks,
> >
> > Paul
> >
> >
> > On Thu, Dec 11, 2014 at 8:22 PM, Niko Ehrenfeuchter <
> > nikolaus.ehrenfeuchter at unibas.ch> wrote:
> >
> > > Hi Paul,
> > >
> > > I doubt this is related - as I understand it, the (current) import
> process
> > > using OMERO.insight just copies the image file(s) onto the server
> before it
> > > is processing them. So the first time the BioFormats kicks in there is
> > > *after* the actual transfer and therefore the network drive issue
> should
> > > not apply here.
> > >
> > > I might be wrong, though...
> > >
> > > Cheers
> > > Niko
> > >
> > > On 11.12.2014 17:29, Paul van Schayck wrote:
> > >
> > >> Hi Melissa,
> > >>
> > >> Would this bug also explain slowness in the OMERO importer while
> > >> importing files which are located on a network drive (in Windows)?
> > >>
> > >> Thanks,
> > >>
> > >> Paul
> > >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20141212/d2844c60/attachment.html>