[ome-users] Network drive slowness

[Paul van Schayck]

Hi Sebastien,

Ah tracing that code  a bit further leads to:

https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/transfers/UploadFileTransfer.java#L71

I think this line shows that both checksum calculation and uploading
are done using one file read on the client side. This takes away my
concern for a bottleneck there.

I was aware that the checksum calculation was repeated server side.
But I'm not worried about calculating the checksum, in our workflow we
are using small JPG files for which calculating the checksum is
trivial. I'm searching for bottlenecks when the import takes place
from a network drive on the client side. This leads to unresponsive
importer client and out of memory server when are importing several
thousand images.

Thanks,

Paul


On Fri, Mar 20, 2015 at 9:55 AM, Sebastien Besson (Staff)
<s.besson at dundee.ac.uk> wrote:
> Hi Paul,
>
>
> On 19 Mar 2015, at 15:37, Paul van Schayck <paul at vanschayck.nl> wrote:
>
> Hi Sebastien,
>
> Ah thank you! Those changes look excellent. I suspect though, that
> they are initially only available for the command line importer, and
> not the Java client?
>
>
> Yes you are right. For OMERO 5.1.0, these changes will be available to
> the command-line interface only.
>
> I guess it is technically difficult to do both the upload and checksum
> in one read, right?
>
>
> Actually, the client-side checksums are calculated together with the upload
> process
> https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L477
>
> But the checksum is also calculating server-side in order to account for
> either transfer
> corruption and/or file-writing problems on the server:
> https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L482
>
> So the checksum calculation effectively happens twice (client-side and
> server-side)
> and this operation can be computationally intensive depending on the file
> size/number
> and the checksum algorithm used.
>
> On OMERO 5.0.x, you should already be able to assess the impact of this
> checksum
> on your import time. Using the command-line interface, you can set the
> checksum
> algorithm to a different method and check the effect on import speed:
> https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/in-place-import.html#getting-started
>
> Kind regards,
>
> Paul
>
>
> Best,
> Sebastien