[ome-users] Network drive slowness

[Josh Moore]

Evening, Paul.

On Fri, Mar 20, 2015 at 9:41 AM, Paul van Schayck <paul at vanschayck.nl> wrote:

> I think this line shows that both checksum calculation and uploading
> are done using one file read on the client side. This takes away my
> concern for a bottleneck there.

Excellent.


> I was aware that the checksum calculation was repeated server side.
> But I'm not worried about calculating the checksum, in our workflow we
> are using small JPG files for which calculating the checksum is
> trivial. I'm searching for bottlenecks when the import takes place
> from a network drive on the client side. This leads to unresponsive
> importer client and out of memory server when are importing several
> thousand images.

Definitely understood, and minimizing overhead is precisely the intent
of the new "--skip" argument in 5.1.0:

  https://www.openmicroscopy.org/site/support/omero5.1-staging/users/cli/import.html#cmdoption-omero-import--skip

If there are still any bottlenecks, the next place to look might be
the number of JVMs which are started with the logical extreme being to
not use a JVM at all. Until we provide a Python import library,
though, this amounts to writing your own and so shouldn't be attempted
if unnecessary.

All the best,
~Josh.


> Thanks,
> Paul
>
>
> On Fri, Mar 20, 2015 at 9:55 AM, Sebastien Besson (Staff)
> <s.besson at dundee.ac.uk> wrote:
>> Hi Paul,
>>
>>
>> On 19 Mar 2015, at 15:37, Paul van Schayck <paul at vanschayck.nl> wrote:
>>
>> Hi Sebastien,
>>
>> Ah thank you! Those changes look excellent. I suspect though, that
>> they are initially only available for the command line importer, and
>> not the Java client?
>>
>>
>> Yes you are right. For OMERO 5.1.0, these changes will be available to
>> the command-line interface only.
>>
>> I guess it is technically difficult to do both the upload and checksum
>> in one read, right?
>>
>>
>> Actually, the client-side checksums are calculated together with the upload
>> process
>> https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L477
>>
>> But the checksum is also calculating server-side in order to account for
>> either transfer
>> corruption and/or file-writing problems on the server:
>> https://github.com/openmicroscopy/openmicroscopy/blob/v.5.0.8/components/blitz/src/ome/formats/importer/ImportLibrary.java#L482
>>
>> So the checksum calculation effectively happens twice (client-side and
>> server-side)
>> and this operation can be computationally intensive depending on the file
>> size/number
>> and the checksum algorithm used.
>>
>> On OMERO 5.0.x, you should already be able to assess the impact of this
>> checksum
>> on your import time. Using the command-line interface, you can set the
>> checksum
>> algorithm to a different method and check the effect on import speed:
>> https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/in-place-import.html#getting-started
>>
>> Kind regards,
>>
>> Paul
>>
>>
>> Best,
>> Sebastien
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users