[ome-users] Importing Metamorph stacks is slow with OMERO-5.0.0

Mon Apr 7 13:41:56 BST 2014

Hi,

Apologies,
I tested an Nd2 file instead of a ND file.

Test summary,
File Type : ND.
Dimensions (XY): 512 * 512
Z-sections/timepoints : 3 * 2
Channels : 1*2
File size : (15.4 MB on disk)

Yes I was able to reproduce the issue,
2014-04-07 13:25:58,818 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
(3-thread-4) Plane:1 X:0 Y:440 TileWidth:512 TileHeight:8
arrayBuf.length:8192
2014-04-07 13:25:58,820 INFO  [       loci.formats.in.MinimalTiffReader]
(3-thread-4) Reading IFDs
2014-04-07 13:25:58,822 INFO  [       loci.formats.in.MinimalTiffReader]
(3-thread-4) Populating metadata
2014-04-07 13:25:58,826 INFO  [          loci.formats.in.BaseTiffReader]
(3-thread-4) Populating OME metadata
2014-04-07 13:25:58,828 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
(3-thread-4) Plane:1 X:0 Y:448 TileWidth:512 TileHeight:8
arrayBuf.length:8192
2014-04-07 13:25:58,830 INFO  [       loci.formats.in.MinimalTiffReader]
(3-thread-4) Reading IFDs
2014-04-07 13:25:58,832 INFO  [       loci.formats.in.MinimalTiffReader]
(3-thread-4) Populating metadata
2014-04-07 13:25:58,838 INFO  [          loci.formats.in.BaseTiffReader]
(3-thread-4) Populating OME metadata

So the whole upload took totally,
24 seconds.

Even though the error is reproducible, the delay in upload is not scaling.
It would be great to have the files as previously suggested.

Best,
Balaji

On 07/04/2014 12:11, "Balaji Ramalingam" <b.ramalingam at dundee.ac.uk> wrote:

>Hi,
>
>Thank you for reporting your bug.
>
>I tested an Nd2 file locally,
>Dimensions (XY) : 1268 * 1024
>Z-sections/timepoints : 1*31
>
>And it got uploaded in ~7 seconds (when the data was sitting locally).
>
>Log information,
>
>o.s.blitz.repo.ManagedImportRequestI] (3-thread-3) Plane:0 X:0 Y:0
>TileWidth:1268 TileHeight:1024 arrayBuf.length:2596864
>2014-04-07 11:57:24,060 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:1 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,155 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:2 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,247 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:3 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,340 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:4 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,430 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:5 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,523 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:6 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,613 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:7 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,705 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:8 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,798 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:9 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,888 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:10 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:24,980 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:11 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,070 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:12 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,162 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:13 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,252 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:14 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,344 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:15 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,436 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:16 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,526 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:17 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,618 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:18 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,708 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:19 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,800 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:20 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,890 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:21 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:25,982 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:22 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,073 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:23 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,163 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:24 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,255 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:25 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,345 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:26 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,437 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:27 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,527 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:28 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,619 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:29 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>2014-04-07 11:57:26,710 DEBUG [    o.s.blitz.repo.ManagedImportRequestI]
>(3-thread-3) Plane:30 X:0 Y:0 TileWidth:1268 TileHeight:1024
>arrayBuf.length:2596864
>
>
>
>Unfortunately I don’t see the exception with the file that we have, i.e
>the TileWidth is read correctly and thus the import is faster.
>
>It would be great if you could upload the files to the following link,
>http://qa.openmicroscopy.org.uk/qa/upload/
>this will help us narrow down the issue and help you better.
>
>Best,
>Balaji
>
>
>
>
>On 05/04/2014 19:00, "Damien Goutte-Gattat"
><d.goutte-gattat at iecb.u-bordeaux.fr> wrote:
>
>>Dear All,
>>
>>After upgrading to OMERO-5.0.0, I have found that importing Metamorph ND
>>files took a very long time.
>>
>>For instance, importing a stack made of 3 channels, 12 Z-slices and 32
>>T-frames took more than 90 minutes. Importing the same kind of stack
>>used to take no more than a few minutes with OMERO-4.4.9.
>>
>>In comparison, on the same server importing a Leica LIF file containing
>>2 channels, 12 Z-slices and 50 T-frames only took a couple of minutes.
>>
>>As most of my images are acquired using Metamorph, this slow import into
>>OMERO is very inconvenient, and any help to reduce the import time would
>>be much appreciated.
>>
>>I can provide the ND file if needed (570 MB), as well as the log file
>>generated by OMERO when importing it (72 MB).
>>
>>Sincerely,
>>
>>Damien
>>
>>
>>P.S.: I don’t know if that may help or if it is even relevant, but I
>>found many occurences of the following lines in the log file:
>>
>>---8<------
>>Plane:0 X:0 Y:0 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>Plane:0 X:0 Y:8 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>Plane:0 X:0 Y:16 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>[...]
>>Plane:0 X:0 Y:496 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>Plane:1 X:0 Y:0 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>Plane:1 X:0 Y:8 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>>--->8-------
>>
>>Whereas in the log file generated when importing the Leica file, I found
>>something like that:
>>
>>---8<------
>>Plane:0 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>>Plane:1 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>>Plane:2 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>>--->8------
>>
>>I am not sure I really understand what’s going on here, but to me it
>>looks like that for the Metamorph file, each plane (of size 510×504) is
>>read in 63 blocks of 510×8 pixels, while for the Leica file, each plane
>>(of size 512×512) is read in a single block.
>>
>>Could this explain why importing the Metamorph file is dramatically
>>slower than importing the Leica file?
>>
>>--
>>Damien Goutte-Gattat        Institut Européen de Chimie et Biologie
>>                            2 rue Robert Escarpit
>>Tel: +33-5-4000-2951        33607 Pessac, FRANCE
>>E-mail: d.goutte-gattat at iecb.u-bordeaux.fr
>>
>

The University of Dundee is a registered Scottish Charity, No: SC015096