[ome-users] Importing Metamorph stacks is slow with OMERO-5.0.0
Balaji Ramalingam
b.ramalingam at dundee.ac.uk
Mon Apr 7 12:11:32 BST 2014
Hi,
Thank you for reporting your bug.
I tested an Nd2 file locally,
Dimensions (XY) : 1268 * 1024
Z-sections/timepoints : 1*31
And it got uploaded in ~7 seconds (when the data was sitting locally).
Log information,
o.s.blitz.repo.ManagedImportRequestI] (3-thread-3) Plane:0 X:0 Y:0
TileWidth:1268 TileHeight:1024 arrayBuf.length:2596864
2014-04-07 11:57:24,060 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:1 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,155 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:2 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,247 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:3 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,340 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:4 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,430 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:5 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,523 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:6 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,613 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:7 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,705 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:8 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,798 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:9 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,888 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:10 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:24,980 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:11 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,070 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:12 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,162 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:13 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,252 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:14 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,344 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:15 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,436 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:16 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,526 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:17 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,618 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:18 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,708 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:19 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,800 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:20 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,890 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:21 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:25,982 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:22 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,073 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:23 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,163 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:24 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,255 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:25 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,345 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:26 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,437 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:27 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,527 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:28 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,619 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:29 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
2014-04-07 11:57:26,710 DEBUG [ o.s.blitz.repo.ManagedImportRequestI]
(3-thread-3) Plane:30 X:0 Y:0 TileWidth:1268 TileHeight:1024
arrayBuf.length:2596864
Unfortunately I don’t see the exception with the file that we have, i.e
the TileWidth is read correctly and thus the import is faster.
It would be great if you could upload the files to the following link,
http://qa.openmicroscopy.org.uk/qa/upload/
this will help us narrow down the issue and help you better.
Best,
Balaji
On 05/04/2014 19:00, "Damien Goutte-Gattat"
<d.goutte-gattat at iecb.u-bordeaux.fr> wrote:
>Dear All,
>
>After upgrading to OMERO-5.0.0, I have found that importing Metamorph ND
>files took a very long time.
>
>For instance, importing a stack made of 3 channels, 12 Z-slices and 32
>T-frames took more than 90 minutes. Importing the same kind of stack
>used to take no more than a few minutes with OMERO-4.4.9.
>
>In comparison, on the same server importing a Leica LIF file containing
>2 channels, 12 Z-slices and 50 T-frames only took a couple of minutes.
>
>As most of my images are acquired using Metamorph, this slow import into
>OMERO is very inconvenient, and any help to reduce the import time would
>be much appreciated.
>
>I can provide the ND file if needed (570 MB), as well as the log file
>generated by OMERO when importing it (72 MB).
>
>Sincerely,
>
>Damien
>
>
>P.S.: I don’t know if that may help or if it is even relevant, but I
>found many occurences of the following lines in the log file:
>
>---8<------
>Plane:0 X:0 Y:0 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>Plane:0 X:0 Y:8 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>Plane:0 X:0 Y:16 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>[...]
>Plane:0 X:0 Y:496 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>Plane:1 X:0 Y:0 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>Plane:1 X:0 Y:8 TileWidth:510 TileHeight:8 arrayBuf.length:8160
>--->8-------
>
>Whereas in the log file generated when importing the Leica file, I found
>something like that:
>
>---8<------
>Plane:0 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>Plane:1 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>Plane:2 X:0 Y:0 TileWidth:512 TileHeight:512 arrayBuf.length:262144
>--->8------
>
>I am not sure I really understand what’s going on here, but to me it
>looks like that for the Metamorph file, each plane (of size 510×504) is
>read in 63 blocks of 510×8 pixels, while for the Leica file, each plane
>(of size 512×512) is read in a single block.
>
>Could this explain why importing the Metamorph file is dramatically
>slower than importing the Leica file?
>
>--
>Damien Goutte-Gattat Institut Européen de Chimie et Biologie
> 2 rue Robert Escarpit
>Tel: +33-5-4000-2951 33607 Pessac, FRANCE
>E-mail: d.goutte-gattat at iecb.u-bordeaux.fr
>
The University of Dundee is a registered Scottish Charity, No: SC015096
More information about the ome-users
mailing list