[ome-users] slow or failing import of large files with large X by Y size

Mon Mar 4 14:52:25 GMT 2013

Hi Damir,

> Indeed, if I save file3.tif using the BioFormats exporter in Fiji,
> it imports into OMERO without problems. Do I understand correctly
> that the B-F exporter produces a file with the large XY slice
> already tiled into bite-sized chunks while my original file3.tif was
> a monolithic XY slice?

Yes, that's correct.  For the next release of OMERO, we will do what we
can to better support files such as file3.tif, but in general having
large images stored as many small tiles is preferable.

> On the difference in import times of the 2 .lsm files: yes, upon
> trying again a few times, I also get about a 10 minute import time
> with file1.lsm (I must have been doing something wrong the previous
> times. Sorry about that.) However, it still surprises me that a
> 100MB file (file1.lsm) takes about 10 times as long to import than a
> 200MB file (file2.lsm). Is that just a consequence of creating the
> tiles for the big-image display?

It's unlikely that you were doing anything wrong; without seeing the
server logs it's difficult to say for sure, but it's very possible that
there was some sort of connection issue at work.

The difference between file1.lsm and file2.lsm is down to the size of
each plane.  Importing files with one or more large image planes is
almost always slower than importing large files with small planes, as
OMERO needs to calculate tiles for each of the zoom levels for large
planes.

Regards,
-Melissa

On Fri, Mar 01, 2013 at 02:18:08PM -0800, Damir Sudar wrote:
> Hi Melissa,
> 
> Thanks much for your quick response and resolution of my queries.
> 
> Indeed, if I save file3.tif using the BioFormats exporter in Fiji,
> it imports into OMERO without problems. Do I understand correctly
> that the B-F exporter produces a file with the large XY slice
> already tiled into bite-sized chunks while my original file3.tif was
> a monolithic XY slice?
> 
> On the difference in import times of the 2 .lsm files: yes, upon
> trying again a few times, I also get about a 10 minute import time
> with file1.lsm (I must have been doing something wrong the previous
> times. Sorry about that.) However, it still surprises me that a
> 100MB file (file1.lsm) takes about 10 times as long to import than a
> 200MB file (file2.lsm). Is that just a consequence of creating the
> tiles for the big-image display?
> 
> Thanks and have a great weekend,
> - Damir
> 
> On 2/26/2013 11:31 PM, Melissa Linkert wrote:
> >Hi Damir,
> >
> >>Thanks for the quick response. I've just uploaded the 3 files
> >>described below. I'm mostly concerned at this point about the import
> >>failure of file3.tif.
> >Thank you for uploading the problematic files.
> >
> >It looks like there are two distinct problems: one is that file3.tif
> >is not viewable (which I can reproduce), and the other is the time
> >required to import .lsm files (which I cannot reproduce).
> >
> >For the file3.tif problem, increasing the amount of memory available to
> >the server should help.  It looks like that file was saved using ImageJ;
> >for the future, you may wish to use the Bio-Formats exporter plugin to
> >save TIFFs from ImageJ, as that will save TIFFs in such a way as to
> >prevent this problem without requiring more memory.
> >
> >For the .lsm import times, I consistently see that file1.lsm imports in
> >10-12 minutes (also on Ubuntu 12.04 with Ice 3.4).  Are you able to
> >consistently reproduce that that file takes 30+ minutes?  If so, could
> >you please send a copy of the server logs so that we can investigate
> >further?
> >
> >Regards,
> >-Melissa
> >
> >On Mon, Feb 25, 2013 at 12:24:40PM -0800, Damir Sudar wrote:
> >>Hi Petr,
> >>
> >>Thanks for the quick response. I've just uploaded the 3 files
> >>described below. I'm mostly concerned at this point about the import
> >>failure of file3.tif.
> >>
> >>Thanks,
> >>- Damir
> >>On 2/25/2013 1:50 AM, Petr Walczysko wrote:
> >>>Hello Damir
> >>>
> >>>Thank you very much for your feedback.
> >>>Could you please upload the files in question using out QA ?
> >>>Please go to
> >>>http://qa.openmicroscopy.org.uk/qa/upload/
> >>>and follow the instructions.
> >>>It would help us to investigate the problem.
> >>>Thank you
> >>>All the best
> >>>Petr
> >>>
> >>>
> >>>
> >>>On 23 Feb 2013, at 01:16, Damir Sudar <dsudar at lbl.gov
> >>><mailto:dsudar at lbl.gov>>
> >>>wrote:
> >>>
> >>>>Hi all,
> >>>>
> >>>>On my installation (Omero 4.4.5/Ice 3.4 on Ubuntu 12.04LTS
> >>>>64-bit) I see the following strange import (using
> >>>>Insight/Importer 4.4.5) behavior:
> >>>>File1.lsm: 102MB is 5120 by 5120 pixels in 2 channels takes
> >>>>30-40 minutes to import
> >>>>File2.lsm: 207MB  is 1024 by 1024 pixels by 50 Z-slices takes 1
> >>>>minute to import
> >>>>File3.tif: 100MB  is 10240 by 10240 pixels simple 8-bit TIFF,
> >>>>import finishes in a few seconds without errors but the
> >>>>resulting image in Omero will not open and gives error: "The
> >>>>image could not be opened. The image is not a valid image." But
> >>>>the archived version can be downloaded without trouble.
> >>>>
> >>>>So there appears to be something wrong with importing image with
> >>>>large X by Y sizes. Has anyone seen this as well and is there a
> >>>>solution?
> >>>>
> >>>>Thanks,
> >>>>- Damir
> >>>>
> >>>>-- 
> >>>>Damir Sudar - Staff Scientist and Deputy for Technology
> >>>>Lawrence Berkeley Laboratory / Life Sciences Division
> >>>>One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
> >>>>T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
> >>>><mailto:DSudar at lbl.gov>
> >>>>WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html
> >>>>
> >>>>_______________________________________________
> >>>>ome-users mailing list
> >>>>ome-users at lists.openmicroscopy.org.uk
> >>>><mailto:ome-users at lists.openmicroscopy.org.uk>
> >>>>http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> >>>>
> >>>
> >>>The University of Dundee is a registered Scottish Charity, No:
> >>>SC015096
> >>-- 
> >>Damir Sudar - Staff Scientist and Deputy for Technology
> >>Lawrence Berkeley Laboratory / Life Sciences Division
> >>One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
> >>T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
> >>WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html
> >>
> >>_______________________________________________
> >>ome-users mailing list
> >>ome-users at lists.openmicroscopy.org.uk
> >>http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> 
> -- 
> Damir Sudar - Staff Scientist and Deputy for Technology
> Lawrence Berkeley Laboratory / Life Sciences Division
> One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
> T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
> WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html
> 