[ome-users] slow or failing import of large files with large X by Y size

[Damir Sudar]

Hi Melissa,

Thanks much for your quick response and resolution of my queries.

Indeed, if I save file3.tif using the BioFormats exporter in Fiji, it 
imports into OMERO without problems. Do I understand correctly that the 
B-F exporter produces a file with the large XY slice already tiled into 
bite-sized chunks while my original file3.tif was a monolithic XY slice?

On the difference in import times of the 2 .lsm files: yes, upon trying 
again a few times, I also get about a 10 minute import time with 
file1.lsm (I must have been doing something wrong the previous times. 
Sorry about that.) However, it still surprises me that a 100MB file 
(file1.lsm) takes about 10 times as long to import than a 200MB file 
(file2.lsm). Is that just a consequence of creating the tiles for the 
big-image display?

Thanks and have a great weekend,
- Damir

On 2/26/2013 11:31 PM, Melissa Linkert wrote:
> Hi Damir,
>
>> Thanks for the quick response. I've just uploaded the 3 files
>> described below. I'm mostly concerned at this point about the import
>> failure of file3.tif.
> Thank you for uploading the problematic files.
>
> It looks like there are two distinct problems: one is that file3.tif
> is not viewable (which I can reproduce), and the other is the time
> required to import .lsm files (which I cannot reproduce).
>
> For the file3.tif problem, increasing the amount of memory available to
> the server should help.  It looks like that file was saved using ImageJ;
> for the future, you may wish to use the Bio-Formats exporter plugin to
> save TIFFs from ImageJ, as that will save TIFFs in such a way as to
> prevent this problem without requiring more memory.
>
> For the .lsm import times, I consistently see that file1.lsm imports in
> 10-12 minutes (also on Ubuntu 12.04 with Ice 3.4).  Are you able to
> consistently reproduce that that file takes 30+ minutes?  If so, could
> you please send a copy of the server logs so that we can investigate
> further?
>
> Regards,
> -Melissa
>
> On Mon, Feb 25, 2013 at 12:24:40PM -0800, Damir Sudar wrote:
>> Hi Petr,
>>
>> Thanks for the quick response. I've just uploaded the 3 files
>> described below. I'm mostly concerned at this point about the import
>> failure of file3.tif.
>>
>> Thanks,
>> - Damir
>> On 2/25/2013 1:50 AM, Petr Walczysko wrote:
>>> Hello Damir
>>>
>>> Thank you very much for your feedback.
>>> Could you please upload the files in question using out QA ?
>>> Please go to
>>> http://qa.openmicroscopy.org.uk/qa/upload/
>>> and follow the instructions.
>>> It would help us to investigate the problem.
>>> Thank you
>>> All the best
>>> Petr
>>>
>>>
>>>
>>> On 23 Feb 2013, at 01:16, Damir Sudar <dsudar at lbl.gov
>>> <mailto:dsudar at lbl.gov>>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> On my installation (Omero 4.4.5/Ice 3.4 on Ubuntu 12.04LTS
>>>> 64-bit) I see the following strange import (using
>>>> Insight/Importer 4.4.5) behavior:
>>>> File1.lsm: 102MB is 5120 by 5120 pixels in 2 channels takes
>>>> 30-40 minutes to import
>>>> File2.lsm: 207MB  is 1024 by 1024 pixels by 50 Z-slices takes 1
>>>> minute to import
>>>> File3.tif: 100MB  is 10240 by 10240 pixels simple 8-bit TIFF,
>>>> import finishes in a few seconds without errors but the
>>>> resulting image in Omero will not open and gives error: "The
>>>> image could not be opened. The image is not a valid image." But
>>>> the archived version can be downloaded without trouble.
>>>>
>>>> So there appears to be something wrong with importing image with
>>>> large X by Y sizes. Has anyone seen this as well and is there a
>>>> solution?
>>>>
>>>> Thanks,
>>>> - Damir
>>>>
>>>> -- 
>>>> Damir Sudar - Staff Scientist and Deputy for Technology
>>>> Lawrence Berkeley Laboratory / Life Sciences Division
>>>> One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
>>>> T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
>>>> <mailto:DSudar at lbl.gov>
>>>> WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> <mailto:ome-users at lists.openmicroscopy.org.uk>
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No:
>>> SC015096
>> -- 
>> Damir Sudar - Staff Scientist and Deputy for Technology
>> Lawrence Berkeley Laboratory / Life Sciences Division
>> One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
>> T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
>> WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

-- 
Damir Sudar - Staff Scientist and Deputy for Technology
Lawrence Berkeley Laboratory / Life Sciences Division
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
WWW: http://www.lbl.gov/lsd/People_&_Organization/Scientific_Staff_Directory/Sudar_Lab.html