[ome-users] Import of larger 2D OME-TIFF fails

[Chris Allan]

Hi Mattias,

On 27 Sep 2012, at 14:41, Matthias Baldauf wrote:

> Hello all,
> 
> I am having problems with the import of larger 2D OME-TIFFs. The OME-TIFF is a multipage TIFF (three pages, one for each color-channel of the RGB-image), striped with 1 row per strip, 16384x9600 pixels, 450 MB, and 8bit, uncompressed.
> 
> For the import I use OMERO.importer 4.4.4 and the OMERO.server Virtual Appliance 4.4.4 with 4 GB RAM assigned. Importer and server are running on the same machine. The Host OS is Windows 7 64bit, 8 GB RAM and i7-2720QM CPU.
> It takes about three minutes to analyze the OME-TIFF. The first two steps ("1/3" & "2/3") seem to work but import fails after 13 minutes during the last step with the following exception (the complete exception is attached as a text file to this email):
> 
> omero.InternalException
>    serverStackTrace = "ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
>                        Java heap space
> 
> So it seems the server is running out of memory, but I can see in the Task Manager that the 4 GB RAM of the server are never used completely during the import. The OMERO.importer has only a slight change in RAM usage for about 30 MB. The image is not a really large one, therefore I am wondering why the import fails?
> I thought that this OME-TIFF format would be ideal for import because when I export small images in OMERO.insight the resulting OME-TIFF has the same structure. I also tried to import a tiled OME-TIFF (one RGB-image, 16384x9600 pixels, 456 MB, and 8bit, uncompressed) but import results in the same exception at step "3/3". The import of a smaller OME-TIFF (10.000 x 5.000 pixels, same settings as the multipage tiff) works and takes about six minutes.

Would it be possible to make these files available so that we might be able to fun some tests locally? I'm fairly certain that the default memory settings for the actual OMERO.server itself are 512MB on the virtual appliance. Details on working with these settings are available here:

http://www.openmicroscopy.org/site/support/omero4/sysadmins/troubleshooting.html#outofmemoryerror-permgen-space-errors-in-omero-server-logs

> 
> 
> I also have some additional questions about the import process:
> 
> Why does the analyze-step and the further steps take about 13 minutes in total until the import fails? The OME-TIFF is uncompressed and already split into single colors (one page for each color-channel) so I thought the import would be faster.

I would think so too but we'd have to see the files.

> Am I right, that OMERO.importer reads the OME-TIFF section after section, so the whole OME-TIFF is not loaded into the RAM?

Correct. It should in fact only read strip after strip.

> Does the OMERO.server need much RAM for the construction of an image pyramid and does OMERO.server need much RAM after the import (e.g. when viewing large images with 60.000 x 40.000 pixels)? I would like to import a few thousand large 2D images and view them.

It needs enough ram to read the smallest tile into memory and compress it. Obviously this is highly dependent on the source image.

> Which criterion is decisive for not processing the image on client-side but uploading the raw files directly to the server?

The size criteria. By default this is (sizeX * sizeY) > (3192 * 3192)

> 
> What are the three import steps? Is the step "1/3" the reading and splitting into the three color-channels

No. It is just reading the metadata.

> , step "2/3" the data-import on the server

This is saving the metadata to the server.

> and "3/3" the construction of the image pyramid?

This is the process of copying data to the server directly in to the image pyramid.

> Which steps are done locally and which steps are done on the server?

All steps are done locally.

> 
> How is the concept of reading images during the import? Is it like: some image formats need to be loaded completely into the RAM in order to read, decompress and import them and other formats can be read section per section resulting in a low RAM usage?

Correct. It all comes down to the way the source data is laid out.

> 
> 
> Thank you in advance for your answers!
> 
> Regards,
> Matthias


-Chris