[ome-users] Import of larger 2D OME-TIFF fails

[Chris Allan]

Hi Matthias,

On 28 Sep 2012, at 23:39, Matthias Baldauf wrote:

> Hello Chris,
> 
> thank you for your fast reply!
> 
> 
>> Would it be possible to make these files available so that we might be able to fun some tests locally?
> 
> Of course, I uploaded the two OME-TIFFs for which the upload failed:
> 1) OME-TIFF, 16384 x 9600 pixels, three pages (one for each color-channel of the RGB-image), striped with 1 row per strip, 450 MB, and 8bit, uncompressed
> Link: http://dl.dropbox.com/u/4868278/test_16384x9600_3pages.ome.tif
> 2) OME-TIFF, 16384 x 9600 pixels, one RGB-image, tiled (512 x 512), 456 MB, and 8bit, uncompressed
> Link: http://dl.dropbox.com/u/4868278/test_16384x9600_singleRGBPage.ome.tif
> The OME-TIFFs are filled with random data and were created with help of the Bio-Formats library.

Thanks. We'll have a look.

> 
> 
>>> Does the OMERO.server need much RAM for the construction of an image pyramid and does OMERO.server need much RAM after the import (e.g. when viewing large images with 60.000 x 40.000 pixels)? I would like to import a few thousand large 2D images and view them.
>> It needs enough ram to read the smallest tile into memory and compress it. Obviously this is highly dependent on the source image.
> 
> Sorry for asking, but does your answer refers to the creation of the image pyramid?

Yes.

> I would like to know more about how the import of large 2D images and tiling works, because it is important for my Master Thesis. As far as I know (please correct me if I am wrong), the image data gets split in each color-channel during the import process and the image data gets stored on the server in a binary file (one file for each color-channel)?

OMERO is a planar system so each channel is treated individually, yes. Regardless of the original source data's storage format. Each "Image" in OMERO parlance is the entire dataset, spatiotemporal (X, Y, Z and T dimensions) as well as wavelength or channel (C dimension). In your case of a large 2D image in OMERO that's a single multipage TIFF with JPEG-2000 compressed tiles. This is in fact how we handle all  image pyramids.

> Is the creation of the image pyramid done before the data gets stored on the server?

No, never. Tiling into the pyramid is either:

 1) Done on the fly; each tile sent to the server one at a time where it is compressed and placed into the image pyramid file
 2) Done in the background; the original data file is sent to the server where it is processed by the PixelData-0 process and all data is placed into the image pyramid file

> Is the local import like that way: based on the source image a tile size is dynamically chosen, client reads tile and sends uncompressed tile to server, server creates six different resolution levels for this tile and stores it, client reads and sends next uncompressed tile, server creates six different resolution levels for this tile and stores it, etc. until finished?

Apart from a few select file formats, yes more or less. We also take advantage of JPEG-2000 sub-resolutions to avoid creating every single resolution level individually.

> At the end there are many binary files on the server but these files are not in a tiled format, is that correct?

Some are, some aren't. It depends on the source data size and the file format.

> In the server configuration the tile size can be defined but is this is only relevant for reading the images from the binary files and could be changed anytime?

Correct.

> It would be very nice from you if you could provide me more insight in the import and tiling concept.
> 
>> The size criteria. By default this is (sizeX * sizeY) > (3192 * 3192)
> 
> I understand - this is the Big Image criterion. However, is it right that OME-TIFFs are always processed locally regardless of this criterion? My OME-TIFFs were all processed locally.

It depends on a series of things, I'll have to look at your source data to be 100% sure of the workflow.

> 
>>> What are the three import steps? Is the step "1/3" the reading and splitting into the three color-channels
>> No. It is just reading the metadata.
>>> , step "2/3" the data-import on the server
>> This is saving the metadata to the server.
>>> and "3/3" the construction of the image pyramid?
>> This is the process of copying data to the server directly in to the image pyramid.
>>> Which steps are done locally and which steps are done on the server?
>> All steps are done locally.
> 
> Thank you for the explanation. When I import the mentioned OME-TIFFs it takes about five minutes for reading and saving the metadata, which is quite long. It seems that OMERO.server is involved, because of the CPU usage during these two steps.
> 
> 
> Thank you in advance for looking at the OME-TIFFs and for your answers!
> 
> Regards,
> Matthias
>