[ome-devel] Archiving large files - 2GB limit

[Alex Herbert]

Dear OME Team,

We currently archive all our images when importing into OMERO. Some of 
these images are large multi-dimensional images that can be over 2GB in 
size. Unfortunately this appears to the limit for archived files.

The 4GB limit for exporting pixels was addressed with fixes to the 
BIG-TIFF support. Images can now be imported and all pixel data viewed 
within OMERO. This can also be exported as OME-TIFF and all the pixel 
data is present.

Unfortunately during import archived files are truncated at 2GB 
(2147483648 bytes). This is the upper limit for a signed integer. If an 
image larger than this is imported the data file in the /OMERO/Files 
directory is limited to 2147483648 bytes. Possibly this is due to the 
use of integers to accumulate the number of bytes read from the file.

When the file is exported it is created using the correct byte size. 
However all bytes after 2147483648 are zero. This leads to black frames 
in our exported archive image, i.e. the export runs out of data to write 
and the allocated file is left blank.

This problem was present on OMERO 4.3.3. It was recently discovered by a 
researcher when analysing original image files and I have confirmed that 
we have over 200 archived images over 2GB that are truncated. Pixel data 
is recoverable using OME-TIFF export but the original image file is not. 
Consequently our current policy is to archive large images on a file store.

I have tested OMERO 4.4.1 and this truncation problem is still present. 
Can you confirm that this is a bug? I would like to verify that it is 
not a combination of our client, server and file system. I have tested 
using Ext3 on CentOS 4 and Ubuntu 10.04 LTS.

It is easy to replicate by generating a 16-bit Hyperstack in ImageJ 
using dimensions 1024x1024x2x4x320 (5GB image). An image of 
1024x1024x2x4x192 (3GB) would also be truncated but obviously cannot be 
used to verify the >4GB export limit on OME-TIFF.

Regards,

Alex