[ome-devel] Archiving large files - 2GB limit

[Chris Allan]

Hi Alex,

Firstly, thank you for your thorough explanation and feedback.

The issue with archiving files > 2.1GB is indeed a bug which has been filed here:

 * https://trac.openmicroscopy.org.uk/ome/ticket/9514

The team has worked on a series of fixes for the issue which are noted on the ticket and are currently in testing. We estimate that a 4.4.2 bug fix release will be made by the end of the week.

Would you be available to validate our fix against your system as well before we make that bug fix release?

Thanks again,

-Chris

On 21 Aug 2012, at 14:04, Alex Herbert wrote:

> Dear OME Team,
> 
> We currently archive all our images when importing into OMERO. Some of these images are large multi-dimensional images that can be over 2GB in size. Unfortunately this appears to the limit for archived files.
> 
> The 4GB limit for exporting pixels was addressed with fixes to the BIG-TIFF support. Images can now be imported and all pixel data viewed within OMERO. This can also be exported as OME-TIFF and all the pixel data is present.
> 
> Unfortunately during import archived files are truncated at 2GB (2147483648 bytes). This is the upper limit for a signed integer. If an image larger than this is imported the data file in the /OMERO/Files directory is limited to 2147483648 bytes. Possibly this is due to the use of integers to accumulate the number of bytes read from the file.
> 
> When the file is exported it is created using the correct byte size. However all bytes after 2147483648 are zero. This leads to black frames in our exported archive image, i.e. the export runs out of data to write and the allocated file is left blank.
> 
> This problem was present on OMERO 4.3.3. It was recently discovered by a researcher when analysing original image files and I have confirmed that we have over 200 archived images over 2GB that are truncated. Pixel data is recoverable using OME-TIFF export but the original image file is not. Consequently our current policy is to archive large images on a file store.
> 
> I have tested OMERO 4.4.1 and this truncation problem is still present. Can you confirm that this is a bug? I would like to verify that it is not a combination of our client, server and file system. I have tested using Ext3 on CentOS 4 and Ubuntu 10.04 LTS.
> 
> It is easy to replicate by generating a 16-bit Hyperstack in ImageJ using dimensions 1024x1024x2x4x320 (5GB image). An image of 1024x1024x2x4x192 (3GB) would also be truncated but obviously cannot be used to verify the >4GB export limit on OME-TIFF.
> 
> Regards,
> 
> Alex
> 
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel