[ome-users] LZW-compression not working for large tiles

[Melissa Linkert]

Hi Matthias,

Thank you very much for reporting this.

> I have developed a small Java-tool which converts a JPEG-image, stored in a NDPI-file, to an OME-TIFF. It also works for images with an edge length larger than 65536 pixels. Therefore, I read the image using tiles.

I would be very interested in knowing how you are reading the images
from the NDPI file.  Are you using a particular library for reading the
JPEG data?

> With large JPEG-files the automatically calculated tile size can reach 3968x3968 pixels (or maybe even more). I use the writer-methods provided by Bio-Formats to save the image as a tiled OME-TIFF. The problem: When using the provided LZW-compression, a NegativeArraySizeException exception is thrown by the LZWCodec-class when the tile size reaches a certain level.  I think the reason for this exception is an integer-overflow in the LZWCodec-class, when calculating the size of the output buffer in the following line (at the beginning of the compress-method):

You're absolutely right - the size of that buffer is being calculated
incorrectly.  I have put in a fix here:

https://github.com/melissalinkert/bioformats/commit/0966339a3a5ba930f43735bb540069d046d6d616

which seems to work correctly with your test case and various other tests
that I have done here.  Note that new builds are not yet available, but you
can try it out by building from this branch:

https://github.com/melissalinkert/bioformats/tree/sprint1.2-bug-fixes

Regards,
-Melissa

On Thu, Aug 02, 2012 at 04:46:19PM +0200, Matthias Baldauf wrote:
> Hello,
> 
> I have developed a small Java-tool which converts a JPEG-image, stored in a NDPI-file, to an OME-TIFF. It also works for images with an edge length larger than 65536 pixels. Therefore, I read the image using tiles. With large JPEG-files the automatically calculated tile size can reach 3968x3968 pixels (or maybe even more). I use the writer-methods provided by Bio-Formats to save the image as a tiled OME-TIFF. The problem: When using the provided LZW-compression, a NegativeArraySizeException exception is thrown by the LZWCodec-class when the tile size reaches a certain level.  I think the reason for this exception is an integer-overflow in the LZWCodec-class, when calculating the size of the output buffer in the following line (at the beginning of the compress-method):
> 
> byte[] output = new byte[(input.length * 141) / 100 + 3];
> 
> Thrown exception:
> Exception in thread "main" java.lang.NegativeArraySizeException
>     at loci.formats.codec.LZWCodec.compress(LZWCodec.java:117)
>     at loci.formats.tiff.TiffCompression.compress(TiffCompression.java:328)
>     at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:371)
>     at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:265)
>     at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:191)
>     at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:197)
>     at LZWTest.main(LZWTest.java:63)
> 
> I am using the latest trunk build of Bio-Formats.
> 
> Below, I added an example-code (LZWTest.java) to reproduce this exception.
> Example settings:
> Tile size: 2250 x 2250 pixels, 3 components, 8bit per component (= works)
> Tile size: 2300 x 2300 pixels, 3 components, 8bit per component (= NegativeArraySizeException)
> 
> Thanks in advance for having a look at it!
> 
> Regards,
> Matthias
> 
> 
> ------------------------------------------------
> LZWTest.java
> ------------------------------------------------
> import loci.common.services.ServiceFactory;
> import loci.formats.FormatTools;
> import loci.formats.MetadataTools;
> import loci.formats.meta.IMetadata;
> import loci.formats.out.OMETiffWriter;
> import loci.formats.out.TiffWriter;
> import loci.formats.services.OMEXMLService;
> import loci.formats.tiff.IFD;
> import loci.formats.tiff.TiffCompression;
> 
> public class LZWTest {
>   
>   static final int PIXEL_TYPE = FormatTools.UINT8;
>   static final int COMPONENTS = 3;
>   static final int w = 2300;
>   static final int h = 2300;
>   
>   public static void main(String[] args) throws Exception {
>     if (args.length < 1) {
>       System.out.println("Please specify an output file name.");
>       System.exit(1);
>     }
>     String id = args[0];
> 
>     // create blank image
>     System.out.println("Creating random image...");
>     byte[] img = new byte[w * h * COMPONENTS * FormatTools.getBytesPerPixel(PIXEL_TYPE)];
>     System.out.println("Image Size in Bytes: " + img.length);
>     
>     /*
>      * Wrong calculated size of output buffer because of integer overflow
>      * Taken from LZWCodec.java - compress-method
>      */
>     System.out.println("Wrong LZW Buffer Size: " + (img.length * 141) / 100 + 3);
>     
>     // This would be the right size for the output buffer
>     System.out.println("Right LZW Buffer Size: " + img.length * 1.41 + 3);
>     
>     // fill image with random data
>     for (int i=0; i<img.length; i++) img[i] = (byte) (256 * Math.random());
> 
>     // create metadata object with minimum required metadata fields
>     System.out.println("Populating metadata...");
>     ServiceFactory factory = new ServiceFactory();
>     OMEXMLService service = factory.getInstance(OMEXMLService.class);
>     IMetadata meta = service.createOMEXMLMetadata();    
>     MetadataTools.populateMetadata(meta, 0, null, false, "XYZCT", FormatTools.getPixelTypeString(PIXEL_TYPE), w, h, 1, COMPONENTS, 1, COMPONENTS);
>    
>     // Set up the writer for OME-TIFF
>     TiffWriter writer = new OMETiffWriter();
>     writer.setMetadataRetrieve(meta);
>     writer.setInterleaved(Boolean.TRUE);
>     writer.setBigTiff(false);
>     writer.setId(id);
> 
>     // Set up IFD for creating a tiled OME-TIFF
>     IFD ifd = new IFD();
>     ifd.put(IFD.TILE_WIDTH, w);
>     ifd.put(IFD.TILE_LENGTH, h);
>     ifd.put(IFD.COMPRESSION, TiffCompression.LZW.getCode());
>     
>     // Save tile
>     writer.saveBytes(0, img, ifd, 0, 0, w, h);
>     
>     writer.close();
>   }
> 
> }
> 

> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users