[ome-users] LZW-compression not working for large tiles

[Melissa Linkert]

Hi Matthias,

> > I would be very interested in knowing how you are reading the images
> > from the NDPI file.  Are you using a particular library for reading
> the
> > JPEG data?
> 
> I am using no external library for reading the JPEG data.
> A self-developed class analyzes (using java.io.RandomAccessFile) the
> NDPI-File and the including JPEG headers.
> To prevent decoding the whole image at once, I use the JPEG Restart
> Markers, which are present in every larger JPEG in the NDPI-file.
> The offsets of all Restart Markers are stored in a Tiff-tag in the
> corresponding Tiff Image File Directory.
> The tool processes large JPEG images by tiling the image into many
> smaller valid JPEG images.
> To put it simply, the conversion is done by performing the following
> four steps:
*snip*

Thank you for the explanation, and the pointer to your project.

> Will the LZW fix be included in the trunk build of Bio-Formats in the
> near future?

Yes, it just needs to be reviewed by at least one other developer first.

> The NDPI-converter was the preliminary work for a Java web-client for
> OMERO.server I am currently working on.
> In this context further questions raised. May I ask you to help me with
> these questions?
> (I am using the current 4.4.1 release of Bio-Formats, OMERO.clients and
> OMERO.server virtual appliance)
> 
> 1) Regarding the NDPI-converter tool:
> I want to store some information of the NDPI-file as original metadata
> in the OME-XML.
> Therefore, I am using a XMLAnnotation with KeyValue as value.
> The code for this can be found in the source-code of my tool:
> FileConverter.java, beginning at line 174.
> But, there is no "Original Metadata" button in the "Acquisition" tab
> when looking at the image in OMERO.insight after importing it with
> OMERO.importer.
> OMERO.server stores the XMLAnnotations correctly, but I thought that
> XMLAnnotations, when containing the OriginalMetadata-tag, are handled as
> original metadata?
> I am not sure if I have to use these XML-tags or if I have to create a
> file "original_metadata.txt" and add it to the OME-XML by using
> FileAnnotation?

What you're doing now is fine, the only issue is that Bio-Formats'
OME-TIFF reader doesn't correctly pick up the XMLAnnotations and pass
them into OMERO.  There is a ticket for that filed here:

http://trac.openmicroscopy.org.uk/ome/ticket/9457

> 2) Regarding OMERO.insight & OMERO.server:
> I imported some OME-TIFFs, created by my tool, using OMERO.insight.
> When I want to export them as OME-TIFF in OMERO.insight it seems that
> exporting only works for images up to a certain width and height.
> Exporting an image with a dimension of 4096x2400 worked, but the "Export
> as OME-TIFF" button is deactivated for an image with a dimension of
> 4352x2368.
> Furthermore, the import of the larger image takes much longer
> (34
> seconds) than the smaller one (9 seconds).

> When viewing This is the error (in the error.log file) when I use “Download” in the
> displayed saving options: “Image:12 is too large for export (sizeX=4352,
> sizeY=2368)”
> Is the export as OME-TIFF only working for images up to a certain width
> and height?
> If true, is there a possibility to export larger images as OME-TIFF by
> using the OMERO Java library?

That is expected, as per this ticket:

http://trac.openmicroscopy.org.uk/ome/ticket/6713

Images that qualify as "big images" (by default, width * height > 3192 * 3192)
cannot be exported at all at the moment as far as I can see.  Note that
the definition of what qualifies as a big image can be configured by
changing the omero.pixeldata.max_plane_width and
omero.pixeldata.max_plane_height properties in the OMERO server's
etc/omero.properties file.

Regards,
-Melissa

On Sat, Aug 04, 2012 at 11:47:24PM +0200, Matthias Baldauf wrote:
> Hello Melissa,
> 
> thank you for the quick response and the fix of the problem!
> 
> 
> > I would be very interested in knowing how you are reading the images
> > from the NDPI file.  Are you using a particular library for reading
> the
> > JPEG data?
> 
> I am using no external library for reading the JPEG data.
> A self-developed class analyzes (using java.io.RandomAccessFile) the
> NDPI-File and the including JPEG headers.
> To prevent decoding the whole image at once, I use the JPEG Restart
> Markers, which are present in every larger JPEG in the NDPI-file.
> The offsets of all Restart Markers are stored in a Tiff-tag in the
> corresponding Tiff Image File Directory.
> The tool processes large JPEG images by tiling the image into many
> smaller valid JPEG images.
> To put it simply, the conversion is done by performing the following
> four steps:
> 
> 1) By looking at the Sampling Factor and the Restart Interval of the
> JPEG, the width and height of one decodable image block between two
> Restart Markers can be determined.
> Basically, this image block is decodable when it is combined with a
> modified JPEG header and the JPEG EOF-marker.
> 
> 2) Because the width of a single image-block is much larger than the
> height, multiple image-blocks are read at different positions in the
> original image, to form a square image tile.
> In combination with the original JPEG header (containing the width and
> height of the image tile) and the EOF-marker, an image tile (= valid
> JPEG image) gets produced.
> 
> 3) Decoding of the image tile is done by using libjpeg-turbo library.
> I am using libjpeg-turbo because many tiles need to be decoded and
> libjpeg-turbo is much faster than the standard Java implementation.
> 
> 4) After this decoding step, methods provided in the Bio-Formats package
> are used to save the image tile in a tiled OME-TIFF.
> The steps of creating, decoding and saving an image tile are done
> sequentially for every image tile.
> Therefore, only one image-tile (compressed and decompressed) has to be
> kept in memory at a time.
> 
> This was a very simplified explanation of the process to convert a large
> (and therefore maybe invalid JPEG) to a tiled OME-TIFF.
> You can find more information, a download of the tool and the source
> code on my website: http://matthias-baldauf.at/software/ndpi_converter/
> I updated the tool today - it includes the fix for the LZW compression
> :-)
> Will the LZW fix be included in the trunk build of Bio-Formats in the
> near future?
> 
> 
> The NDPI-converter was the preliminary work for a Java web-client for
> OMERO.server I am currently working on.
> In this context further questions raised. May I ask you to help me with
> these questions?
> (I am using the current 4.4.1 release of Bio-Formats, OMERO.clients and
> OMERO.server virtual appliance)
> 
> 1) Regarding the NDPI-converter tool:
> I want to store some information of the NDPI-file as original metadata
> in the OME-XML.
> Therefore, I am using a XMLAnnotation with KeyValue as value.
> The code for this can be found in the source-code of my tool:
> FileConverter.java, beginning at line 174.
> But, there is no "Original Metadata" button in the "Acquisition" tab
> when looking at the image in OMERO.insight after importing it with
> OMERO.importer.
> OMERO.server stores the XMLAnnotations correctly, but I thought that
> XMLAnnotations, when containing the OriginalMetadata-tag, are handled as
> original metadata?
> I am not sure if I have to use these XML-tags or if I have to create a
> file "original_metadata.txt" and add it to the OME-XML by using
> FileAnnotation?
> 
> 2) Regarding OMERO.insight & OMERO.server:
> I imported some OME-TIFFs, created by my tool, using OMERO.insight.
> When I want to export them as OME-TIFF in OMERO.insight it seems that
> exporting only works for images up to a certain width and height.
> Exporting an image with a dimension of 4096x2400 worked, but the "Export
> as OME-TIFF" button is deactivated for an image with a dimension of
> 4352x2368.
> Furthermore, the import of the larger image takes much longer
> (34
> seconds) than the smaller one (9 seconds).
> When viewing This is the error (in the error.log file) when I use “Download” in the
> displayed saving options: “Image:12 is too large for export (sizeX=4352,
> sizeY=2368)”
> Is the export as OME-TIFF only working for images up to a certain width
> and height?
> If true, is there a possibility to export larger images as OME-TIFF by
> using the OMERO Java library?
> 
> 
> Thank you so much for your help!
> 
> 
> Regards,
> Matthias
>