[ome-users] LZW-compression not working for large tiles

[Matthias Baldauf]

Hello Melissa,

thank you for the quick response and the fix of the problem!


> I would be very interested in knowing how you are reading the images
> from the NDPI file.  Are you using a particular library for reading
the
> JPEG data?

I am using no external library for reading the JPEG data.
A self-developed class analyzes (using java.io.RandomAccessFile) the
NDPI-File and the including JPEG headers.
To prevent decoding the whole image at once, I use the JPEG Restart
Markers, which are present in every larger JPEG in the NDPI-file.
The offsets of all Restart Markers are stored in a Tiff-tag in the
corresponding Tiff Image File Directory.
The tool processes large JPEG images by tiling the image into many
smaller valid JPEG images.
To put it simply, the conversion is done by performing the following
four steps:

1) By looking at the Sampling Factor and the Restart Interval of the
JPEG, the width and height of one decodable image block between two
Restart Markers can be determined.
Basically, this image block is decodable when it is combined with a
modified JPEG header and the JPEG EOF-marker.

2) Because the width of a single image-block is much larger than the
height, multiple image-blocks are read at different positions in the
original image, to form a square image tile.
In combination with the original JPEG header (containing the width and
height of the image tile) and the EOF-marker, an image tile (= valid
JPEG image) gets produced.

3) Decoding of the image tile is done by using libjpeg-turbo library.
I am using libjpeg-turbo because many tiles need to be decoded and
libjpeg-turbo is much faster than the standard Java implementation.

4) After this decoding step, methods provided in the Bio-Formats package
are used to save the image tile in a tiled OME-TIFF.
The steps of creating, decoding and saving an image tile are done
sequentially for every image tile.
Therefore, only one image-tile (compressed and decompressed) has to be
kept in memory at a time.

This was a very simplified explanation of the process to convert a large
(and therefore maybe invalid JPEG) to a tiled OME-TIFF.
You can find more information, a download of the tool and the source
code on my website: http://matthias-baldauf.at/software/ndpi_converter/
I updated the tool today - it includes the fix for the LZW compression
:-)
Will the LZW fix be included in the trunk build of Bio-Formats in the
near future?


The NDPI-converter was the preliminary work for a Java web-client for
OMERO.server I am currently working on.
In this context further questions raised. May I ask you to help me with
these questions?
(I am using the current 4.4.1 release of Bio-Formats, OMERO.clients and
OMERO.server virtual appliance)

1) Regarding the NDPI-converter tool:
I want to store some information of the NDPI-file as original metadata
in the OME-XML.
Therefore, I am using a XMLAnnotation with KeyValue as value.
The code for this can be found in the source-code of my tool:
FileConverter.java, beginning at line 174.
But, there is no "Original Metadata" button in the "Acquisition" tab
when looking at the image in OMERO.insight after importing it with
OMERO.importer.
OMERO.server stores the XMLAnnotations correctly, but I thought that
XMLAnnotations, when containing the OriginalMetadata-tag, are handled as
original metadata?
I am not sure if I have to use these XML-tags or if I have to create a
file "original_metadata.txt" and add it to the OME-XML by using
FileAnnotation?

2) Regarding OMERO.insight & OMERO.server:
I imported some OME-TIFFs, created by my tool, using OMERO.insight.
When I want to export them as OME-TIFF in OMERO.insight it seems that
exporting only works for images up to a certain width and height.
Exporting an image with a dimension of 4096x2400 worked, but the "Export
as OME-TIFF" button is deactivated for an image with a dimension of
4352x2368.
Furthermore, the import of the larger image takes much longer 
(34
seconds) than the smaller one (9 seconds).
When viewing This is the error (in the error.log file) when I use “Download” in the
displayed saving options: “Image:12 is too large for export (sizeX=4352,
sizeY=2368)”
Is the export as OME-TIFF only working for images up to a certain width
and height?
If true, is there a possibility to export larger images as OME-TIFF by
using the OMERO Java library?


Thank you so much for your help!


Regards,
Matthias