[ome-users] can ome files OMETIFFWriter compress embedded xml?

[Roger Leigh]

On 23/04/18 11:15, Mario Emmenlauer wrote:
> On 23.04.2018 11:31, Roger Leigh wrote:
[...]
>> The primary concern with adding such support would be breaking
>> of the existing specification, that is compressed metadata would not be
>> readable by existing readers in compliance with the OME-TIFF specification¹
>>
>> An existing approach to reducing space wastage is the use of a companion
>> file² so that the metadata is placed in a separate XML file and the TIFF
>> files only have a small bit of XML referencing it.  If you're using
>> large multi-file datasets, this will work today and is what I'd
>> recommend using in the interim if possible.
>>
>> Metadata compression could likely be done by compression and
>> base64-encoding of the ImageDescription content.  If using a companion
>> file, it could be directly compressed without the base64 encoding
>> (companion.ome.xml.gz|bz2).  I have created a Trello card³ to track
>> this.  However, I should stress that because this would require careful
>> investigation due to the significant compatibility break it introduces,
>> it is unlikely we would be able to work on this in the near-term.  Other
>> formats we are investigating, such as HDF5, offer transparent
>> compression and as we add support for them we can add support for
>> metadata compression from the start, without any compatibility concerns.
>
> Thanks for this very detailed explanation! I share your concerns about
> the backwards compatibility. But on the other hand it could help us with
> the adoption of OME TIFF if we could already foresee a more space-
> efficient solution down the road. My understanding is that the number
> of options is quite limited: I can think of only three or four ways how
> to implement compressed meta data in OME TIFF. Do you think it would be
> possible to continue our discussion and come up with a reasonable proposal
> over the next 6 months? Your estimate would certainly help me a lot!

Dear Mario,

The options are indeed limited, though some are less invasive to
implement.  For example, supporting compression of the companion XML
file would not introduce any changes to the TIFF, though readers would
be required to handle decompression of the companion and it would
therefore still have compatibility concerns.  We would also need to add
proper writing support for companion files (they currently need writing
by hand or with other tools).

Our priorities with OME-TIFF are currently the addition of support for
sub-resolution pyramids to the OME-TIFF reader and writer, and following
that we are also working on support for other binary containers such as
HDF5, BDV and KLB.  I would expect these to take priority due to
existing commitments, including funding commitments.  While the addition
of compression support to OME-TIFF has a great deal of merit, we do need
to deliver these other features first, and so we would not be able to
commit to delivering TIFF metadata compression in the near term.
However, this would not preclude working on the specification for it,
nor for anyone else to work on the feature if they wished to contribute it.

If you are attending the annual users meeting in Dundee, this is
certainly something which we can discuss in more detail then.
Otherwise, this list is fine.


Kind regards,
Roger

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096