[ome-devel] Proposed tweak to µManager data files in 2.0

Wed Apr 13 19:16:11 BST 2016

Hi Chris,

TL;DR: Embed the OME-XML into one or more OME-TIFF headers. Do not rely
solely on a companion file.

-------------

For what it's worth, here is how we do multi-file multi-page OME-TIFF
format with WiscScan:

- We write all focal planes for a given timepoint+position+channel to a
single OME-TIFF file.
- Each timepoint, position and channel gets its own OME-TIFF file.

E.g.:
HCx19_Ant_LP3_A1_C<c>_TP<t>_SP<sp>_FW<fw>.ome.tiff

Where:
- '<c>' is the 0-indexed channel number, i.e. the OME C dimension
- '<t>' is the 0-indexed timepoint, i.e. the OME T dimension
- '<sp>' is the 0-indexed stage position, with each constituting a separate
OME Image
- '<fw>' is the 0-indexed filter wheel filter number, when imaging with
multiple filters, with each constituting a separate OME Image

So you end up with a collection of OME-TIFF files described by an OME-XML
block which may span multiple <Image> elements—one per <sp>+<fw>
combination. This scheme is extensible to any number of extra 'dimensions'
although of course the OME-XML won't retain any structural details of how
these various Image elements are related.

As for storing actual OME-XML string:

- We store the full thing into the first (0, 0, ..., 0) TIFF file
ImageDescription (as per the spec [1])

- We store the full thing again into every Nth TIFF file, where N is
configurable but typically 10

- In the other OME-TIFFs, we write an OME-XML stub using the <BinaryOnly>
element—e.g.:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><OME xmlns="
http://www.openmicroscopy.org/Schemas/OME/2012-06" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://www.openmicroscopy.org/Schemas/OME/2012-06
https://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd"><BinaryOnly
MetadataFile="HCx33_Post_Long_20xObj_x1pt5_int10_p220_g282_xystep383pt4_zstep27_C0_TP0_SP0_FW0.ome.tiff"
UUID="urn:uuid:a14af27e-c6f0-48b4-bb14-e0d4f98925f0"/></OME>

For us, this is a nice compromise between redundancy (in case one or more
TIFF files are lost) and efficiency (since the OME-XML can become several
megabytes or more in size).

We also write out a copy of the OME-XML metadata to a companion file, but
this is for the user's convenience only—it is not used or referenced from
the actual OME-TIFF files. When OME-TIFF was first invented, it was deemed
an important quality that the metadata _not_ be stored only in a separate
companion file, because users would too frequently lose or delete said
metadata file, leaving them in a bad place.

To actually do the work:

- WiscScan uses the Bio-Formats C++ bindings to build up the OME-XML.

- The OME-XML is kept in the Bio-Formats memory structure as it is being
built up.

- It is written out to the temporary companion file as the run proceeds, in
case of a crash.

- WiscScan writes the TIFF binary data using libtiff, leaving a placeholder
stub in ImageDescription for later overwriting with the OME-XML.

- When the OME-XML is finalized (i.e. when the run is complete, and all
metadata is known and populated into the data structure), then the OME-XML
is appended to the relevant TIFF file(s) efficiently, using the Bio-Formats
"overwrite comment" approach [2].

Maybe some of that helps you guys? I wish you luck with it.

Regards,
Curtis

[1]
https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/specification.html#distributing-an-image-sequence-across-multiple-ome-tiff-files
[2]
https://github.com/openmicroscopy/bioformats/blob/v5.1.8/components/formats-bsd/src/loci/formats/tiff/TiffSaver.java#L870-L875

--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/

On Wed, Apr 13, 2016 at 12:26 PM, Chris Weisiger <cweisiger at msg.ucsf.edu>
wrote:

> Hello all,
>
> After discussion with some of our users and with the OME developers, I'm
> creating this thread to invite comments on a proposed minor change to
> µManager's "image stack file" storage format, in µManager 2.0 only (1.4
> would not be changed). The "image stack file" format (which is the default
> format) stores multiple image planes in the same TIFF file. This change
> would be to the situation where multiple of these multi-plane TIFF files
> are generated in a single dataset.
>
> µManager writes multiple TIFF files in two situations: first, if there is
> more than 4GB of data to store (as our TIFFs don't yet support BigTIFF),
> and second, if the user has elected to store data for each stage position
> in a separate file. Currently, the OME XML is written to a single one of
> these files, and the other files are "pointed" to that file so they know
> where to look to load the data in the XML. Thus in order to load any file
> in the dataset, you must have not only that file, but also the TIFF file
> that contains the OME XML.
>
> The proposal is to instead write the XML to a newly-created file
> (tentatively named "OMEXMLMetadata.companion.ome") that would exist in the
> same directory as the TIFF files, display settings, comments, and
> metadata.txt, as appropriate. As before, the TIFF files would be "pointed"
> to this file; it would need to be present to use the BioFormats Importer or
> any other OME library loader.
>
> In situations where there is only one TIFF file, the OME XML would still
> be stored in that TIFF file; the "OMEXMLMetadata.companion.ome" file would
> not be generated. For reference, the XML would look something like this:
>
> http://pastebin.com/cTzcQwEZ
>
> I've tested this change locally and haven't detected any problems, but I'm
> not familiar with the kinds of workflows that our users may want to
> perform, so any comments are welcome. Thanks for your time.
>
> -Chris Weisiger and the µManager dev team
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160413/eec74eeb/attachment.html>