[ome-devel] Proposed tweak to µManager data files in 2.0

[Chris Weisiger]

Hi Curtis,

First I just want to say thank you very much for the detailed breakdown.
It's great to get insight into the design processes that other groups have
gone through, to help guide our own design decisions and avoid reinventing
wheels. In the long term, we're going to be taking a close, hard look at
how we deal with files and what we ought to be doing differently; there are
things we want to be able to do that the current format is not well-suited
for, in particular dealing with high-speed acquisitions and handling data
whose parameters can't be known prior to starting writing. That's all
matters for another day, however.

On Wed, Apr 13, 2016 at 11:16 AM, Curtis Rueden <ctrueden at wisc.edu> wrote:

> Hi Chris,
>
> TL;DR: Embed the OME-XML into one or more OME-TIFF headers. Do not rely
> solely on a companion file.
>

You mention one reason for this below, which is that people tend to lose
the companion file. Are there other reasons to avoid doing this?

In µManager currently, there is already only one file that contains the
OME-XML, and that file must be present if users are to load their data. So
effectively, we already have one "companion file", that happens to be
bundled together with image data. If users lose the special TIFF that
contains the XML, then they will be unable to load their data.

One of the goals of this change is to make it easier for users to separate
out individual subsets of their data. For example, if a user does a
96-well-plate acquisition and wants to be able to look at a single well,
currently they must retain the entire dataset, or else figure out which of
the TIFF files contains the XML; otherwise that single well's data cannot
be loaded.

Please do correct me if I'm wrong, but doesn't the BinaryOnly system only
allow you to point to a single source for the "external" XML? Thus if we
rely on XML embedded in the TIFF headers, we cannot use the external file
as a fallback, and thus are unable to separate a single TIFF from the
dataset as a whole.

We also cannot rely on being able to embed the XML in every TIFF, as we may
butt up against the 4GB limit, and the XML cannot be written until after
acquisition is completed. Currently we write the XML to the tail end of one
of the TIFFs (that has enough room under the 4GB limit), and then modify
the headers of each TIFF to point to that location.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160413/b3f8ffaf/attachment.html>