[ome-devel] Proposed tweak to µManager data files in 2.0

[Curtis Rueden]

Hi Chris,

> You mention one reason for this below, which is that people tend to
> lose the companion file. Are there other reasons to avoid doing this?

People tend to lose it, and also to edit it. There are a certain class of
people who would never ever edit it, for fear of breaking something. But
then there are a class of users who do these sorts of things, and then get
confused when things stop working.

A few years back, LOCI explored the idea of allowing edits to the companion
file to "take precedence" over what is in the embedded OME-XML, as long as
the former was well-formed, but that was part of a now-long-abandoned "OME
Metadata Editor" project. Nothing like that ever became standard across the
whole suite of OME tools.

One of OME's founding principles for data storage was that your whole
dataset—pixels and metadata—should be stored in one single file, to
minimize the chance of data loss. But that attitude has certainly evolved a
lot over the past 15 years, and these days I don't think anyone is too
militant about it. I just wanted to mention that historically it has been
an issue in practice. But I guess Micro-Manager has plenty of
evidence/experience with its metadata.txt file: how often do people lose
that? etc.

> We also cannot rely on being able to embed the XML in every TIFF

Yeah, I wouldn't think you should embed it in _every_ TIFF. But probably
more than one would be nice, in case the "lynchpin" TIFF gets lost.

> we may butt up against the 4GB limit

Heh, it might not be a big deal, since the limit is not actually 4GB... but
rather 32-bit offset pointer and 32-bit length, yeah? So as long as the
OME-XML is appended to the very end of the file, and _starts_ before the
4GB cutoff, and is less than another 4GB in length (highly likely ;-), you
should be OK? Worth testing anyway.

Regards,
Curtis


--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Wed, Apr 13, 2016 at 1:49 PM, Chris Weisiger <cweisiger at msg.ucsf.edu>
wrote:

> Hi Curtis,
>
> First I just want to say thank you very much for the detailed breakdown.
> It's great to get insight into the design processes that other groups have
> gone through, to help guide our own design decisions and avoid reinventing
> wheels. In the long term, we're going to be taking a close, hard look at
> how we deal with files and what we ought to be doing differently; there are
> things we want to be able to do that the current format is not well-suited
> for, in particular dealing with high-speed acquisitions and handling data
> whose parameters can't be known prior to starting writing. That's all
> matters for another day, however.
>
> On Wed, Apr 13, 2016 at 11:16 AM, Curtis Rueden <ctrueden at wisc.edu> wrote:
>
>> Hi Chris,
>>
>> TL;DR: Embed the OME-XML into one or more OME-TIFF headers. Do not rely
>> solely on a companion file.
>>
>
> You mention one reason for this below, which is that people tend to lose
> the companion file. Are there other reasons to avoid doing this?
>
> In µManager currently, there is already only one file that contains the
> OME-XML, and that file must be present if users are to load their data. So
> effectively, we already have one "companion file", that happens to be
> bundled together with image data. If users lose the special TIFF that
> contains the XML, then they will be unable to load their data.
>
> One of the goals of this change is to make it easier for users to separate
> out individual subsets of their data. For example, if a user does a
> 96-well-plate acquisition and wants to be able to look at a single well,
> currently they must retain the entire dataset, or else figure out which of
> the TIFF files contains the XML; otherwise that single well's data cannot
> be loaded.
>
> Please do correct me if I'm wrong, but doesn't the BinaryOnly system only
> allow you to point to a single source for the "external" XML? Thus if we
> rely on XML embedded in the TIFF headers, we cannot use the external file
> as a fallback, and thus are unable to separate a single TIFF from the
> dataset as a whole.
>
> We also cannot rely on being able to embed the XML in every TIFF, as we
> may butt up against the 4GB limit, and the XML cannot be written until
> after acquisition is completed. Currently we write the XML to the tail end
> of one of the TIFFs (that has enough room under the 4GB limit), and then
> modify the headers of each TIFF to point to that location.
>
> -Chris
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160413/e5bbb9b0/attachment.html>