[ome-devel] Proposed tweak to µManager data files in 2.0

[Chris Weisiger]

Thank you, everyone, for your feedback so far. Keep it coming! I do want to
highlight a few points and provide some clarifications.

There seems to be fairly broad support for the idea of generating an
external OME XML file. It sounds in general like you would prefer us to
either *always* generate the OME XML externally (i.e. as a separate
dedicated file), or *never* do so, which seems fair enough. Also, if the
file is externally-generated, then it should ideally have a name tied to
the TIFF filenames, in case users mix datasets together.

Note for those considering parsing these files themselves: we also
currently have the option to always generate a metadata.txt file, which is
a JSON file that contains complete metadata (including the state of every
device in the scope) for every image in the dataset. It will be much more
complete than the OME XML. Of course, if you already have code for parsing
OME XML, or want to use the OME libraries, then parsing the XML instead
would be easier.

We are not at this time considering making major structural changes to our
data files. That is definitely a project we want to tackle in the future,
but it is going to require serious thinking, because we want to have a file
format that we can continue to use for the next decade at least. Every time
we change our file format, we break users' workflows, so it's not something
we would do lightly (of course, we will continue to support existing file
formats as well). When the New µManager File Format project (NµFF :)) does
start, you can expect there to be some significant discussion on these
forums to make certain that we're not missing important use cases.

That said, feel free to continue to brainstorm! I'm listening to your
suggestions.

-Chris

On Wed, Apr 13, 2016 at 10:26 AM, Chris Weisiger <cweisiger at msg.ucsf.edu>
wrote:

> Hello all,
>
> After discussion with some of our users and with the OME developers, I'm
> creating this thread to invite comments on a proposed minor change to
> µManager's "image stack file" storage format, in µManager 2.0 only (1.4
> would not be changed). The "image stack file" format (which is the default
> format) stores multiple image planes in the same TIFF file. This change
> would be to the situation where multiple of these multi-plane TIFF files
> are generated in a single dataset.
>
> µManager writes multiple TIFF files in two situations: first, if there is
> more than 4GB of data to store (as our TIFFs don't yet support BigTIFF),
> and second, if the user has elected to store data for each stage position
> in a separate file. Currently, the OME XML is written to a single one of
> these files, and the other files are "pointed" to that file so they know
> where to look to load the data in the XML. Thus in order to load any file
> in the dataset, you must have not only that file, but also the TIFF file
> that contains the OME XML.
>
> The proposal is to instead write the XML to a newly-created file
> (tentatively named "OMEXMLMetadata.companion.ome") that would exist in the
> same directory as the TIFF files, display settings, comments, and
> metadata.txt, as appropriate. As before, the TIFF files would be "pointed"
> to this file; it would need to be present to use the BioFormats Importer or
> any other OME library loader.
>
> In situations where there is only one TIFF file, the OME XML would still
> be stored in that TIFF file; the "OMEXMLMetadata.companion.ome" file would
> not be generated. For reference, the XML would look something like this:
>
> http://pastebin.com/cTzcQwEZ
>
> I've tested this change locally and haven't detected any problems, but I'm
> not familiar with the kinds of workflows that our users may want to
> perform, so any comments are welcome. Thanks for your time.
>
> -Chris Weisiger and the µManager dev team
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160414/b92fbe70/attachment.html>