[ome-devel] Proposed tweak to µManager data files in 2.0

[Sebastien Besson (Staff)]

Hi,

First and foremost, thanks Chris for raising this discussion publicly and for all the input received so far. From our (OME) end, reading about the community requirements, expectations and limitations is extremely instructive.

Below are some comments about various points raised along the thread:

- For images stored across multiple OME-TIFF files, storing partial OME-XML metadata and using a companion OME-XML file are both fully valid from the point of view of the specification. Both storage mechanisms are completely supported by the rest of the OME stack, especially Bio-Formats and OMERO.

- OME-TIFF datasets can be stored using different file layouts (single vs multiple files, BigTIFF, partial metadata...). Various groups across our community have opted for different structures for storing large datasets. Using their experience to document the pros and cons of each design is critical and we will be working on our OME-TIFF format documentation to reflect the assumptions and implications of each decision.

-  While some core metadata should be expressed using the OME Model specification (XYZCT dimensions, pixel sizes, etc), it is also possible to use structured annotations to define additional metadata in a flexible way [1]. In particular, the MapAnnotation concept introduced in the 2015-01 version of the OME Model provides a mechanism for storing label and value metadata as a key-value pairs.  MapAnnotations have been supported throughout the OME stack in Bio-Formats and OMERO releases since April 2015. For the purposes of this discussion, this means that any hardware parameters or configurations created by MM can be stored in standard OME-XML, read in Bio-Formats, etc.  You can see an example of this flexible metadata in the right panel on http://idr-demo.openmicroscopy.org/webclient/?show=image-109846. A description of the concepts and usage was published last year in http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4773399/.

- On the performance aspect, there are probably several issues related to large datasets to cover here. The MM team has correctly raised the issue of requiring a solution for updating metadata after file opening. We also need to engage with them to precisely define and address bottlenecks during the writing of large multi-file OME-TIFF datasets. Also, the general issue of caching metadata, whether at the reader level (as we have done for .nd2 and others in Bio-Formats [2]), in OMERO or other applications is worth discussing.

[1] http://www.openmicroscopy.org/site/support/ome-model/developers/structured-annotations.html
[2] http://blog.openmicroscopy.org/file-formats/data-model/future-plans/2015/01/13/bf-update/

Best,
Sebastien

On 13 Apr 2016, at 18:26, Chris Weisiger <cweisiger at msg.ucsf.edu<mailto:cweisiger at msg.ucsf.edu>> wrote:

Hello all,

After discussion with some of our users and with the OME developers, I'm creating this thread to invite comments on a proposed minor change to µManager's "image stack file" storage format, in µManager 2.0 only (1.4 would not be changed). The "image stack file" format (which is the default format) stores multiple image planes in the same TIFF file. This change would be to the situation where multiple of these multi-plane TIFF files are generated in a single dataset.

µManager writes multiple TIFF files in two situations: first, if there is more than 4GB of data to store (as our TIFFs don't yet support BigTIFF), and second, if the user has elected to store data for each stage position in a separate file. Currently, the OME XML is written to a single one of these files, and the other files are "pointed" to that file so they know where to look to load the data in the XML. Thus in order to load any file in the dataset, you must have not only that file, but also the TIFF file that contains the OME XML.

The proposal is to instead write the XML to a newly-created file (tentatively named "OMEXMLMetadata.companion.ome") that would exist in the same directory as the TIFF files, display settings, comments, and metadata.txt, as appropriate. As before, the TIFF files would be "pointed" to this file; it would need to be present to use the BioFormats Importer or any other OME library loader.

In situations where there is only one TIFF file, the OME XML would still be stored in that TIFF file; the "OMEXMLMetadata.companion.ome" file would not be generated. For reference, the XML would look something like this:

http://pastebin.com/cTzcQwEZ

I've tested this change locally and haven't detected any problems, but I'm not familiar with the kinds of workflows that our users may want to perform, so any comments are welcome. Thanks for your time.

-Chris Weisiger and the µManager dev team
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel


The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160418/2c6a8d93/attachment.html>