<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi,<br class="">
<br class="">
First and foremost, thanks Chris for raising this discussion publicly and for all the input received so far. From our (OME) end, reading about the community requirements, expectations and limitations is extremely instructive.<br class="">
<br class="">
Below are some comments about various points raised along the thread:<br class="">
<br class="">
<div class="">- For images stored across multiple OME-TIFF files, storing partial OME-XML metadata and using a companion OME-XML file are both fully valid from the point of view of the specification. Both storage mechanisms are completely supported by the rest
of the OME stack, especially Bio-Formats and OMERO.<br class="">
<br class="">
</div>
<div class="">- OME-TIFF datasets can be stored using different file layouts (single vs multiple files, BigTIFF, partial metadata...). Various groups across our community have opted for different structures for storing large datasets. Using their experience
to document the pros and cons of each design is critical and we will be working on our OME-TIFF format documentation to reflect the assumptions and implications of each decision.<br class="">
<br class="">
</div>
<div class="">- While some core metadata should be expressed using the OME Model specification (XYZCT dimensions, pixel sizes, etc), it is also possible to use structured annotations to define additional metadata in a flexible way [1]. In particular, the MapAnnotation
concept introduced in the 2015-01 version of the OME Model provides a mechanism for storing label and value metadata as a key-value pairs. MapAnnotations have been supported throughout the OME stack in Bio-Formats and OMERO releases since April 2015. For
the purposes of this discussion, this means that any hardware parameters or configurations created by MM can be stored in standard OME-XML, read in Bio-Formats, etc. You can see an example of this flexible metadata in the right panel on <a href="http://idr-demo.openmicroscopy.org/webclient/?show=image-109846" class="">http://idr-demo.openmicroscopy.org/webclient/?show=image-109846</a>.
A description of the concepts and usage was published last year in <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4773399/" class="">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4773399/</a>. <br class="">
<br class="">
</div>
<div class="">- On the performance aspect, there are probably several issues related to large datasets to cover here. The MM team has correctly raised the issue of requiring a solution for updating metadata after file opening. We also need to engage with them
to precisely define and address bottlenecks during the writing of large multi-file OME-TIFF datasets. Also, the general issue of caching metadata, whether at the reader level (as we have done for .nd2 and others in Bio-Formats [2]), in OMERO or other applications
is worth discussing. </div>
<br class="">
[1] <a href="http://www.openmicroscopy.org/site/support/ome-model/developers/structured-annotations.html" class="">http://www.openmicroscopy.org/site/support/ome-model/developers/structured-annotations.html</a>
<div class=""><span class="">[2] <a href="http://blog.openmicroscopy.org/file-formats/data-model/future-plans/2015/01/13/bf-update/" class="">http://blog.openmicroscopy.org/file-formats/data-model/future-plans/2015/01/13/bf-update/</a></span></div>
<div class=""><span class=""><br class="">
</span><span class="">Best,<br class="">
</span><span class="">Sebastien</span>
<div class=""><font face="Arial" class=""><span style="font-size: 15px;" class=""><br class="">
</span></font>
<div>
<blockquote type="cite" class="">
<div class="">On 13 Apr 2016, at 18:26, Chris Weisiger <<a href="mailto:cweisiger@msg.ucsf.edu" class="">cweisiger@msg.ucsf.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hello all,<br class="">
<br class="">
</div>
<div class="">After discussion with some of our users and with the OME developers, I'm creating this thread to invite comments on a proposed minor change to µManager's "image stack file" storage format, in µManager 2.0 only (1.4 would not be changed). The "image
stack file" format (which is the default format) stores multiple image planes in the same TIFF file. This change would be to the situation where multiple of these multi-plane TIFF files are generated in a single dataset.<br class="">
<br class="">
</div>
<div class="">µManager writes multiple TIFF files in two situations: first, if there is more than 4GB of data to store (as our TIFFs don't yet support BigTIFF), and second, if the user has elected to store data for each stage position in a separate file. Currently,
the OME XML is written to a single one of these files, and the other files are "pointed" to that file so they know where to look to load the data in the XML. Thus in order to load any file in the dataset, you must have not only that file, but also the TIFF
file that contains the OME XML.<br class="">
<br class="">
</div>
<div class="">The proposal is to instead write the XML to a newly-created file (tentatively named "OMEXMLMetadata.companion.ome") that would exist in the same directory as the TIFF files, display settings, comments, and metadata.txt, as appropriate. As before,
the TIFF files would be "pointed" to this file; it would need to be present to use the BioFormats Importer or any other OME library loader.<br class="">
<br class="">
</div>
<div class="">In situations where there is only one TIFF file, the OME XML would still be stored in that TIFF file; the "OMEXMLMetadata.companion.ome" file would not be generated. For reference, the XML would look something like this:<br class="">
<br class="">
<a href="http://pastebin.com/cTzcQwEZ" class="">http://pastebin.com/cTzcQwEZ</a><br class="">
<br class="">
</div>
<div class="">I've tested this change locally and haven't detected any problems, but I'm not familiar with the kinds of workflows that our users may want to perform, so any comments are welcome. Thanks for your time.<br class="">
<br class="">
</div>
<div class="">-Chris Weisiger and the µManager dev team<br class="">
</div>
</div>
_______________________________________________<br class="">
ome-devel mailing list<br class="">
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" class="">ome-devel@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>