<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Thanks again Curtis
<div><br>
</div>
<div>I’ll try that next week.</div>
<div><br>
</div>
<div>Ian<br>
<div><br>
</div>
<div><br>
<div>
<div>On 15 Aug 2014, at 20:28, Curtis Rueden <<a href="mailto:ctrueden@wisc.edu">ctrueden@wisc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">Hi Ian,
<div><br>
</div>
<div>
<div>> My understanding was that there is, currently, no reader or writer for</div>
<div>> this in bio-formats. I’d be happy to be wrong. </div>
</div>
<div><br>
</div>
<div>I did a quick test with a two-file OME-TIFF dataset, where the first file has the full block, and the second file uses a BinaryOnly reference to the first file as described on the website [1]. Bio-Formats was able to read the full set of metadata and detected
the correct dimensions. So I think it works. There is also certainly some logic in the Bio-Formats source code to handle it [2, 3].</div>
<div><br>
</div>
<div>On the other hand, investigating recent commit messages surrounding this logic reveals a major inconsistency. One commit message [4] states:</div>
<div><br>
</div>
<div> "Some Prairie datasets contain 'OME-TIFF' files with a BinaryOnly entry that points to a different 'OME-TIFF', which is not valid."</div>
<div><br>
</div>
<div>And another [5] strongly implies that the metadata file must be OME-XML, not OME-TIFF.</div>
<div><br>
</div>
<div>But the OME-TIFF Specification [1] gives an OME-TIFF as the example MetadataFile. So I think the original intent of the BinaryOnly element was lost somewhere along the way.</div>
<div><br>
</div>
<div>Anyway, in practice it seems the BinaryOnly approach should be viable for you.<br>
</div>
<div><br>
</div>
<div>-Curtis</div>
<div><br>
</div>
<div>[1] <a href="https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/specification.html#storing-partial-metadata-blocks">
https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/specification.html#storing-partial-metadata-blocks</a><br>
</div>
<div>[2] <a href="https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L203-L206">https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L203-L206</a></div>
<div>[3] <a href="https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L421-L450">https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L421-L450</a></div>
<div>[4] <a href="https://github.com/openmicroscopy/bioformats/commit/c4a81ff2903fbfb32820f1eca43aa83788f1cd58">
https://github.com/openmicroscopy/bioformats/commit/c4a81ff2903fbfb32820f1eca43aa83788f1cd58</a></div>
<div>[5] <a href="https://github.com/openmicroscopy/bioformats/commit/7cbe988ebaf2ff81792a56489c7ea311d8af25bc">https://github.com/openmicroscopy/bioformats/commit/7cbe988ebaf2ff81792a56489c7ea311d8af25bc</a></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Aug 15, 2014 at 1:47 PM, Munro, Ian <span dir="ltr">
<<a href="mailto:i.munro@imperial.ac.uk" target="_blank">i.munro@imperial.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Hi Curtis</div>
<div><br>
</div>
My understanding was that there is, currently, no reader or writer for this in bio-formats.
<div>I’d be happy to be wrong. </div>
<span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div>Ian</div>
</font></span>
<div>
<div class="h5">
<div><br>
</div>
<div> <br>
<div>
<div>On 15 Aug 2014, at 18:23, Curtis Rueden <<a href="mailto:ctrueden@wisc.edu" target="_blank">ctrueden@wisc.edu</a>> wrote:</div>
<br>
<blockquote type="cite">
<p dir="ltr">Hi Ian,</p>
<p dir="ltr">I like option 5, storing the metadata into only a subset of the TIFFs. LOCI has been doing this for a long time now and it works very well. Not sure what you mean about that approach not yet being available.</p>
<p dir="ltr">-Curtis<br>
</p>
<div class="gmail_quote">On Aug 15, 2014 9:24 AM, "Munro, Ian" <<a href="mailto:i.munro@imperial.ac.uk" target="_blank">i.munro@imperial.ac.uk</a>> wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>A bit of background first.</div>
<div><br>
</div>
As you perhaps know the main driver for our involvement in OMERO is the need to share data & make some data
<div>publicly available.</div>
<div><br>
</div>
<div>To achieve this we have been modifying our acquisition software to generate ome-tiffs which can then be imported into OMERO.</div>
<div>We have also modified FLIMfit to allow it to read ome-tiffs.</div>
<div>This is in order to be able to use the “download original files” functionality in OMERO.web which wasn’t compatible with our earlier approach </div>
<div>of writing data directly to OMERO via the OMERO api.</div>
<div><br>
</div>
<div>We now have software in place to write all our data types as ome-tiff with one exception.</div>
<div>We now need to decide how best to handle FLIM plate data.</div>
<div><br>
</div>
<div>As a first step<span style="font-size:11px"> have now succeeded in writing Java code to create a dummy SPW plate and write it into a Fileset.</span></div>
<div><span style="font-size:11px"><br>
</span></div>
<div><a href="https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352" target="_blank">https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352</a></div>
<div><br>
</div>
<div>Our original plan was to have one ome-tiff per FOV.</div>
<div><span style="font-size:11px"><br>
</span></div>
<div>However If I’ve written the above code correctly (?) then the issue is, OME-XML’ bloat.’ </div>
<div>The metadata for the entire plate is required to be in each file of the Fileset therefore the total amount of metadata rises with the number of files.</div>
<div style="margin:0px">Guesstimating the size of the metadata by linearly scaling from my dummy plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.</div>
<div style="margin:0px">Where typical’ is assumed to be 80 wells x 5 FOVs/well x 5 time-points/FOV i.e. 400 FLIMages.</div>
<div style="margin:0px">For comparison the data stored as a stack of tiffs would occupy 140Mb.</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px;min-height:13px">We therefore have a range of options & we need your input to select one that allows the data to be downloaded ( when this capability becomes available).</div>
<div style="margin:0px;min-height:13px">see <a href="https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client" target="_blank">https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client</a></div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px;min-height:13px">We note that a solution is in place for JCB</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">Some of our options are:</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">1) Don’t bother with OME SPW data at all.Simply store the plate data in a dataset</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Estimated ‘bloat’ 0bytes.</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Pros: Could simplify the FLIMfit code by removing all Screen/plate menu items.</div>
<div style="margin:0px"> FLIMfit can already read this data by parsing the filename.</div>
<div style="margin:0px"> A hypothetical public downloader (MOP) would get a single file per FOV (easy for them to interpret).</div>
<div style="margin:0px"> Code already in place to generate ome-tiffs.</div>
<div style="margin:0px"> “Original file download" already works!</div>
<div style="margin:0px"> <span style="white-space:pre-wrap"> </span>Cons: Data doesn’t display as a plate in OMERO clients.</div>
<div style="margin:0px"> Any future plate tools won’t work.</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">2) Put the whole plate in a small number (say 1-4) files.</div>
<div style="margin:0px"> Estimated ‘bloat’ 1.2 - 4.8 Mb ( <2% of the data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Pros: Data will import into OMERO without problem as a plate Minimal bloat.</div>
<div style="margin:0px"> <span style="white-space:pre-wrap"> </span>Cons: FLIMfit needs modification to use SPW metadata rather than filename. </div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Big! files.</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span> A MOP will get huge ome-tiffs</div>
<div style="margin:0px"> They will also need to sort out the metadata to work out what goes where.</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">3) Have one file per FOV. </div>
<div style="margin:0px"> Estimated ‘bloat’ 480Mb (350% of the data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"> Pros: No modification required to FLIMfit ( each file is a FOV as now). </div>
<div style="margin:0px"> Will import to OMERO as a plate.</div>
<div style="margin:0px"> Cons: bloat!</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">4) Have one file per well.</div>
<div style="margin:0px"> Estimated ‘bloat’ 96Mb (70% of the data size stored as ‘stack of tiffs)</div>
<div style="margin:0px"> Pros: Data will import into OMERO without problem as a plate.</div>
<div style="margin:0px"> Cons: FLIMfit needs modification.</div>
<div style="margin:0px"> ‘Bloat’ still large<span style="white-space:pre-wrap">
</span> </div>
<div style="margin:0px"> Non-trivial for hypothetical MOP.</div>
<div style="margin:0px"> Although easier than 2) as they can determine the well from the Filename </div>
<div style="margin:0px;min-height:13px"> <br>
</div>
<div style="margin:0px">5) Use BinaryOnly ome-tiff format. </div>
<div style="margin:0px"> [ In this format there is only one copy of the metadata. all the other files just contain a reference to this. “master’ file. ]</div>
<div style="margin:0px"> Estimated ‘bloat’ 1.2Mb ( < 1 % of data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"> Pros: No mods required to FLIMfit (one file per FOV) although using the SPW metadata is a desirable feature.. Minimal ‘bloat’</div>
<div style="margin:0px"> Cons: Not yet available. </div>
<div style="margin:0px"> </div>
<div style="margin:0px">6) Store data as ome-tiffs with no SPW xml but add this info in OMERO after import (either via dataset-to-plate script or the API)</div>
<div style="margin:0px"> Estimated bloat. Unclear -ask OME team</div>
<div style="margin:0px"> Pros. Might allow one file per FOV & work with future plate tools.</div>
<div style="margin:0px"> Cons: Might not work - ask OME team</div>
<div style="margin:0px"> Big software effort and end of project is approaching rapidly.</div>
<div style="margin:0px"> </div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">Thanks in advance</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">Ian</div>
<div style="margin:0px;min-height:14px"> <br>
</div>
<div style="margin:0px;min-height:14px"><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
<br>
</blockquote>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>