<p dir="ltr">Hi Ian,</p>
<p dir="ltr">I like option 5, storing the metadata into only a subset of the TIFFs. LOCI has been doing this for a long time now and it works very well. Not sure what you mean about that approach not yet being available.</p>
<p dir="ltr">-Curtis<br></p>
<div class="gmail_quote">On Aug 15, 2014 9:24 AM, "Munro, Ian" <<a href="mailto:i.munro@imperial.ac.uk">i.munro@imperial.ac.uk</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>A bit of background first.</div>
<div><br>
</div>
As you perhaps know the main driver for our involvement in OMERO is the need to share data & make some data
<div>publicly available.</div>
<div><br>
</div>
<div>To achieve this we have been modifying our acquisition software to generate ome-tiffs which can then be imported into OMERO.</div>
<div>We have also modified FLIMfit to allow it to read ome-tiffs.</div>
<div>This is in order to be able to use the “download original files” functionality in OMERO.web which wasn’t compatible with our earlier approach </div>
<div>of writing data directly to OMERO via the OMERO api.</div>
<div><br>
</div>
<div>We now have software in place to write all our data types as ome-tiff with one exception.</div>
<div>We now need to decide how best to handle FLIM plate data.</div>
<div><br>
</div>
<div>As a first step<span style="font-size:11px"> have now succeeded in writing Java code to create a dummy SPW plate and write it into a Fileset.</span></div>
<div><span style="font-size:11px"><br>
</span></div>
<div><a href="https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352" target="_blank">https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352</a></div>
<div><br>
</div>
<div>Our original plan was to have one ome-tiff per FOV.</div>
<div><span style="font-size:11px"><br>
</span></div>
<div>However If I’ve written the above code correctly (?) then the issue is, OME-XML’ bloat.’ </div>
<div>The metadata for the entire plate is required to be in each file of the Fileset therefore the total amount of metadata rises with the number of files.</div>
<div style="margin:0px">Guesstimating the size of the metadata by linearly scaling from my dummy plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.</div>
<div style="margin:0px">Where typical’ is assumed to be 80 wells x 5 FOVs/well x 5 time-points/FOV i.e. 400 FLIMages.</div>
<div style="margin:0px">For comparison the data stored as a stack of tiffs would occupy 140Mb.</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px;min-height:13px">We therefore have a range of options & we need your input to select one that allows the data to be downloaded ( when this capability becomes available).</div>
<div style="margin:0px;min-height:13px">see <a href="https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client" target="_blank">https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client</a></div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px;min-height:13px">We note that a solution is in place for JCB</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">Some of our options are:</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">1) Don’t bother with OME SPW data at all.Simply store the plate data in a dataset</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Estimated ‘bloat’ 0bytes.</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Pros: Could simplify the FLIMfit code by removing all Screen/plate menu items.</div>
<div style="margin:0px"> FLIMfit can already read this data by parsing the filename.</div>
<div style="margin:0px"> A hypothetical public downloader (MOP) would get a single file per FOV (easy for them to interpret).</div>
<div style="margin:0px"> Code already in place to generate ome-tiffs.</div>
<div style="margin:0px"> “Original file download" already works!</div>
<div style="margin:0px"> <span style="white-space:pre-wrap"> </span>
Cons: Data doesn’t display as a plate in OMERO clients.</div>
<div style="margin:0px"> Any future plate tools won’t work.</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">2) Put the whole plate in a small number (say 1-4) files.</div>
<div style="margin:0px"> Estimated ‘bloat’ 1.2 - 4.8 Mb ( <2% of the data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Pros: Data will import into OMERO without problem as a plate Minimal bloat.</div>
<div style="margin:0px"> <span style="white-space:pre-wrap"> </span>
Cons: FLIMfit needs modification to use SPW metadata rather than filename. </div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span>Big! files.</div>
<div style="margin:0px"><span style="white-space:pre-wrap"></span> A MOP will get huge ome-tiffs</div>
<div style="margin:0px"> They will also need to sort out the metadata to work out what goes where.</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">3) Have one file per FOV. </div>
<div style="margin:0px"> Estimated ‘bloat’ 480Mb (350% of the data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"> Pros: No modification required to FLIMfit ( each file is a FOV as now). </div>
<div style="margin:0px"> Will import to OMERO as a plate.</div>
<div style="margin:0px"> Cons: bloat!</div>
<div style="margin:0px;min-height:13px"><br>
</div>
<div style="margin:0px">4) Have one file per well.</div>
<div style="margin:0px"> Estimated ‘bloat’ 96Mb (70% of the data size stored as ‘stack of tiffs)</div>
<div style="margin:0px"> Pros: Data will import into OMERO without problem as a plate.</div>
<div style="margin:0px"> Cons: FLIMfit needs modification.</div>
<div style="margin:0px"> ‘Bloat’ still large<span style="white-space:pre-wrap">
</span> </div>
<div style="margin:0px"> Non-trivial for hypothetical MOP.</div>
<div style="margin:0px"> Although easier than 2) as they can determine the well from the Filename </div>
<p style="margin:0px;min-height:13px"> <br>
</p>
<div style="margin:0px">5) Use BinaryOnly ome-tiff format. </div>
<div style="margin:0px"> [ In this format there is only one copy of the metadata. all the other files just contain a reference to this. “master’ file. ]</div>
<div style="margin:0px"> Estimated ‘bloat’ 1.2Mb ( < 1 % of data size stored as ‘stack of tiffs’)</div>
<div style="margin:0px"> Pros: No mods required to FLIMfit (one file per FOV) although using the SPW metadata is a desirable feature.. Minimal ‘bloat’</div>
<div style="margin:0px"> Cons: Not yet available. </div>
<div style="margin:0px"> </div>
<div style="margin:0px">6) Store data as ome-tiffs with no SPW xml but add this info in OMERO after import (either via dataset-to-plate script or the API)</div>
<div style="margin:0px"> Estimated bloat. Unclear -ask OME team</div>
<div style="margin:0px"> Pros. Might allow one file per FOV & work with future plate tools.</div>
<div style="margin:0px"> Cons: Might not work - ask OME team</div>
<div style="margin:0px"> Big software effort and end of project is approaching rapidly.</div>
<div style="margin:0px"> </div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">Thanks in advance</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">Ian</div>
<p style="margin:0px;min-height:14px"> <br>
</p>
<div style="margin:0px;min-height:14px"><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
<br></blockquote></div>