[ome-devel] SPW ome-tiffs & how to proceed?

Sat Aug 16 12:39:34 BST 2014

Thanks again Curtis

I’ll try that next week.

Ian

On 15 Aug 2014, at 20:28, Curtis Rueden <ctrueden at wisc.edu<mailto:ctrueden at wisc.edu>> wrote:

Hi Ian,

> My understanding was that there is, currently, no reader or writer for
> this in bio-formats. I’d be happy to be wrong.

I did a quick test with a two-file OME-TIFF dataset, where the first file has the full block, and the second file uses a BinaryOnly reference to the first file as described on the website [1]. Bio-Formats was able to read the full set of metadata and detected the correct dimensions. So I think it works. There is also certainly some logic in the Bio-Formats source code to handle it [2, 3].

On the other hand, investigating recent commit messages surrounding this logic reveals a major inconsistency. One commit message [4] states:

   "Some Prairie datasets contain 'OME-TIFF' files with a BinaryOnly entry that points to a different 'OME-TIFF', which is not valid."

And another [5] strongly implies that the metadata file must be OME-XML, not OME-TIFF.

But the OME-TIFF Specification [1] gives an OME-TIFF as the example MetadataFile. So I think the original intent of the BinaryOnly element was lost somewhere along the way.

Anyway, in practice it seems the BinaryOnly approach should be viable for you.

-Curtis

[1] https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/specification.html#storing-partial-metadata-blocks
[2] https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L203-L206
[3] https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L421-L450
[4] https://github.com/openmicroscopy/bioformats/commit/c4a81ff2903fbfb32820f1eca43aa83788f1cd58
[5] https://github.com/openmicroscopy/bioformats/commit/7cbe988ebaf2ff81792a56489c7ea311d8af25bc

On Fri, Aug 15, 2014 at 1:47 PM, Munro, Ian <i.munro at imperial.ac.uk<mailto:i.munro at imperial.ac.uk>> wrote:
Hi Curtis

My understanding was that there is, currently, no reader or writer for this in bio-formats.
I’d be happy to be wrong.

Ian

On 15 Aug 2014, at 18:23, Curtis Rueden <ctrueden at wisc.edu<mailto:ctrueden at wisc.edu>> wrote:

Hi Ian,

I like option 5, storing the metadata into only a subset of the TIFFs. LOCI has been doing this for a long time now and it works very well. Not sure what you mean about that approach not yet being available.

-Curtis

On Aug 15, 2014 9:24 AM, "Munro, Ian" <i.munro at imperial.ac.uk<mailto:i.munro at imperial.ac.uk>> wrote:
A bit of background first.

As you perhaps know the main driver for our involvement in OMERO is the need to share data & make some data
publicly available.

To achieve this we have been modifying our acquisition software to generate ome-tiffs which can then be imported into OMERO.
We have also modified  FLIMfit to allow it to read ome-tiffs.
This is in order to be able to use the “download original files” functionality in OMERO.web which wasn’t compatible with our earlier approach
of writing data directly to OMERO via the OMERO api.

We now have software in place to write all our data types as ome-tiff with one exception.
We now need to decide how best to handle FLIM plate data.

As a first step have now succeeded in writing Java code to create a dummy SPW plate and write it into a Fileset.

https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352

Our original plan was to have one ome-tiff per FOV.

However If I’ve written the above code  correctly  (?) then the issue  is, OME-XML’ bloat.’
The metadata for the entire plate is required to be in each file of the Fileset therefore the total amount of metadata rises with the number of files.
Guesstimating the size of the metadata by linearly scaling from my dummy plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.
Where typical’ is assumed to be  80 wells x 5 FOVs/well x 5 time-points/FOV i.e. 400 FLIMages.
For comparison the data stored as a stack of tiffs would occupy 140Mb.

We therefore have a range of options & we need your input to select one that allows the data to be downloaded ( when this capability becomes available).
see   https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client

We note that a solution is in place for JCB

Some of our options are:

1) Don’t bother with OME SPW data at all.Simply store the plate data in a dataset
Estimated  ‘bloat’ 0bytes.
Pros: Could simplify the FLIMfit code by removing all Screen/plate menu items.
                  FLIMfit can already read this data by parsing the filename.
                  A hypothetical public downloader (MOP) would get a single file per FOV (easy for them to interpret).
                  Code already in place to generate ome-tiffs.
                  “Original file download" already works!
  Cons: Data doesn’t display as a plate in OMERO clients.
                   Any future plate tools won’t work.

2) Put the whole plate in a small number (say 1-4) files.
        Estimated  ‘bloat’ 1.2 - 4.8 Mb  ( <2% of  the data size stored as ‘stack of tiffs’)
Pros: Data will import into OMERO without problem as a plate  Minimal bloat.
  Cons: FLIMfit needs modification to use SPW metadata rather than filename.
Big! files.
  A MOP will get huge ome-tiffs
                  They will also need to sort out the metadata  to work out what goes where.

3) Have one file per FOV.
        Estimated ‘bloat’ 480Mb (350% of  the data size stored as ‘stack of tiffs’)
        Pros: No modification required to FLIMfit ( each file is a FOV as now).
                Will import to OMERO as a plate.
        Cons: bloat!

4) Have one file per well.
        Estimated ‘bloat’ 96Mb  (70% of the data size stored as ‘stack of tiffs)
        Pros: Data will import into OMERO without problem as a plate.
       Cons: FLIMfit needs modification.
                ‘Bloat’ still large
                 Non-trivial for hypothetical MOP.
                 Although easier  than 2) as they can determine the well from the Filename

5) Use BinaryOnly ome-tiff format.
  [ In this format there is only one copy of the metadata. all the other files just contain a reference to this. “master’ file. ]
        Estimated ‘bloat’ 1.2Mb  ( < 1 % of data size stored as ‘stack of tiffs’)
        Pros: No mods required to FLIMfit (one file per FOV) although using the SPW metadata is a desirable feature.. Minimal ‘bloat’
        Cons: Not yet available.

6)  Store data as ome-tiffs with no SPW xml but add this info in OMERO after import (either via dataset-to-plate script or the API)
       Estimated bloat. Unclear -ask OME team
       Pros. Might allow one file per FOV & work with future plate tools.
       Cons: Might not work - ask OME team
                 Big software effort and end of project is approaching rapidly.

Thanks in advance

Ian

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140816/5ec886ce/attachment.html>