[ome-devel] SPW ome-tiffs & how to proceed?

[Munro, Ian]

A bit of background first.

As you perhaps know the main driver for our involvement in OMERO is the need to share data & make some data
publicly available.

To achieve this we have been modifying our acquisition software to generate ome-tiffs which can then be imported into OMERO.
We have also modified  FLIMfit to allow it to read ome-tiffs.
This is in order to be able to use the “download original files” functionality in OMERO.web which wasn’t compatible with our earlier approach
of writing data directly to OMERO via the OMERO api.

We now have software in place to write all our data types as ome-tiff with one exception.
We now need to decide how best to handle FLIM plate data.

As a first step have now succeeded in writing Java code to create a dummy SPW plate and write it into a Fileset.

https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352

Our original plan was to have one ome-tiff per FOV.

However If I’ve written the above code  correctly  (?) then the issue  is, OME-XML’ bloat.’
The metadata for the entire plate is required to be in each file of the Fileset therefore the total amount of metadata rises with the number of files.
Guesstimating the size of the metadata by linearly scaling from my dummy plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.
Where typical’ is assumed to be  80 wells x 5 FOVs/well x 5 time-points/FOV i.e. 400 FLIMages.
For comparison the data stored as a stack of tiffs would occupy 140Mb.

We therefore have a range of options & we need your input to select one that allows the data to be downloaded ( when this capability becomes available).
see   https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client

We note that a solution is in place for JCB

Some of our options are:

1) Don’t bother with OME SPW data at all.Simply store the plate data in a dataset
Estimated  ‘bloat’ 0bytes.
Pros: Could simplify the FLIMfit code by removing all Screen/plate menu items.
                  FLIMfit can already read this data by parsing the filename.
                  A hypothetical public downloader (MOP) would get a single file per FOV (easy for them to interpret).
                  Code already in place to generate ome-tiffs.
                  “Original file download" already works!
  Cons: Data doesn’t display as a plate in OMERO clients.
                   Any future plate tools won’t work.

2) Put the whole plate in a small number (say 1-4) files.
        Estimated  ‘bloat’ 1.2 - 4.8 Mb  ( <2% of  the data size stored as ‘stack of tiffs’)
Pros: Data will import into OMERO without problem as a plate  Minimal bloat.
  Cons: FLIMfit needs modification to use SPW metadata rather than filename.
Big! files.
  A MOP will get huge ome-tiffs
                  They will also need to sort out the metadata  to work out what goes where.

3) Have one file per FOV.
        Estimated ‘bloat’ 480Mb (350% of  the data size stored as ‘stack of tiffs’)
        Pros: No modification required to FLIMfit ( each file is a FOV as now).
                Will import to OMERO as a plate.
        Cons: bloat!

4) Have one file per well.
        Estimated ‘bloat’ 96Mb  (70% of the data size stored as ‘stack of tiffs)
        Pros: Data will import into OMERO without problem as a plate.
       Cons: FLIMfit needs modification.
                ‘Bloat’ still large
                 Non-trivial for hypothetical MOP.
                 Although easier  than 2) as they can determine the well from the Filename



5) Use BinaryOnly ome-tiff format.
  [ In this format there is only one copy of the metadata. all the other files just contain a reference to this. “master’ file. ]
        Estimated ‘bloat’ 1.2Mb  ( < 1 % of data size stored as ‘stack of tiffs’)
        Pros: No mods required to FLIMfit (one file per FOV) although using the SPW metadata is a desirable feature.. Minimal ‘bloat’
        Cons: Not yet available.

6)  Store data as ome-tiffs with no SPW xml but add this info in OMERO after import (either via dataset-to-plate script or the API)
       Estimated bloat. Unclear -ask OME team
       Pros. Might allow one file per FOV & work with future plate tools.
       Cons: Might not work - ask OME team
                 Big software effort and end of project is approaching rapidly.


Thanks in advance

Ian







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140815/391a7179/attachment.html>