[ome-devel] SPW ome-tiffs & how to proceed?

[Curtis Rueden]

Hi Ian,

I like option 5, storing the metadata into only a subset of the TIFFs. LOCI
has been doing this for a long time now and it works very well. Not sure
what you mean about that approach not yet being available.

-Curtis
On Aug 15, 2014 9:24 AM, "Munro, Ian" <i.munro at imperial.ac.uk> wrote:

>  A bit of background first.
>
>  As you perhaps know the main driver for our involvement in OMERO is the
> need to share data & make some data
> publicly available.
>
>  To achieve this we have been modifying our acquisition software to
> generate ome-tiffs which can then be imported into OMERO.
> We have also modified  FLIMfit to allow it to read ome-tiffs.
> This is in order to be able to use the “download original files”
> functionality in OMERO.web which wasn’t compatible with our earlier
> approach
> of writing data directly to OMERO via the OMERO api.
>
>  We now have software in place to write all our data types as ome-tiff
> with one exception.
> We now need to decide how best to handle FLIM plate data.
>
>  As a first step have now succeeded in writing Java code to create a
> dummy SPW plate and write it into a Fileset.
>
>
> https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352
>
>  Our original plan was to have one ome-tiff per FOV.
>
>  However If I’ve written the above code  correctly  (?) then the issue
>  is, OME-XML’ bloat.’
> The metadata for the entire plate is required to be in each file of the
> Fileset therefore the total amount of metadata rises with the number of
> files.
> Guesstimating the size of the metadata by linearly scaling from my dummy
> plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.
> Where typical’ is assumed to be  80 wells x 5 FOVs/well x 5
> time-points/FOV i.e. 400 FLIMages.
> For comparison the data stored as a stack of tiffs would occupy 140Mb.
>
>  We therefore have a range of options & we need your input to select one
> that allows the data to be downloaded ( when this capability becomes
> available).
> see
> https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client
>
>  We note that a solution is in place for JCB
>
>  Some of our options are:
>
>  1) Don’t bother with OME SPW data at all.Simply store the plate data in
> a dataset
> Estimated  ‘bloat’ 0bytes.
> Pros: Could simplify the FLIMfit code by removing all Screen/plate menu
> items.
>                   FLIMfit can already read this data by parsing the
> filename.
>                   A hypothetical public downloader (MOP) would get a
> single file per FOV (easy for them to interpret).
>                   Code already in place to generate ome-tiffs.
>                   “Original file download" already works!
>   Cons: Data doesn’t display as a plate in OMERO clients.
>                    Any future plate tools won’t work.
>
>  2) Put the whole plate in a small number (say 1-4) files.
>         Estimated  ‘bloat’ 1.2 - 4.8 Mb  ( <2% of  the data size stored as
> ‘stack of tiffs’)
> Pros: Data will import into OMERO without problem as a plate  Minimal
> bloat.
>   Cons: FLIMfit needs modification to use SPW metadata rather than
> filename.
> Big! files.
>   A MOP will get huge ome-tiffs
>                   They will also need to sort out the metadata  to work
> out what goes where.
>
>  3) Have one file per FOV.
>         Estimated ‘bloat’ 480Mb (350% of  the data size stored as ‘stack
> of tiffs’)
>         Pros: No modification required to FLIMfit ( each file is a FOV as
> now).
>                 Will import to OMERO as a plate.
>         Cons: bloat!
>
>  4) Have one file per well.
>         Estimated ‘bloat’ 96Mb  (70% of the data size stored as ‘stack of
> tiffs)
>         Pros: Data will import into OMERO without problem as a plate.
>        Cons: FLIMfit needs modification.
>                 ‘Bloat’ still large
>                  Non-trivial for hypothetical MOP.
>                  Although easier  than 2) as they can determine the well
> from the Filename
>
>
>  5) Use BinaryOnly ome-tiff format.
>   [ In this format there is only one copy of the metadata. all the other
> files just contain a reference to this. “master’ file. ]
>         Estimated ‘bloat’ 1.2Mb  ( < 1 % of data size stored as ‘stack of
> tiffs’)
>         Pros: No mods required to FLIMfit (one file per FOV) although
> using the SPW metadata is a desirable feature.. Minimal ‘bloat’
>         Cons: Not yet available.
>
> 6)  Store data as ome-tiffs with no SPW xml but add this info in OMERO
> after import (either via dataset-to-plate script or the API)
>        Estimated bloat. Unclear -ask OME team
>        Pros. Might allow one file per FOV & work with future plate tools.
>        Cons: Might not work - ask OME team
>                  Big software effort and end of project is approaching
> rapidly.
>
>
>  Thanks in advance
>
>  Ian
>
>
>
>
>
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140815/54ab0999/attachment-0001.html>