[ome-devel] SPW ome-tiffs & how to proceed?
Curtis Rueden
ctrueden at wisc.edu
Fri Aug 15 20:28:20 BST 2014
Hi Ian,
> My understanding was that there is, currently, no reader or writer for
> this in bio-formats. I’d be happy to be wrong.
I did a quick test with a two-file OME-TIFF dataset, where the first file
has the full block, and the second file uses a BinaryOnly reference to the
first file as described on the website [1]. Bio-Formats was able to read
the full set of metadata and detected the correct dimensions. So I think it
works. There is also certainly some logic in the Bio-Formats source code to
handle it [2, 3].
On the other hand, investigating recent commit messages surrounding this
logic reveals a major inconsistency. One commit message [4] states:
"Some Prairie datasets contain 'OME-TIFF' files with a BinaryOnly entry
that points to a different 'OME-TIFF', which is not valid."
And another [5] strongly implies that the metadata file must be OME-XML,
not OME-TIFF.
But the OME-TIFF Specification [1] gives an OME-TIFF as the example
MetadataFile. So I think the original intent of the BinaryOnly element was
lost somewhere along the way.
Anyway, in practice it seems the BinaryOnly approach should be viable for
you.
-Curtis
[1]
https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/specification.html#storing-partial-metadata-blocks
[2]
https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L203-L206
[3]
https://github.com/openmicroscopy/bioformats/blob/v5.0.3/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L421-L450
[4]
https://github.com/openmicroscopy/bioformats/commit/c4a81ff2903fbfb32820f1eca43aa83788f1cd58
[5]
https://github.com/openmicroscopy/bioformats/commit/7cbe988ebaf2ff81792a56489c7ea311d8af25bc
On Fri, Aug 15, 2014 at 1:47 PM, Munro, Ian <i.munro at imperial.ac.uk> wrote:
> Hi Curtis
>
> My understanding was that there is, currently, no reader or writer for
> this in bio-formats.
> I’d be happy to be wrong.
>
> Ian
>
>
> On 15 Aug 2014, at 18:23, Curtis Rueden <ctrueden at wisc.edu> wrote:
>
> Hi Ian,
>
> I like option 5, storing the metadata into only a subset of the TIFFs.
> LOCI has been doing this for a long time now and it works very well. Not
> sure what you mean about that approach not yet being available.
>
> -Curtis
> On Aug 15, 2014 9:24 AM, "Munro, Ian" <i.munro at imperial.ac.uk> wrote:
>
>> A bit of background first.
>>
>> As you perhaps know the main driver for our involvement in OMERO is the
>> need to share data & make some data
>> publicly available.
>>
>> To achieve this we have been modifying our acquisition software to
>> generate ome-tiffs which can then be imported into OMERO.
>> We have also modified FLIMfit to allow it to read ome-tiffs.
>> This is in order to be able to use the “download original files”
>> functionality in OMERO.web which wasn’t compatible with our earlier
>> approach
>> of writing data directly to OMERO via the OMERO api.
>>
>> We now have software in place to write all our data types as ome-tiff
>> with one exception.
>> We now need to decide how best to handle FLIM plate data.
>>
>> As a first step have now succeeded in writing Java code to create a
>> dummy SPW plate and write it into a Fileset.
>>
>>
>> https://github.com/imunro/bioformats/commit/aeea8cea014a9d546e22c3e5cb577787d2e60352
>>
>> Our original plan was to have one ome-tiff per FOV.
>>
>> However If I’ve written the above code correctly (?) then the issue
>> is, OME-XML’ bloat.’
>> The metadata for the entire plate is required to be in each file of the
>> Fileset therefore the total amount of metadata rises with the number of
>> files.
>> Guesstimating the size of the metadata by linearly scaling from my dummy
>> plate gives us 1.2Mb per file of Metadata for a ‘typical’ plate.
>> Where typical’ is assumed to be 80 wells x 5 FOVs/well x 5
>> time-points/FOV i.e. 400 FLIMages.
>> For comparison the data stored as a stack of tiffs would occupy 140Mb.
>>
>> We therefore have a range of options & we need your input to select one
>> that allows the data to be downloaded ( when this capability becomes
>> available).
>> see
>> https://trello.com/c/nYSjzdnM/260-rfe-extend-batch-download-of-original-files-to-include-spw-in-web-client
>>
>> We note that a solution is in place for JCB
>>
>> Some of our options are:
>>
>> 1) Don’t bother with OME SPW data at all.Simply store the plate data in
>> a dataset
>> Estimated ‘bloat’ 0bytes.
>> Pros: Could simplify the FLIMfit code by removing all Screen/plate menu
>> items.
>> FLIMfit can already read this data by parsing the
>> filename.
>> A hypothetical public downloader (MOP) would get a
>> single file per FOV (easy for them to interpret).
>> Code already in place to generate ome-tiffs.
>> “Original file download" already works!
>> Cons: Data doesn’t display as a plate in OMERO clients.
>> Any future plate tools won’t work.
>>
>> 2) Put the whole plate in a small number (say 1-4) files.
>> Estimated ‘bloat’ 1.2 - 4.8 Mb ( <2% of the data size stored
>> as ‘stack of tiffs’)
>> Pros: Data will import into OMERO without problem as a plate Minimal
>> bloat.
>> Cons: FLIMfit needs modification to use SPW metadata rather than
>> filename.
>> Big! files.
>> A MOP will get huge ome-tiffs
>> They will also need to sort out the metadata to work
>> out what goes where.
>>
>> 3) Have one file per FOV.
>> Estimated ‘bloat’ 480Mb (350% of the data size stored as ‘stack
>> of tiffs’)
>> Pros: No modification required to FLIMfit ( each file is a FOV as
>> now).
>> Will import to OMERO as a plate.
>> Cons: bloat!
>>
>> 4) Have one file per well.
>> Estimated ‘bloat’ 96Mb (70% of the data size stored as ‘stack of
>> tiffs)
>> Pros: Data will import into OMERO without problem as a plate.
>> Cons: FLIMfit needs modification.
>> ‘Bloat’ still large
>> Non-trivial for hypothetical MOP.
>> Although easier than 2) as they can determine the well
>> from the Filename
>>
>> 5) Use BinaryOnly ome-tiff format.
>> [ In this format there is only one copy of the metadata. all the other
>> files just contain a reference to this. “master’ file. ]
>> Estimated ‘bloat’ 1.2Mb ( < 1 % of data size stored as ‘stack of
>> tiffs’)
>> Pros: No mods required to FLIMfit (one file per FOV) although
>> using the SPW metadata is a desirable feature.. Minimal ‘bloat’
>> Cons: Not yet available.
>>
>> 6) Store data as ome-tiffs with no SPW xml but add this info in OMERO
>> after import (either via dataset-to-plate script or the API)
>> Estimated bloat. Unclear -ask OME team
>> Pros. Might allow one file per FOV & work with future plate tools.
>> Cons: Might not work - ask OME team
>> Big software effort and end of project is approaching
>> rapidly.
>>
>>
>> Thanks in advance
>>
>> Ian
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20140815/799dce91/attachment-0001.html>
More information about the ome-devel
mailing list