[ome-devel] ome-tiff files: does it really needs to be namedxx.ome.tif ??

Tue Jan 13 22:15:23 GMT 2009

Hi Ghislain,

I am glad that this is issue is raised as .ext1.ext2 is in no way a standard
> extension.
>
>
Not a common practice perhaps, but I am curious what you see as the
practical pitfalls of this compound extension. It seems to me that using
.ome.tif gives us the best of both worlds: 1) an unambiguous extension for
the OME-TIFF format, and 2) compatibility with existing TIFF software.

Perhaps, using simply a .tiff with a format specific IFD would make more
> sense.
>

We have discussed this idea in the past. Do you see any advantages to your
approach other than those you mention below?

In addition, this would allow for a mechanism to transform Tiff based file
> formats more efficiently, and provide backward compatibility with other
> readers.
>
>
How would a custom IFD entry be more efficient than the current mechanism?
Currently you could inject the OME-XML metadata extremely efficiently by
surgically overwriting the ImageDescription tag. The only downside is that a
TIFF cannot simultaneously be an OME-TIFF and some other flavor of TIFF that
also uses the ImageDescription tag for its metadata. A custom IFD entry
would alleviate that issue.

How is the current specification not backwardly compatible with other
readers? The only way I can think of is if those other readers are also
expecting custom metadata from the same ImageDescription tag.

For instance the Opera .flex file is roughly a Tiff file with some specific
> header (and in some cases a specific compression of the pixel data). One
> could think that adding the OME-XML header under an OME specific IFD would
> make the most sense. While the original IFD would remain unchanged.
>
>
Is your concern that you do not wish to overwrite the Flex file's existing
ImageDescription tag? I checked a bunch of our sample Flex files, and none
of them have an existing ImageDescription tag. Are you using this field?

The rationale thus far has been that any actual "image description" -- i.e.,
comment -- can go in the OME-XML block's Description tag. But I understand
that there are cases where this solution is undesirable: like I said, if the
same field holds metadata in some other structure, it could be a problem.
Examples include TIFFs saved by ImageJ, and Leica TCS TIFFs.

Being able to combine these forms of metadata within a single TIFF is of
potential benefit. Is there anyone out there who wants to do this, and wants
to see the OME-TIFF specification changed?

On another note and still about metadata. As the metadata can become
> extremely large, would it make sense to provide a mechanism to compress it
> using deflate for instance? Or is there a mechanism for this?
>

For OME-TIFF, there is no mechanism to compress the XML at the moment. You
can do it with the OME-XML file format, using extension .omez, with zlib.
However, the metadata is a tiny fraction of the total data size, even when
represented in a rather inefficient XML format.

The major goals of OME-TIFF are performance and compatibility. Uncompressed
TIFF planes are essentially raw, and storing the metadata in the first
ImageDescription tag allows quick access (not to say that a custom tag
wouldn't be just as quick). Compressing the XML block would increase the
amount of time necessary to parse the metadata, though that is no reason not
to allow it as an option. If this option would really significantly reduce
the size of your data, we can discuss how best to add such a facility.

In conclusion, I am resistant to changing the OME-TIFF specification without
a compelling practical benefit. The spec has already seen some breaking
changes that have made it difficult to maintain compatibility within
Bio-Formats, and I would prefer to avoid any further complications in the
implementation. But it would be foolish not to discuss and evaluate any
potential benefits to such changes.

In this case, unless someone in the community would directly benefit from
the custom IFD entry approach, I think the advantages are mostly theoretical
and would not warrant the disruption caused by changing the specification
again. Please feel free to argue if you disagree. :-)

-Curtis

On Tue, Jan 13, 2009 at 3:09 PM, Ghislain Bonamy <GBonamy at gnf.org> wrote:

>  Curtis, Frans,
>
>
>
> I am glad that this is issue is raised as .ext1.ext2 is in no way a
> standard extension.
>
>
>
> Perhaps, using simply a .tiff with a format specific IFD would make more
> sense. In addition, this would allow for a mechanism to transform Tiff based
> file formats more efficiently, and provide backward compatibility with other
> readers.
>
>
>
> For instance the Opera .flex file is roughly a Tiff file with some specific
> header (and in some cases a specific compression of the pixel data). One
> could think that adding the OME-XML header under an OME specific IFD would
> make the most sense. While the original IFD would remain unchanged.
>
>
>
> On another note and still about metadata. As the metadata can become
> extremely large, would it make sense to provide a mechanism to compress it
> using deflate for instance? Or is there a mechanism for this?
>
>
>
> Best,
>
>
>
> Ghislain Bonamy, PhD
>
> __________________________________________
>
> Research Investigator I
>
>
>
> Genomic Institute of the
>
> Novartis Research
>
> Foundation
>
> Department of Molecular & Cell Biology, room G214
>
> 10675 John Jay Hopkins Drive
>
> San Diego CA 92121
>
> USA
>
>
>
> +1 (858) 812-1534 (W & F)
>
> +1 (757) 941-4194 (H)
>
> +1 (858) 354-7388 (M)
>
> www.gnf.org
>
>
>
> Hudson-Alpha Institute for Biotechnology
>
> www.hudsonalpha.org <http://www.haib.org>
>
>
>
> *From:* ome-devel-bounces at lists.openmicroscopy.org.uk [mailto:
> ome-devel-bounces at lists.openmicroscopy.org.uk] *On Behalf Of *Curtis
> Rueden
> *Sent:* Tuesday, January 13, 2009 1:00 PM
> *To:* Cornelissen, Frans [PRDBE]
> *Cc:* ome-devel at lists.openmicroscopy.org.uk
> *Subject:* Re: [ome-devel] ome-tiff files: does it really needs to be
> namedxx.ome.tif ??
>
>
>
> Hi Frans,
>
> When using Tiff files, we would like to convert them to OME-tiff so that
> they do contain the OME-XML metadata.
>
> Currently the new files have to contain the .ome.tiff as extension
> In our analysis processes, the altered name causes a disruption.
>
>
> Originally, the specification did not require the .ome.tif extension, but
> we decided it would reduce ambiguity to prefer a more specific extension --
> and the .ome.tif extension allows non-OME-aware TIFF programs to continue
> seeing the files as regular TIFFs.
>
> Question: is it really a hard requirement that the .ome. part is in the
> filename?
>
>
> At the moment, for Bio-Formats and hence OMERO, yes it is a hard
> requirement. We are not necessarily opposed to parsing OME-TIFF metadata out
> of files without the .ome.tif extension, but at the moment there are some
> technical barriers to doing so efficiently.
>
> This in itself is no proof of the fact that the file *really* contains a
> valid OME-xml structure, so an application is probably going the check
> internally to decide whether it is an OME file anyway...
>
>
> True. The same is true for every file extension -- the only way to verify
> that the file *really* contains correctly structured data of the indicated
> type is to attempt to fully parse it. However, file extension is an
> extremely useful hint that greatly improves performance. In some cases
> (e.g., certain raw data formats) it might even be impossible to completely
> determine the file format without the filename extension.
>
> Could the .ome. extension requirement be removed for importing ome-tiff
> files into OMERO?
>
>
> Yes, we always parse a TIFF file's ImageDescription block. Ideally, we
> should be properly parsing any OME-XML we find there. However, as I said,
> there are some performance challenges we need to sort out. The fix shouldn't
> be too bad. We'll file a ticket to keep you posted.
>
> -Curtis
>
> On Mon, Jan 12, 2009 at 8:43 AM, Cornelissen, Frans [PRDBE] <
> FCORNELI at its.jnj.com> wrote:
>
> Hi,
>
> When using Tiff files, we would like to convert them to OME-tiff so that
> they do contain the OME-XML metadata.
>
> Currently the new files have to contain the .ome.tiff as extension
> In our analysis processes, the altered name causes a disruption.
>
> Question: is it really a hard requirement that the .ome. part is in the
> filename?
> This in itself is no proof of the fact that the file *really* contains a
> valid OME-xml structure, so an application is probably going the check
> internally to decide whether it is an OME file anyway...
>
> Could the .ome. extension requirement be removed for importing ome-tiff
> files into OMERO?
>
> Best regards, frans cornelissen
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090113/772e65fd/attachment.htm 