<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='color:black'>Curtis,</span><o:p></o:p></p>
<div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>I am glad that this is issue is
raised as .ext1.ext2 is in no way a standard extension.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
Not a common practice perhaps, but I am curious what you see as the practical
pitfalls of this compound extension. It seems to me that using .ome.tif gives
us the best of both worlds: 1) an unambiguous extension for the OME-TIFF
format, and 2) compatibility with existing TIFF software.<o:p></o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Arial","sans-serif";color:red'>I concede this, and my comment was
more about the form. In addition, I was more focused on whether or not the
reader should assume a file format based on the extension, and the fact that
other tiff based format may have a different extension than Tiff such as the
.flex files from evotech. This can be a problem since renaming the file would
make it inaccessible to other file systems etc. So perhaps, supposing than any
unknown extension is a Tiff based format and prioritizing tiffs to be first
assumed to me OME complient and then be a different flavor of it would make
sense. For instance a flex file with a flex extension is considered as such
unless it has a tag indicating that it is an OME compliant file (in which case
it would be handled by the OMETiffReader).<o:p></o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>Perhaps, using simply a .tiff
with a format specific IFD would make more sense.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
We have discussed this idea in the past. Do you see any advantages to your
approach other than those you mention below?<span style='color:black'> </span><o:p></o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Arial","sans-serif";color:red'>No these are the advantages that I
am referring too.<o:p></o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>In addition, this would allow for
a mechanism to transform Tiff based file formats more efficiently, and provide
backward compatibility with other readers.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
How would a custom IFD entry be more efficient than the current mechanism?
Currently you could inject the OME-XML metadata extremely efficiently by
surgically overwriting the ImageDescription tag. The only downside is that a
TIFF cannot simultaneously be an OME-TIFF and some other flavor of TIFF that
also uses the ImageDescription tag for its metadata. A custom IFD entry would
alleviate that issue.<br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'>This
is precisely my point.<o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
How is the current specification not backwardly compatible with other readers?
The only way I can think of is if those other readers are also expecting custom
metadata from the same ImageDescription tag.<span style='color:black'><o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Arial","sans-serif";color:black'>I meant that the custom IFD being
conserved, a TIFF format could still be read by the other readers.<o:p></o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>For instance the Opera .flex file
is roughly a Tiff file with some specific header (and in some cases a specific
compression of the pixel data). One could think that adding the OME-XML header
under an OME specific IFD would make the most sense. While the original IFD
would remain unchanged.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
Is your concern that you do not wish to overwrite the Flex file's existing
ImageDescription tag? I checked a bunch of our sample Flex files, and none of
them have an existing ImageDescription tag. Are you using this field?<span
style='color:black'> </span><br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'>Perhaps,
I am getting confused between IFDs and ImageDescription tags. The Opera Flex
file contains an IFD= 65200, which points to an XML file containing the proprietary
metadata. My idea would be to do the same for OME-Tiff so that the metadata can
be read by OME complient or proprietary readers.</span><span style='font-size:
11.0pt;font-family:"Arial","sans-serif";color:black'><o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
The rationale thus far has been that any actual "image description"
-- i.e., comment -- can go in the OME-XML block's Description tag. But I
understand that there are cases where this solution is undesirable: like I
said, if the same field holds metadata in some other structure, it could be a
problem. Examples include TIFFs saved by ImageJ, and Leica TCS TIFFs.<br>
<br>
Being able to combine these forms of metadata within a single TIFF is of
potential benefit. Is there anyone out there who wants to do this, and wants to
see the OME-TIFF specification changed?<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-right:0cm'>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>On another note and still about
metadata. As the metadata can become extremely large, would it make sense to
provide a mechanism to compress it using deflate for instance? Or is there a
mechanism for this?</span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
For OME-TIFF, there is no mechanism to compress the XML at the moment. You can
do it with the OME-XML file format, using extension .omez, with zlib. However,
the metadata is a tiny fraction of the total data size, even when represented
in a rather inefficient XML format.<br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'>I
agree with this, which is why Tiff format might be bit easier to implement
this. One could compress the pixel data using an optimized algorithm such as
the JPEG2000, LZW… and compress the Metadata contained in the ImageDescription
tag (or in an IFD) using his preferred algorithm. Although compressing this
kind of data takes some time decompressing it is extremely fast (cf. using GZIp
compression). The easiest solution would be to use an extra IFD to indicate if
the metadata is compressed or not! <o:p></o:p></span></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
The major goals of OME-TIFF are performance and compatibility. Uncompressed
TIFF planes are essentially raw, and storing the metadata in the first
ImageDescription tag allows quick access (not to say that a custom tag wouldn't
be just as quick). Compressing the XML block would increase the amount of time
necessary to parse the metadata, though that is no reason not to allow it as an
option. If this option would really significantly reduce the size of your data,
we can discuss how best to add such a facility.<br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'>Again
allowing compression using for instance GZIP is slow for the compression step
but almost instantaneous when reading. In addition, right now metadata are manageably
small, but with ROI and other flavor of metdata it may become more and more consequent.
I am merely mentioning something which I foresee could be useful in the future,
while remaining optional depending on the users need.</span><br>
In conclusion, I am resistant to changing the OME-TIFF specification without a
compelling practical benefit. The spec has already seen some breaking changes
that have made it difficult to maintain compatibility within Bio-Formats, and I
would prefer to avoid any further complications in the implementation. But it
would be foolish not to discuss and evaluate any potential benefits to such
changes.<br>
<br>
In this case, unless someone in the community would directly benefit from the
custom IFD entry approach, I think the advantages are mostly theoretical and
would not warrant the disruption caused by changing the specification again.
Please feel free to argue if you disagree. :-)<br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'>Yes,
this is still theoretical and it really depends on everyone’s need. Obviously
this would be most useful to us and other people who do high-throughput imaging
and collect millions of images for a screen and where metadata becomes larger
and larger. Perhaps, this would be nice to keep in mind if an HTF format flavor
is ever implemented. But once again, perhaps, am I the only one in this
situation… so far ;)</span><br>
<br>
<span style='font-size:11.0pt;font-family:"Arial","sans-serif";color:red'><o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Arial","sans-serif";color:red'>-Ghislain<o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>On Tue, Jan 13, 2009 at 3:09
PM, Ghislain Bonamy <<a href="mailto:GBonamy@gnf.org">GBonamy@gnf.org</a>>
wrote:<o:p></o:p></p>
<div>
<div>
<p><span style='font-size:11.0pt;color:black'>Curtis, Frans,</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>I am glad that this is issue is
raised as .ext1.ext2 is in no way a standard extension.</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Perhaps, using simply a .tiff
with a format specific IFD would make more sense. In addition, this would allow
for a mechanism to transform Tiff based file formats more efficiently, and
provide backward compatibility with other readers.</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>For instance the Opera .flex file
is roughly a Tiff file with some specific header (and in some cases a specific
compression of the pixel data). One could think that adding the OME-XML header
under an OME specific IFD would make the most sense. While the original IFD
would remain unchanged.</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>On another note and still about
metadata. As the metadata can become extremely large, would it make sense to
provide a mechanism to compress it using deflate for instance? Or is there a
mechanism for this?</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Best,</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Ghislain Bonamy, PhD</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>__________________________________________</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Research Investigator I</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Genomic Institute of the</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Novartis Research</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Foundation</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Department of Molecular &
Cell Biology, room G214</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>10675 John Jay Hopkins Drive</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>San Diego CA 92121</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>USA</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>+1 (858) 812-1534 (W & F)</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>+1 (757) 941-4194 (H)</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>+1 (858) 354-7388 (M)</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'><a href="http://www.gnf.org"
target="_blank">www.gnf.org</a></span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'>Hudson-Alpha Institute for
Biotechnology</span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'><a href="http://www.haib.org"
target="_blank">www.hudsonalpha.org</a></span><o:p></o:p></p>
<p><span style='font-size:11.0pt;color:black'> </span><o:p></o:p></p>
<div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;
border-color:-moz-use-text-color -moz-use-text-color'>
<p><b><span style='font-size:10.0pt'>From:</span></b><span style='font-size:
10.0pt'> <a href="mailto:ome-devel-bounces@lists.openmicroscopy.org.uk"
target="_blank">ome-devel-bounces@lists.openmicroscopy.org.uk</a> [mailto:<a
href="mailto:ome-devel-bounces@lists.openmicroscopy.org.uk" target="_blank">ome-devel-bounces@lists.openmicroscopy.org.uk</a>]
<b>On Behalf Of </b>Curtis Rueden<br>
<b>Sent:</b> Tuesday, January 13, 2009 1:00 PM<br>
<b>To:</b> Cornelissen, Frans [PRDBE]<br>
<b>Cc:</b> <a href="mailto:ome-devel@lists.openmicroscopy.org.uk"
target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<b>Subject:</b> Re: [ome-devel] ome-tiff files: does it really needs to be
namedxx.ome.tif ??</span><o:p></o:p></p>
</div>
<div>
<div>
<p> <o:p></o:p></p>
<p style='margin-bottom:12.0pt'>Hi Frans,<o:p></o:p></p>
<div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p>When using Tiff files, we would like to convert them to OME-tiff so that<br>
they do contain the OME-XML metadata.<br>
<br>
Currently the new files have to contain the .ome.tiff as extension<br>
In our analysis processes, the altered name causes a disruption.<o:p></o:p></p>
</blockquote>
<div>
<p style='margin-bottom:12.0pt'><br>
Originally, the specification did not require the .ome.tif extension, but we
decided it would reduce ambiguity to prefer a more specific extension -- and
the .ome.tif extension allows non-OME-aware TIFF programs to continue seeing
the files as regular TIFFs.<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p>Question: is it really a hard requirement that the .ome. part is in the<br>
filename?<o:p></o:p></p>
</blockquote>
<div>
<p style='margin-bottom:12.0pt'><br>
At the moment, for Bio-Formats and hence OMERO, yes it is a hard requirement.
We are not necessarily opposed to parsing OME-TIFF metadata out of files
without the .ome.tif extension, but at the moment there are some technical
barriers to doing so efficiently.<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p>This in itself is no proof of the fact that the file *really* contains a<br>
valid OME-xml structure, so an application is probably going the check<br>
internally to decide whether it is an OME file anyway...<o:p></o:p></p>
</blockquote>
<div>
<p style='margin-bottom:12.0pt'><br>
True. The same is true for every file extension -- the only way to verify that
the file *really* contains correctly structured data of the indicated type is
to attempt to fully parse it. However, file extension is an extremely useful
hint that greatly improves performance. In some cases (e.g., certain raw data
formats) it might even be impossible to completely determine the file format
without the filename extension.<o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;
margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;
border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color rgb(204, 204, 204)'>
<p>Could the .ome. extension requirement be removed for importing ome-tiff<br>
files into OMERO?<o:p></o:p></p>
</blockquote>
</div>
<p style='margin-bottom:12.0pt'><br>
Yes, we always parse a TIFF file's ImageDescription block. Ideally, we should
be properly parsing any OME-XML we find there. However, as I said, there are
some performance challenges we need to sort out. The fix shouldn't be too bad.
We'll file a ticket to keep you posted.<br>
<br>
-Curtis<o:p></o:p></p>
<div>
<p>On Mon, Jan 12, 2009 at 8:43 AM, Cornelissen, Frans [PRDBE] <<a
href="mailto:FCORNELI@its.jnj.com" target="_blank">FCORNELI@its.jnj.com</a>>
wrote:<o:p></o:p></p>
<p>Hi,<br>
<br>
When using Tiff files, we would like to convert them to OME-tiff so that<br>
they do contain the OME-XML metadata.<br>
<br>
Currently the new files have to contain the .ome.tiff as extension<br>
In our analysis processes, the altered name causes a disruption.<br>
<br>
Question: is it really a hard requirement that the .ome. part is in the<br>
filename?<br>
This in itself is no proof of the fact that the file *really* contains a<br>
valid OME-xml structure, so an application is probably going the check<br>
internally to decide whether it is an OME file anyway...<br>
<br>
Could the .ome. extension requirement be removed for importing ome-tiff<br>
files into OMERO?<br>
<br>
Best regards, frans cornelissen<br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel"
target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><o:p></o:p></p>
</div>
<p> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>