Hi Mike,<br><br>Thanks very much for all your comments.<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;">
<div><div><div><div><blockquote type="cite"><div>My
name is Mike Riffle and I work with Trisha Davis over here in Seattle,
handling the informatics endeavors for the Yeast Resource Center. As
you already know, we've developed a large database of fluorescence
microscopy images (YRC Public Image Repository, <a href="http://images.yeastrc.org/%29" target="_blank">http://images.yeastrc.org/)</a>,
for which I've implemented support for disseminating the data as
fully-valid OME TIFF files. Trisha forwarded some email to me (from
you) regarding this web site and suggested that I send you some
feedback about OME and my experience getting it set up in our site. I
know how hard feedback can be to come by, and I'm more than happy to
oblige.<br>
<br>I spent a /lot /of time going over the ample documentation the OME
project provides on their websites, and I know that I've barely
scratched the surface of all that OME has to offer. My goal was
relatively simple: I wanted to distribute the 500,000+ TIFF images and
accompanying experimental metadata as OME TIFFs in our public image
database. The idea being that users who download these data will be
able to plug them directly into software that supports the OME format
specifications.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>Excellent!<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>Accomplishing this task, however, was not entirely trivial. As you
know, the XML specification for OME-XML is very extensive. While
documentation does exist, I did have some challenges figuring out how
our metadata should be mapped onto the elements present in the XML
specification. I don't know if the documentation was intended for
people doing what I was doing, but expanding the documentation to more
generally describe what is meant by each element (from a high-level
light microscopy POV) could potentially be beneficial to developers.
Also, a high level description or diagram of the data model in terms
of items present in most light microscopy experiments, and where these
common elements should go in the XML would be great. For example,
given 4 emissions/excitation wavelength channels and 6 z-sections of
each channel, this is how that data should be organized in an XML file.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>We definitely want to encourage this sort of thing. My group (LOCI) is in the middle of a similar effort, mapping all our acquisition metadata to OME-XML, and working to produce valid OME-TIFF files in every case.<br>
<br>So yes, the documentation is supposed to help. I agree that the documentation in the schema itself could use a lot of improvement. My group has a graduate student in library sciences, Caitlin Sticco, who is currently analyzing the schema and compiling a list of suggestions and improvements; hopefully we can update the documentation as well as we work to improve the schema.<br>
<br>Caitlin and I are also discussing how to make better diagrams—one idea we had is to generate them in Graphviz format from the XSD file. Any other suggestions for ways to make useful pictures would be welcome.<br><br>
Your idea of a list of common use cases, and how to express them in OME-XML, is a great one. We should add that to the OME-XML web site. For that, we need A) a list of which common use cases to include; and B) someone with enough time to work on writing it up. But at the least we can file a ticket about it on the OME-XML Trac at <a href="http://www.ome-xml.org/">http://www.ome-xml.org/</a>. And actually, you are welcome to file such tickets too, using the ome/ome login. :-) To be clear, we want to encourage external suggestions through that mechanism.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>Once I figured out where I thought the metadata should appear in
the XML, I set about using the bio-formats Java library to assemble OME
TIFFs from my TIFF image data and metadata. I found the bio-formats
library to be great and the documentation very helpful. While it's
obvious some of the documentation exists as "stubs", it got the job
done. I did have some questions about how to do certain things and
used the mailing list. Curtis Rueden responded promptly and helped
guide me through my problems and I got everything working. I was
definitely impressed by his promptness and eagerness to help.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>Thanks for the kind words. If you have any specific comments about which documentation you would like to see expanded, it would be very helpful.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>
However, as it turns out, the bio-formats Java library doesn't
produce OME-TIFFs that pass validation using the OME-XML validation
tool at <a href="http://validator.openmicroscopy.org.uk/" target="_blank">http://validator.openmicroscopy.org.uk/</a>.
I talked to Curtis and this is a known problem. I was a bit worried
about producing "invalid" OME-TIFFs for distribution in our site, so I
went ahead and ended up writing my own code from scratch to produce
valid OME-TIFFs. This required me to become even more intimately
familiar with the XML specification and all of its particular
requirements (such as the fact that it strictly enforces the order of
the XML elements in the OME-XML portion of the OME-TIFF file).<br></div></blockquote></div></div></div></div></div></blockquote><div><br>We are in the process of adding a stylesheet transformation within Bio-Formats that will ensure a correct ordering of the elements. Once this is in place, we will still need to fix a few more validation issues, but we are getting closer to Bio-Formats producing valid OME-XML. I am very sorry that you ended up needing to write your own code, rather than leveraging an existing tool.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>
Our site is now producing OME-TIFFs that pass validation at the OME
validator and include all of our experimental metadata, so I'm quite
pleased. This certainly wouldn't have been possible without all of the
great documentation and the help of the OME staff. Overall, it was
probably a bit more complicated than it needed to be, but it was a fine
experience and I learned a lot about the OME tools and file
specification.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>That is great news. And again, if you have any more specific comments with regard to the documentation, or other aspects of the process that you felt were unnecessarily difficult or complex, please let us know.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>
Another suggestion I have is regarding the possibility of adding in
support for describing the experiments biologically--particularly
relating to describing the tagged proteins, protein tags or dyes used
and possibly the ability to include some manual curation from
controlled vocabularies, such as the Gene Ontology. I ended up using
user-defined elements to include these types of descriptive elements in
the OME-TIFF files. I didn't see any official way to include this
information, and it definitely seems like information relevant to the
data being viewed.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>Our current plan is to somehow integrate the OME-XML specification with ontologies in the field, in some sort of general way. We want to avoid redoing any of the work that has already been done in this area.<br>
<br>It sounds like you did not find an existing data model appropriate to your biological metadata, although there may be appropriate ontologies for the vocabularies.<br><br>A list of your biological metadata would be extremely useful for getting us started in the right direction. I took at look at one of the datasets available from the YRC site, and checked out the custom fields:<br>
<br> <CA:OriginalMetadata ID="OriginalMetadata:0" Name="ImageWidth" Value="512"/><br> <CA:OriginalMetadata ID="OriginalMetadata:1" Name="ImageHeight" Value="512"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:2" Name="BitsPerSample" Value="16"/><br> <CA:OriginalMetadata ID="OriginalMetadata:3" Name="Compression" Value="None"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:4" Name="PhotometricInterpretation" Value="BlackIsZero"/><br> <CA:OriginalMetadata ID="OriginalMetadata:5" Name="MetaDataPhotometricInterpretation" Value="Monochrome"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:6" Name="SamplesPerPixel" Value="1"/><br> <CA:OriginalMetadata ID="OriginalMetadata:7" Name="NewSubfileType" Value="0"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:8" Name="BitsPerSample" Value="16"/><br> <CA:OriginalMetadata ID="OriginalMetadata:9" Name="NumberOfChannels" Value="1"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:10" Name="MetaMorph" Value="no"/><br> <CA:OriginalMetadata ID="OriginalMetadata:11" Name="PhotoSensor" Value="582741"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:12" Name="VisibleProtein" Value="CMD1"/><br> <CA:OriginalMetadata ID="OriginalMetadata:13" Name="VisibleProteinSpecies" Value="4932"/><br>
<CA:OriginalMetadata ID="OriginalMetadata:14" Name="VisibleProteinTag" Value="YFP"/><br><br>It looks like VisibleProtein, VisibleProteinSpecies and VisibleProteinTag are relevant fields biologically (though I am not sure of PhotoSensor—sounds like an instrument setting?). Are there other important biological metadata fields being archived in other datasets?<br>
<br>Also, I also see that you are storing some TIFF-specific metadata here—is that a side effect? And BitsPerSample appears twice. :-)<br><br>Any other suggestions regarding representation of biological metadata in OME-XML?<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><div><div><div><blockquote type="cite"><div>
Again, it was a positive experience and I look forward to learning
more about OME and how we can make our data play well with your tools.
If you are ever looking for feedback or for anyone to help contribute
to the cause, please don't hesitate to ask. I would love to be
involved.<br></div></blockquote></div></div></div></div></div></blockquote><div><br>Great, thanks for the offer! The main thing is to let us know specifically how you would like to see things move forward, and try any tools we release along those lines as time goes on. Like you said, feedback is rare but extremely valuable.<br>
<br>Cheers,<br>Curtis</div></div>