[ome-devel] Fwd: experience with OME
Curtis Rueden
ctrueden at wisc.edu
Wed Sep 16 23:09:16 BST 2009
Hi Mike,
Thanks very much for all your comments.
My name is Mike Riffle and I work with Trisha Davis over here in Seattle,
> handling the informatics endeavors for the Yeast Resource Center. As you
> already know, we've developed a large database of fluorescence microscopy
> images (YRC Public Image Repository, http://images.yeastrc.org/)<http://images.yeastrc.org/%29>,
> for which I've implemented support for disseminating the data as fully-valid
> OME TIFF files. Trisha forwarded some email to me (from you) regarding this
> web site and suggested that I send you some feedback about OME and my
> experience getting it set up in our site. I know how hard feedback can be
> to come by, and I'm more than happy to oblige.
>
> I spent a /lot /of time going over the ample documentation the OME project
> provides on their websites, and I know that I've barely scratched the
> surface of all that OME has to offer. My goal was relatively simple: I
> wanted to distribute the 500,000+ TIFF images and accompanying experimental
> metadata as OME TIFFs in our public image database. The idea being that
> users who download these data will be able to plug them directly into
> software that supports the OME format specifications.
>
>
Excellent!
Accomplishing this task, however, was not entirely trivial. As you know,
> the XML specification for OME-XML is very extensive. While documentation
> does exist, I did have some challenges figuring out how our metadata should
> be mapped onto the elements present in the XML specification. I don't know
> if the documentation was intended for people doing what I was doing, but
> expanding the documentation to more generally describe what is meant by each
> element (from a high-level light microscopy POV) could potentially be
> beneficial to developers. Also, a high level description or diagram of the
> data model in terms of items present in most light microscopy experiments,
> and where these common elements should go in the XML would be great. For
> example, given 4 emissions/excitation wavelength channels and 6 z-sections
> of each channel, this is how that data should be organized in an XML file.
>
>
We definitely want to encourage this sort of thing. My group (LOCI) is in
the middle of a similar effort, mapping all our acquisition metadata to
OME-XML, and working to produce valid OME-TIFF files in every case.
So yes, the documentation is supposed to help. I agree that the
documentation in the schema itself could use a lot of improvement. My group
has a graduate student in library sciences, Caitlin Sticco, who is currently
analyzing the schema and compiling a list of suggestions and improvements;
hopefully we can update the documentation as well as we work to improve the
schema.
Caitlin and I are also discussing how to make better diagrams—one idea we
had is to generate them in Graphviz format from the XSD file. Any other
suggestions for ways to make useful pictures would be welcome.
Your idea of a list of common use cases, and how to express them in OME-XML,
is a great one. We should add that to the OME-XML web site. For that, we
need A) a list of which common use cases to include; and B) someone with
enough time to work on writing it up. But at the least we can file a ticket
about it on the OME-XML Trac at http://www.ome-xml.org/. And actually, you
are welcome to file such tickets too, using the ome/ome login. :-) To be
clear, we want to encourage external suggestions through that mechanism.
Once I figured out where I thought the metadata should appear in the XML, I
> set about using the bio-formats Java library to assemble OME TIFFs from my
> TIFF image data and metadata. I found the bio-formats library to be great
> and the documentation very helpful. While it's obvious some of the
> documentation exists as "stubs", it got the job done. I did have some
> questions about how to do certain things and used the mailing list. Curtis
> Rueden responded promptly and helped guide me through my problems and I got
> everything working. I was definitely impressed by his promptness and
> eagerness to help.
>
>
Thanks for the kind words. If you have any specific comments about which
documentation you would like to see expanded, it would be very helpful.
However, as it turns out, the bio-formats Java library doesn't produce
> OME-TIFFs that pass validation using the OME-XML validation tool at
> http://validator.openmicroscopy.org.uk/. I talked to Curtis and this is a
> known problem. I was a bit worried about producing "invalid" OME-TIFFs for
> distribution in our site, so I went ahead and ended up writing my own code
> from scratch to produce valid OME-TIFFs. This required me to become even
> more intimately familiar with the XML specification and all of its
> particular requirements (such as the fact that it strictly enforces the
> order of the XML elements in the OME-XML portion of the OME-TIFF file).
>
>
We are in the process of adding a stylesheet transformation within
Bio-Formats that will ensure a correct ordering of the elements. Once this
is in place, we will still need to fix a few more validation issues, but we
are getting closer to Bio-Formats producing valid OME-XML. I am very sorry
that you ended up needing to write your own code, rather than leveraging an
existing tool.
Our site is now producing OME-TIFFs that pass validation at the OME
> validator and include all of our experimental metadata, so I'm quite
> pleased. This certainly wouldn't have been possible without all of the
> great documentation and the help of the OME staff. Overall, it was probably
> a bit more complicated than it needed to be, but it was a fine experience
> and I learned a lot about the OME tools and file specification.
>
>
That is great news. And again, if you have any more specific comments with
regard to the documentation, or other aspects of the process that you felt
were unnecessarily difficult or complex, please let us know.
Another suggestion I have is regarding the possibility of adding in support
> for describing the experiments biologically--particularly relating to
> describing the tagged proteins, protein tags or dyes used and possibly the
> ability to include some manual curation from controlled vocabularies, such
> as the Gene Ontology. I ended up using user-defined elements to include
> these types of descriptive elements in the OME-TIFF files. I didn't see any
> official way to include this information, and it definitely seems like
> information relevant to the data being viewed.
>
>
Our current plan is to somehow integrate the OME-XML specification with
ontologies in the field, in some sort of general way. We want to avoid
redoing any of the work that has already been done in this area.
It sounds like you did not find an existing data model appropriate to your
biological metadata, although there may be appropriate ontologies for the
vocabularies.
A list of your biological metadata would be extremely useful for getting us
started in the right direction. I took at look at one of the datasets
available from the YRC site, and checked out the custom fields:
<CA:OriginalMetadata ID="OriginalMetadata:0" Name="ImageWidth"
Value="512"/>
<CA:OriginalMetadata ID="OriginalMetadata:1" Name="ImageHeight"
Value="512"/>
<CA:OriginalMetadata ID="OriginalMetadata:2" Name="BitsPerSample"
Value="16"/>
<CA:OriginalMetadata ID="OriginalMetadata:3" Name="Compression"
Value="None"/>
<CA:OriginalMetadata ID="OriginalMetadata:4"
Name="PhotometricInterpretation" Value="BlackIsZero"/>
<CA:OriginalMetadata ID="OriginalMetadata:5"
Name="MetaDataPhotometricInterpretation" Value="Monochrome"/>
<CA:OriginalMetadata ID="OriginalMetadata:6" Name="SamplesPerPixel"
Value="1"/>
<CA:OriginalMetadata ID="OriginalMetadata:7" Name="NewSubfileType"
Value="0"/>
<CA:OriginalMetadata ID="OriginalMetadata:8" Name="BitsPerSample"
Value="16"/>
<CA:OriginalMetadata ID="OriginalMetadata:9"
Name="NumberOfChannels" Value="1"/>
<CA:OriginalMetadata ID="OriginalMetadata:10" Name="MetaMorph"
Value="no"/>
<CA:OriginalMetadata ID="OriginalMetadata:11" Name="PhotoSensor"
Value="582741"/>
<CA:OriginalMetadata ID="OriginalMetadata:12" Name="VisibleProtein"
Value="CMD1"/>
<CA:OriginalMetadata ID="OriginalMetadata:13"
Name="VisibleProteinSpecies" Value="4932"/>
<CA:OriginalMetadata ID="OriginalMetadata:14"
Name="VisibleProteinTag" Value="YFP"/>
It looks like VisibleProtein, VisibleProteinSpecies and VisibleProteinTag
are relevant fields biologically (though I am not sure of PhotoSensor—sounds
like an instrument setting?). Are there other important biological metadata
fields being archived in other datasets?
Also, I also see that you are storing some TIFF-specific metadata here—is
that a side effect? And BitsPerSample appears twice. :-)
Any other suggestions regarding representation of biological metadata in
OME-XML?
Again, it was a positive experience and I look forward to learning more
> about OME and how we can make our data play well with your tools. If you
> are ever looking for feedback or for anyone to help contribute to the cause,
> please don't hesitate to ask. I would love to be involved.
>
>
Great, thanks for the offer! The main thing is to let us know specifically
how you would like to see things move forward, and try any tools we release
along those lines as time goes on. Like you said, feedback is rare but
extremely valuable.
Cheers,
Curtis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090916/49af596c/attachment-0001.html>
More information about the ome-devel
mailing list