[ome-users] [ome-devel] Problem with metadata

Thu Oct 30 20:09:21 GMT 2008

Hi Sebastian,

As Jason said, there are several steps in the import process. OMERO.insight
is currently limited in which types of metadata it can show, but that
doesn't necessarily mean your metadata has been dropped. To troubleshoot a
problem with a specific piece of metadata from a particular file, the first
thing is to identify how far your favorite metadata field got along...

*The Bio-Formats/OMERO Metadata Import Process
*
A. Bio-Formats parses the file's metadata in its original structure.

B. Bio-Formats converts as much metadata as it can into the OME data model.

C. The OMERO.importer feeds the standardized OME metadata to the OMERO
database.

D. Once import is complete, client applications (e.g., OMERO.insight) can
query the metadata field from the database, display it to the user, and
otherwise make use of it.

The good news is: it is very easy to verify whether steps A and B are
happening correctly for your metadata of interest, using the Bio-Formats
plugins for ImageJ. Here is how you do it!

Let's take Bio-Rad PIC as an example, though this procedure will work for
any supported file format.

1. If you haven't already done so, download loci_tools.jar from the
Bio-Formats web site (http://www.loci.wisc.edu/ome/formats-download.html)
and drop it into your ImageJ plugins folder. Make sure you are using the
latest trunk build, as it contains the latest bug fixes as well as the
"Display OME-XML metadata" feature we will be using.

2. Start up ImageJ.

3. Choose Plugins > LOCI > Bio-Formats Importer from the menu.

4. Select the file you wish to test and click OK.

5. For "View stack with" choose "Metadata only." Uncheck all the options
except for "Display metadata in results window" and "Display OME-XML
metadata." Click OK and two windows should pop up fairly quickly.

6. To see whether step A is happening, take a look at the "Original
Metadata" window.

For illustration, suppose I am interested in the physical width of my image
in microns, and the magnification factor of the lens used to acquire the
data. Browsing through the list, I find a key labeled "AXIS_4 distance (X)
in microns" that looks pretty promising for the physical width, and another
couple of keys labeled "LENS_MAGNIFICATION" and "lens" equal to "10" which
matches the 10X lens used during the experiment. So we have determined that
Bio-Formats is successfully parsing the metadata I'm interested in.

One important note: please remember that the names of these keys will be
different for each file format.

7. Next we want to check whether step B is happening, so we look at the "OME
Metadata" window. Notice that the display is in an XML tree structure.

According to the OME data model, the physical width in microns is saved as
an attribute of Pixels beneath Image. Looking at my tree, I see that indeed,
the Pixels has a PhysicalSizeX attribute that has been populated.

Next is the lens magnification, which is saved as a property of the
instrument's objective. (There are actually two types of lens magnifications
that OME can represent: a "nominal" magnification, which is the listed
magnification factor of the lens, and a "calibrated" magnification, which is
the actual measured magnification, since it can deviate slightly from the
nominal factor. In our case, we are looking for the nominal magnification,
since the Bio-Rad software merely recorded that I have a 10X lens.) The
specific path we want is Instrument -> Objective -> NominalMagnification.
But looking at my tree, I see that there is no Instrument node at all, much
less an Objective with a NominalMagnification.

So we have verified that Bio-Formats successfully standardized the physical
image width, but we also found a limitation: Bio-Formats does not yet
standardize lens magnification factor for Bio-Rad PIC files!

At this point, you have sufficient information to complain to us ;-) about
exactly what you need added in order to meet your metadata requirements.
Though we are doing our best, it can be very tedious going through each
metadata field one by one, which is why we highly value reports from users
like you about specifically desired metadata fields. And it helps us to
prioritize which file formats and metadata fields to focus on first.

If you prefer the command line to GUIs, it is also possible to follow these
same steps using Bio-Formats from the console instead of ImageJ. To do so,
download the Bio-Formats command line tools from the web site, and use
"showinf -nopix -omexml myFile.pic" to see a dump of both the original
metadata key/value pairs, and the standardized OME-XML tree.

Another question you might have is: assuming my metadata field passes steps
A and B, how can I tell whether step C was successful?

Unfortunately I am not an OMERO expert so I don't know an easy way -- but if
you are a programmer you can take a look at the source code responsible for
it:

https://trac.openmicroscopy.org.uk/omero/browser/trunk/components/importer/src/ome/formats/OMEROMetadataStore.java

Doing a text search for "NominalMagnification" I can find a method called
"setObjectiveNominalMagnification" which sounds like it might do the right
thing, and the code does look like it's actually doing something. In
contrast, the method "setOTFOpticalAxisAveraged" has a debug statement that
says "TODO" and the "setExperimentDescription" method has a "TODO" comment,
so it is probably a safe bet that the OTF optical axis averaging and
experiment description fields are not currently getting past step C.

In conclusion, metadata standardization is a massive undertaking and we are
hugely appreciative of your feedback. I hope this guide has given you a
sense of how to isolate specific things you would like to see added or
improved to Bio-Formats and OMERO, so that we can provide you with a suite
of software tools that meets your needs.

Cheers,
Curtis

On Thu, Oct 30, 2008 at 11:54 AM, Jason Swedlow
<jason at lifesci.dundee.ac.uk>wrote:

> Hi Sebastian-
> Ok, again; a few problems.
>
> Many people have noted the limited exposure of metadata in OMERO.insight.
>  The good news is that this is a subset of what is in the database-- so much
> of what you want is in the DB.  But it is not all there-- some metadata is
> not supported by Bio-formats, some doesn't have a home in OMERO.
>
> This is why we are (as I type) are in the middle of a "metadata
> completeness" iteration-- from Bio-Formats through to OMERO.insight and
> OMERO.editor, we are working hard to address a complete metadata workflow.
> This is a massive undertaking, and much of the team is focussed on it. As an
> example, it means going through each file format and ensuring we have a home
> for all metadata somewhere in OMERO.  Given the huge disparity in metadata
> support across different vendor file formats, this takes some work.  It also
> means finding ways to define what people want to see and how, and raises the
> question of what metadata is editable.  We've had long discussions about
> this for many years, and we also have survey data coming from our partners
> at Usable Image.  We'll provide updates on release of these facilities over
> the next few days.
>
> It's good to hear the 510 and 710 file formats are behaving similarly--
> this is helpful.
>
> I'll ask Brian to follow-up on your question about Bio-Rad .PIC.
>
> Cheers,
>
> Jason
>
> Sebastian Cachero wrote:
>
> Hi Jason,
> We are having the same problem with both types of files.
>
> If I open a 710 or 510 file in imageJ using bio-formats importer and the
> ask for the file info I get a whole bunch of metadata but after importing
> them into OMERO I can only see a basic set of data.
>
> A second problem seems to be that the LSM software from Zeiss gives a
> different voxel size than the metadata retrieved by bio-formats
> importer/OMERO (the LSM toolbox gives the same voxel size as the LSM soft, I
> guess that bio-formats might be calculating voxel size differently) .
>
> Lastly, is there anything special about importing the biorad PIC files? I
> usually have a folder with two pic files (one for each channel) and one
> data.raw, lse.xml and one experiment.lse files.
>
> Best,
>
> Sebastian
>
>
> On 30 Oct 2008, at 12:23, Jason Swedlow wrote:
>
> Hi Sebastian-
> These are problems we are dealing with right now (you have a number of
> issues, from complete metadata support to number of channels) -- the bad
> news is the vendors are doing various things in their file formats that we
> have to deal with.  They tailor their software to handle the problem for
> their file format-- we get to do it for all of them!
>
> A more detailed response will follow, but can you confirm whether the
> problem you are having with Zeiss is a 510 or 710 file?
>
> We are having a meeting to discuss all these problems tomorrow and will
> report back to the community over the weekend.
>
> Thanks!
>
> Cheers,
>
> Jason
>
> On 30 Oct 2008, at 12:03, Sebastian Cachero wrote:
>
> Hi,
> We have just installed omero in the lab and are having a couple of problems
> while trying to import some confocal stacks from a number of different
> microscopes (leica sp5, zeiss 510-710, biorad).
>
> 1. For some reason importing of leica .lif files fails (although
> bioformats-importer has no trouble importing them to imageJ).
>
> 2. Zeiss files are imported correctly but we get only a limited amount of
> the metadata from the original file.
>
> 3. Biorad .pic files get imported but have little metadata (same as zeiss
> files) and the channel information is incorrect (a two channel file gets
> imported as 6 channels, each one looking as being one slice offset the
> others)
>
> Any help with this will be greatly appreciated!
>
> Thanks!
>
> Sebastian
>
> MRC Laboratory of Molecular Biology,
> Hills Road,
> Cambridge,
> CB2 2QH.
> United Kingdom.
>
> Office: +44 (0)1223 252944
> Lab +44 (0)1223 252943
> Fax: +44 (0)1223 402310
> E-mail: scachero at mrc-lmb.cam.ac.uk
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
>
> **************************
> Wellcome Trust Centre for Gene Regulation & Expression
> College of Life Sciences
> MSI/WTB/JBC Complex
> University of Dundee
> Dow Street
> Dundee  DD1 5EH
> United Kingdom
>
> phone (01382) 385819
> Intl phone:  44 1382 385819
> FAX   (01382) 388072
> email: jason at lifesci.dundee.ac.uk
>
> Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/
> Open Microscopy Environment: http://openmicroscopy.org
> **************************
>
> The University of Dundee is a Scottish Registered Charity, No. SC015096.
>
>
>
>
> MRC Laboratory of Molecular Biology,
> Hills Road,
> Cambridge,
> CB2 2QH.
> United Kingdom.
>
> Office: +44 (0)1223 252944
> Lab +44 (0)1223 252943
> Fax: +44 (0)1223 402310
> E-mail: scachero at mrc-lmb.cam.ac.uk
>
>
>
>
>
>
> **************************
> Wellcome Trust Centre for Gene Regulation & Expression
> College of Life Sciences
> MSI/WTB/JBC Complex
> University of Dundee
> Dow Street
> Dundee  DD1 5EH
> United Kingdom
>
> phone (01382) 385819
> Intl phone:  44 1382 385819
> FAX   (01382) 388072
> email: jason at lifesci.dundee.ac.uk
>
> Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/
> Open Microscopy Environment: http://openmicroscopy.org
> **************************
>
> The University of Dundee is a Scottish Registered Charity, No. SC015096.
>
>
>
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20081030/d679dc07/attachment.html>