[ome-devel] New developmental metadata functionality

Tue Aug 16 15:04:45 BST 2016

On 15 August 2016 at 20:47, Damir Sudar <dsudar at lbl.gov<mailto:dsudar at lbl.gov>> wrote:
Hi Simon,

Thanks for the quick response.

On 8/15/2016 2:31 AM, Simon Li wrote:

Just a few questions and possibly there are some suggestions for improvement hidden in there:
1) when I upload a .csv file with "bin/omero metadata populate --file filename.csv --context csv Screen:<screen_id>"
the resulting Table is called "bulk_annotations". Is there a way to upload with a different name?

Not at the moment, though it should be possible to change the name since OMERO.web uses the file-annotation's namespace. You can change it manually if you want:

omero obj update OriginalFile:123 Name=experimental-data.h5

Ah yes, good thought. I'll try to do that.

2) one reason I ask is that I would like to upload 2 different annotations to a Screen: #1 is the experimental metadata that describes the experiment and #2 is a feature measurements file after analysis which is likely to be done at a later time, thus the desire to separate that from the experimental metadata. Currently both attached Tables care named bulk_annotations.

That sounds similar to something we're doing at the moment in the IDR, though we're not displaying the feature values at the moment. One thing we're considering is having a separate namespace for the bulk-annotations and the feature analysis.

Do you have some ideas for how this should be presented, or is the current tabular display OK for you?

For now, the tabular display is fine since our current feature tables are per-image summary statistics, not per-cell feature measurement data. Once one wants to store per-cell features as a table per image (and we will want to do so in the near future) the simple tabular display will no longer suffice since it will be at least a multi-column table. In order to display anything like, you probably will need a separate window/table, maybe similar to: http://www.openmicroscopy.org/site/support/omero5.2/developers/analysis.html#third-party-analysis-and-omero-tables
But that's probably something for a serious design review and I presume will rely on all the new ROI work going on, right?

You're right. We do have cell-level analysis data for some of the IDR screens, but we decided against displaying it in the UI due to the significant work required. It's certainly important, so if it's OK we might ask for your input in future with regards to it's design.

3) so while it is possible to upload 2 different annotation files and both show up as Attachments to the Screen object (with the same "bulk_annotations" name), when I look at the Image objects within that Screen, they only show in the Tables tab the contents from the most recently uploaded annotation file.

That was a workaround we implemented a few years ago
https://github.com/openmicroscopy/openmicroscopy/pull/512#issuecomment-11372968
Now that the metadata work is seeing more use it's something we should improve.

Ah, I see. For now there are a few work-arounds that I can use: 1) merge all my annotation files into one and upload that (and redo that when the due to a new analysis algorithm there are different measured features), and 2) (and this is what I am currently doing) upload the static experimental metadata, create the key-value pairs using the metadata populate functionality for everything I want people to see, then upload the feature annotation file and that then shows up in the Tables tab. A little clunky but workable.
The original intention in the IDR was to use Map-annotations for display, and to not display the content of the tables at all, though they'd still be accessible as file-annotations. I'll make a note that it should be configurable!

4) the metadata annotation works great for "normal" plates that have numbered columns and "lettered" rows. However, when one has lettered columns and numbered rows, everything gets jumbled and when there are more than 26 rows the routine throws an exception (see my earlier message). I worked around this issue (mostly with Will Moore's help, see: https://github.com/openmicroscopy/openmicroscopy/pull/4743) but it would be great if that issue was really fixed before releasing the metadata functionality into the wild.

Again this is something that's only come to light as more people use it, I've added it to our bug-list.

Thanks. That is actually a bug I'd love to see disappear.

Thanks for your feedback!

And thank you guys for building all this functionality I need.

One more question: in my current workflow I do not create a column in my experimental metadata annotation file called "Image Name", mostly because I wouldn't know what for to use that, and thus the populator generates lots of messages: "INFO:omero.util.populate_metadata:Missing image name column, skipping."
Not a big problem but annoying. Since it appears to be just an INFO message, is there a way to make it appear only once? Or am I missing something and I really should have a Image Name column?

After some discussion in the team we're not aware of a reason, I'll add a ticket for the log messages to be turned down.

Simon

Cheers,
- Damir

Simon

Thanks,
- Damir

On 6/27/2016 2:03 PM, Eleanor Williams wrote:

Hi Damir

To convert the omero tables to key-value pairs annotations we are again using the CLI metadata populate script along with a yaml-format configuration file to specify which columns we want to take from the omero tables and put into Key-Value Annotations.

This component of the metadata functionality is still in the development stage and so to use it you need to:

1. use the latest version of omero (5.2.4) and

2. then run as

OMERO_DEV_PLUGINS=1  bin/omero metadata populate --context bulkmap --cfg bulkmap-config.yml Screen:n

where n is the screen number in omero.

An example config file for one of our screens is here

<https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml>https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml

You can see I have just picked out some of the columns from theannotation.csv<https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-annotation.csv> file, in this case ones that I thought IDR users might use as search terms, and I have specified URLs for some.  You can also change the 'display name' of the column using the ' clientname' tag and create URLs using the 'clientvalue' tag.

Let us know if you have any problems running this, or suggestions for improvement.

Best regards
Eleanor

--
Damir Sudar - Affiliate Scientist
Lawrence Berkeley Natl Laboratory / MBIB
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346<tel:510%2F486-5346> - F: 510/486-5586<tel:510%2F486-5586> - E: DSudar at lbl.gov<mailto:DSudar at lbl.gov>
http://biosciences.lbl.gov/profiles/damir-sudar-2/

Visiting Scientist, Oregon Health & Science University

The University of Dundee is a registered Scottish Charity, No: SC015096

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160816/630b23b4/attachment.html>