[ome-devel] New developmental metadata functionality

Mon Aug 15 20:47:20 BST 2016

Hi Simon,

Thanks for the quick response.

On 8/15/2016 2:31 AM, Simon Li wrote:
>
>     Just a few questions and possibly there are some suggestions for
>     improvement hidden in there:
>     1) when I upload a .csv file with "bin/omero metadata populate
>     --file filename.csv --context csv Screen:<screen_id>"
>     the resulting Table is called "bulk_annotations". Is there a way
>     to upload with a different name?
>
>
> Not at the moment, though it should be possible to change the name 
> since OMERO.web uses the file-annotation's namespace. You can change 
> it manually if you want:
>
> omero obj update OriginalFile:123 Name=experimental-data.h5

Ah yes, good thought. I'll try to do that.
>
>     2) one reason I ask is that I would like to upload 2 different
>     annotations to a Screen: #1 is the experimental metadata that
>     describes the experiment and #2 is a feature measurements file
>     after analysis which is likely to be done at a later time, thus
>     the desire to separate that from the experimental metadata.
>     Currently both attached Tables care named bulk_annotations.
>
>
> That sounds similar to something we're doing at the moment in the IDR, 
> though we're not displaying the feature values at the moment. One 
> thing we're considering is having a separate namespace for the 
> bulk-annotations and the feature analysis.
>
> Do you have some ideas for how this should be presented, or is the 
> current tabular display OK for you?

For now, the tabular display is fine since our current feature tables 
are per-image summary statistics, not per-cell feature measurement data. 
Once one wants to store per-cell features as a table per image (and we 
will want to do so in the near future) the simple tabular display will 
no longer suffice since it will be at least a multi-column table. In 
order to display anything like, you probably will need a separate 
window/table, maybe similar to: 
http://www.openmicroscopy.org/site/support/omero5.2/developers/analysis.html#third-party-analysis-and-omero-tables
But that's probably something for a serious design review and I presume 
will rely on all the new ROI work going on, right?

>
>     3) so while it is possible to upload 2 different annotation files
>     and both show up as Attachments to the Screen object (with the
>     same "bulk_annotations" name), when I look at the Image objects
>     within that Screen, they only show in the Tables tab the contents
>     from the most recently uploaded annotation file.
>
>
> That was a workaround we implemented a few years ago
> https://github.com/openmicroscopy/openmicroscopy/pull/512#issuecomment-11372968 
> <https://github.com/openmicroscopy/openmicroscopy/pull/512#issuecomment-11372968>
> Now that the metadata work is seeing more use it's something we should 
> improve.

Ah, I see. For now there are a few work-arounds that I can use: 1) merge 
all my annotation files into one and upload that (and redo that when the 
due to a new analysis algorithm there are different measured features), 
and 2) (and this is what I am currently doing) upload the static 
experimental metadata, create the key-value pairs using the metadata 
populate functionality for everything I want people to see, then upload 
the feature annotation file and that then shows up in the Tables tab. A 
little clunky but workable.
>
>     4) the metadata annotation works great for "normal" plates that
>     have numbered columns and "lettered" rows. However, when one has
>     lettered columns and numbered rows, everything gets jumbled and
>     when there are more than 26 rows the routine throws an exception
>     (see my earlier message). I worked around this issue (mostly with
>     Will Moore's help, see:
>     https://github.com/openmicroscopy/openmicroscopy/pull/4743
>     <https://github.com/openmicroscopy/openmicroscopy/pull/4743>) but
>     it would be great if that issue was really fixed before releasing
>     the metadata functionality into the wild.
>
>
> Again this is something that's only come to light as more people use 
> it, I've added it to our bug-list.

Thanks. That is actually a bug I'd love to see disappear.
>
> Thanks for your feedback!

And thank you guys for building all this functionality I need.

One more question: in my current workflow I do not create a column in my 
experimental metadata annotation file called "Image Name", mostly 
because I wouldn't know what for to use that, and thus the populator 
generates lots of messages: "INFO:omero.util.populate_metadata:Missing 
image name column, skipping."
Not a big problem but annoying. Since it appears to be just an INFO 
message, is there a way to make it appear only once? Or am I missing 
something and I really should have a Image Name column?

Cheers,
- Damir

>
> Simon
>
>     Thanks,
>     - Damir
>
>     On 6/27/2016 2:03 PM, Eleanor Williams wrote:
>>
>>     Hi Damir
>>
>>     To convert the omero tables to key-value pairs annotations we are
>>     again using the CLI metadata populate script along with a
>>     yaml-format configuration file to specify which columns we want
>>     to take from the omero tables and put into Key-Value Annotations.
>>
>>
>>     This component of the metadata functionality is still in the
>>     development stage and so to use it you need to:
>>
>>
>>     1. use the latest version of omero (5.2.4) and
>>
>>     2. then run as
>>
>>     OMERO_DEV_PLUGINS=1  bin/omero metadata populate --context
>>     bulkmap --cfg bulkmap-config.yml Screen:n
>>
>>     where n is the screen number in omero.
>>
>>
>>     An example config file for one of our screens is here
>>
>>     <https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml>https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml
>>     <https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml>
>>
>>
>>     You can see I have just picked out some of the columns from
>>     theannotation.csv
>>     <https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-annotation.csv>file,
>>     in this case ones that I thought IDR users might use as search
>>     terms, and I have specified URLs for some.  You can also change
>>     the 'display name' of the column using the ' clientname' tag and
>>     create URLs using the 'clientvalue' tag.
>>
>>
>>     Let us know if you have any problems running this, or suggestions
>>     for improvement.
>>
>>     Best regards
>>     Eleanor
>>

-- 
Damir Sudar - Affiliate Scientist
Lawrence Berkeley Natl Laboratory / MBIB
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
http://biosciences.lbl.gov/profiles/damir-sudar-2/

Visiting Scientist, Oregon Health & Science University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160815/02a8912c/attachment.html>