[ome-devel] [IDR Project] Zegami / IDR
Damir Sudar
dsudar at lbl.gov
Mon Jul 4 22:03:36 BST 2016
Hi Will,
Thanks very much for this very decent partial fix. Indeed, this now
works for me. There is still the issue noted in the TODO note but it now
completes the metadata population for my 20 by 35 arrays without errors
and I can go on with my work.
This is great!
- Damir
On 7/4/2016 5:16 AM, William Moore (Staff) wrote:
> Hi Damir,
>
> (Apologies if you get duplicates of this message - been updating my
> ome-devel settings)
> (PR opened at I’ve opened a PR with this fix at
> https://github.com/openmicroscopy/openmicroscopy/pull/4743)
>
>
> How do you want to name your Wells in the csv file?
> If you name Wells like
>
> …
> Z01
> AA01
> AA02
> etc.
>
> then you can add these lines to populate_metadata.py to extend the
> AS_ALPHA by another 26 rows.
>
> AS_ALPHA = [chr(v) for v in range(97, 122 + 1)] # a-z
> + # Support more than 26 rows
> + for v in range(97, 122 + 1):
> + AS_ALPHA.append('a' + chr(v))
> WELL_REGEX = re.compile(r'^([a-zA-Z]+)(\d+)$’)
>
> This works for me with my local setup.
>
> I haven’t discussed with the Glencoe authors of the script whether
> this is the “right” way to fix this, but
> please let us know if it works for you?
>
>
> Cheers,
>
> Will.
>
>
>
>> On 3 Jul 2016, at 01:15, Damir Sudar <dsudar at lbl.gov
>> <mailto:dsudar at lbl.gov>> wrote:
>>
>> Hi Eleanor and OME team,
>>
>> When trying to use the various metadata populate utilities for our
>> project I ran into what I think is a bug:
>>
>> When the number of rows in a plate exceeds 26 (i.e. beyond row "Z"),
>> the Populate_Metadata.py script (and presumably the underlying
>> functionality that does the actual work) fails with the error:
>>> Traceback (most recent call last):
>>> File "./script", line 130, in <module>
>>> message = populate_metadata(client, conn, scriptParams)
>>> File "./script", line 85, in populate_metadata
>>> ctx = ParsingContext(client, omero_object, "")
>>> File "/home/omero_user/OMERO.server/lib/python/omero/util/populate_metadata.py", line 442, in __init__
>>> self.value_resolver = ValueResolver(self.client, self.target_object)
>>> File "/home/omero_user/OMERO.server/lib/python/omero/util/populate_metadata.py", line 245, in __init__
>>> return self.load_plate()
>>> File "/home/omero_user/OMERO.server/lib/python/omero/util/populate_metadata.py", line 318, in load_plate
>>> self.target_object, wells_by_location, wells_by_id, images_by_id
>>> File "/home/omero_user/OMERO.server/lib/python/omero/util/populate_metadata.py", line 330, in parse_plate
>>> columns = wells_by_location[self.AS_ALPHA[row]]
>>> IndexError: list index out of range
>>
>> I'm working with high density printed microarray grids of 20 by 35
>> cell-spots in 8-well plates and I'm abusing the the Screen/Plate/Well
>> functionality by treating each cell spot as a Well, each Array as a
>> Plate, and each of my 8-well plates as a Screen. So far all this
>> works great until this point.
>>
>> I tried some work-arounds such as changing the naming scheme of Rows
>> and Columns (as Dataset_To_Plate.py allows) so that the Rows get the
>> numbers and the Columns get the letters but alas, that fails with the
>> same error so the max number of rows limit of 26 appears to be pretty
>> deeply embedded somewhere.
>>
>> So I looked at the piece of code (populate_metadata.py) where this
>> fails and found two issues:
>> 1) the TODO note: which explains why my work-around failed
>>> # TODO: This should use the PlateNamingConvention. We're
>>> assuming rows
>>> # as alpha and columns as numeric.
>> 2) the AS_ALPHA macro evaluates incorrectly when the row is
>> interpreted as a single letter: which is why anything past "Z" fails
>>
>> Any chance someone smarter than I could have a look and fix?
>>
>> Thanks,
>> - Damir
>>
>>
>> On 6/27/2016 2:03 PM, Eleanor Williams wrote:
>>>
>>> Hi Damir
>>>
>>> To convert the omero tables to key-value pairs annotations we are
>>> again using the CLI metadata populate script along with a
>>> yaml-format configuration file to specify which columns we want to
>>> take from the omero tables and put into Key-Value Annotations.
>>>
>>> This component of the metadata functionality is still in the
>>> development stage and so to use it you need to:
>>>
>>> 1. use the latest version of omero (5.2.4) and
>>> 2. then run as
>>> OMERO_DEV_PLUGINS=1 bin/omero metadata populate --context bulkmap
>>> --cfg bulkmap-config.yml Screen:n
>>> where n is the screen number in omero.
>>>
>>> An example config file for one of our screens is here
>>> <https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml>https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-bulkmap-config.yml
>>>
>>> You can see I have just picked out some of the columns from
>>> theannotation.csv
>>> <https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-annotation.csv>file,
>>> in this case ones that I thought IDR users might use as search
>>> terms, and I have specified URLs for some. You can also change the
>>> 'display name' of the column using the ' clientname' tag and create
>>> URLs using the 'clientvalue' tag.
>>>
>>> Let us know if you have any problems running this, or suggestions
>>> for improvement.
>>> Best regards
>>> Eleanor
>>>
>>> On 27/06/2016 03:04, Damir Sudar wrote:
>>>> Hi Eleanor and Josh,
>>>>
>>>> I'm learning a lot from your work on putting the IDR together and
>>>> I'm applying it for our image repository for our LINCS project
>>>> (http://www.lincsproject.org/centers/data-and-signature-generating-centers/oregon-u/)
>>>> that hopefully will go live in a few months. We're also looking at
>>>> Zegami as a possible way to provide visualizations of that data
>>>> (Great demo, Roger!! and thanks for the code for that).
>>>>
>>>> I've gotten as far as putting my metadata into a a format that can
>>>> be used to generate the omero tables using the Populate Metadata
>>>> script. But I couldn't find how you guys then populate the
>>>> Key-Value Annotations. It appears to be a subset of what's in the
>>>> Tables. Do you have some magic scripts for that and are those
>>>> available?
>>>>
>>>> Cheers,
>>>> - Damir
>>>>
>>>> On 6/16/2016 2:21 AM, Eleanor Williams wrote:
>>>>>
>>>>> Hi Steve and Roger
>>>>>
>>>>> I think idr0002-heriche-condensation would be a good screen to
>>>>> start with. It is relatively small at 12 plates and has good
>>>>> annotation with control, quality control and phenotype values. It
>>>>> can be found here
>>>>> http://idr-demo.openmicroscopy.org/webclient/?show=screen-102.
>>>>>
>>>>> I am doing a bit of a stock-take of the annotation files I've
>>>>> produced so far and will be changing a few aspects over the next
>>>>> few weeks, mainly to include more of the information that is
>>>>> currently held in the study files. For example I'd like to get
>>>>> more of the sample attributes such as species and cell line into
>>>>> the annotation.csv file, and then into the omero tables and Map
>>>>> Annotations. Sample attributes for screens were originally
>>>>> recorded in the study file because it didn't vary per screen, but
>>>>> now that we are also looking at non-screen data where this might
>>>>> vary it makes sense to always have it visible with the images.
>>>>> Also we haven't got as far as exposing much of the study
>>>>> information in omero yet.
>>>>>
>>>>> I would really like to work with you to understand how you are
>>>>> going to use the experimental and analytic annotations and how we
>>>>> can help in providing the metadata you need and in the most usable
>>>>> format. For example would it be useful if I indicate what is
>>>>> sample attribute information in the annotation.csv file by using
>>>>> something like a constistent header format e.g. Characterisics
>>>>> [Organism], Characteristics [Cell Line], Characteristics [Organism
>>>>> Part] etc, or is this not really necessary for your purposes and
>>>>> 'Organism', 'Cell Line' column headings would be enough?
>>>>>
>>>>> Finally, it was Balaji Ramalingam
>>>>> <https://www.openmicroscopy.org/site/about/development-teams/jason/balaji.png/view>
>>>>> that you were talking to about streaming the thumbnails. I'm sure
>>>>> he would be happy to discuss this further with you.
>>>>>
>>>>> Best regards
>>>>>
>>>>> Eleanor
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 15/06/2016 20:59, Josh Moore wrote:
>>>>>> Forwarding this to the mailing list so that anyone who had expressed
>>>>>> interested inhttp://zegami.com/ at this year's users' meeting can
>>>>>> follow along.
>>>>>>
>>>>>> Cheers,
>>>>>> ~Josh
>>>>>>
>>>>>> P.S. If you weren't able to attend #OME2016, a recap is now up on the
>>>>>> bloghttp://blog.openmicroscopy.org/community/2016/06/15/user-meeting/
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Stephen Taylor<stephen.taylor at imm.ox.ac.uk>
>>>>>> Date: Tue, Jun 14, 2016 at 8:17 AM
>>>>>> Subject: Re: [IDR Project] Zegami / IDR
>>>>>> Cc: Roger Noble<roger at zegami.com>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’ve been look at a few examples from the link Josh sent to me and
>>>>>> there are quite a lot! What would be a good data set to get import
>>>>>> into Zegami from IDR? I am thinking a set that has decent metadata
>>>>>> and a study that people are more familiar with that would make an
>>>>>> interesting use case.
>>>>>>
>>>>>> Kind regards,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> From: Stephen Taylor
>>>>>> Sent: 10 June 2016 14:25
>>>>>> Cc: Roger Noble<roger at zegami.com>; 'e.x.williams at dundee.ac.uk'
>>>>>> <e.x.williams at dundee.ac.uk>
>>>>>> Subject: RE: [IDR Project] Zegami / IDR
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am interested generally in getting data in and out of OMERO from
>>>>>> both internal and external. Over a WAN the images are less of an issue
>>>>>> since they are streamed on demand but large metadata on the other hand
>>>>>> can get out of hand since it all has to be loaded completely into
>>>>>> Zegami’s database.
>>>>>>
>>>>>> Currently I do “OMERO project” to “Zegami collection” using a Zegami
>>>>>> export script (modified version of the OMERO batch export script).
>>>>>> This takes a project or selection of images from OMERO and produces
>>>>>> thumbnails and data in tab delimited Zegami format and then I manually
>>>>>> import that into Zegami for processing. The OMERO id is preserved so I
>>>>>> can link to the original image in OMERO using the OMERO web client. I
>>>>>> only export certain metadata currently (filenames and OMERO tags) and
>>>>>> my plan is to make this more comprehensive and allow MapAnnotations
>>>>>> and Tables to be exported using this mechanism as well. I need to find
>>>>>> out more how to query these data types. Since we have internal
>>>>>> projects at Oxford that need this I’d like to finish this and make
>>>>>> that plugin freely available for other OMERO users to publish their
>>>>>> data in Zegami.
>>>>>>
>>>>>>
>>>>>> Longer term for IDR/Zegami
>>>>>>
>>>>>>
>>>>>> One of your team (apologies I didn’t get his name but he was doing
>>>>>> OMERO/ImageJ in the Unconference session) suggested we just stream in
>>>>>> the thumbnail and just have 1 level (i.e. no zoom) and when you want
>>>>>> to see the image just open it up in OMERO. This would not be as nice
>>>>>> to use but may be sufficient for HCS and would allow you to use
>>>>>> Zegami’s facet based query engine and would be probably little (no?)
>>>>>> work for you guys. We need to test this to see what it looks like and
>>>>>> it assumes OMERO can supply a unique thumbnail URL for each image and
>>>>>> that the metadata will provide a primary key so that we can get to
>>>>>> this image via a thumbnail id. To get at the metadata Josh and Eleanor
>>>>>> said they were thinking of making the CSV downloadable for a
>>>>>> collection, which could be loaded into Zegami on the fly (we have
>>>>>> instances where we dynamically load metadata via URL already).
>>>>>> However, major caveat is on how well this would perform over a WAN
>>>>>> with a large amount of data.
>>>>>>
>>>>>> BTW we have developer docs for Zegami for those interested at:
>>>>>>
>>>>>> https://support.zegami.com/hc/en-us/categories/200350657-Developers
>>>>>>
>>>>>> and would welcome feedback.
>>>>>>
>>>>>> Kind regards and thanks,
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> From: Jason Swedlow (Staff) [mailto:j.r.swedlow at dundee.ac.uk]
>>>>>> Sent: 08 June 2016 13:09
>>>>>> Cc: Eleanor Williams (Staff)<e.x.williams at dundee.ac.uk>
>>>>>> Subject: Re: [IDR Project] Zegami / IDR
>>>>>>
>>>>>>
>>>>>> Hi Josh et al—
>>>>>>
>>>>>> My two cents— can we consider this use case from an external users’
>>>>>> perspective, mindful that distribution of such a solution would
>>>>>> definitely cause us to consider how we ship thumbnails, handle
>>>>>> straws, etc.
>>>>>>
>>>>>> Cheers,
>>>>>> Jason
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 8, 2016 at 1:31 PM, Josh Moore<josh at glencoesoftware.com> wrote:
>>>>>>
>>>>>> Stephen,
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 8, 2016 at 9:52 AM, Stephen Taylor
>>>>>> <stephen.taylor at imm.ox.ac.uk> wrote:
>>>>>>
>>>>>>> Hi Josh,
>>>>>>>
>>>>>>>
>>>>>>> Yes! What would be the best way of bulk downloading the thumbnails for a screen or plate
>>>>>> Interesting question. Guess the reverse question is: do you want to do
>>>>>> this as an external user or as if you had access to the OMERO?
>>>>>>
>>>>>>
>>>>>>> ( I notice the "Run Script" button isn't active ).
>>>>>> Correct. This server is stripped down in several ways as a public resource.
>>>>>> ~J.
>>>>>>
>>>>>>
>>>>>>> Kind regards,
>>>>>>> Steve
>>>>>> -----Original Message-----
>>>>>> From: Josh Moore [mailto:josh at glencoesoftware.com]
>>>>>> Sent: 08 June 2016 08:07
>>>>>> To: Stephen Taylor<stephen.taylor at imm.ox.ac.uk>
>>>>>> Cc: Eleanor Williams<exwilliams at dundee.ac.uk>;
>>>>>> e.x.williams at dundee.ac.uk;j.a.moore at dundee.ac.uk
>>>>>>
>>>>>> Subject: Re: Zegami / IDR
>>>>>>
>>>>>>
>>>>>> Hi Stephen,
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 8, 2016 at 8:40 AM, Stephen Taylor
>>>>>> <stephen.taylor at imm.ox.ac.uk> wrote:
>>>>>>
>>>>>>> Hi Eleanor,
>>>>>>>
>>>>>>> Thanks very useful. Is the web site publically available?
>>>>>> You meanhttp://idr-demo.openmicroscopy.org/webclient ?
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> ~Josh
>>>>>>
>>>>>>
>>>>>>> Kind regards and thanks,
>>>>>>>
>>>>>>>
>>>>>>> Steve
>>>>>>> From: Eleanor Williams [mailto:exwilliams at dundee.ac.uk]
>>>>>>> Sent: 07 June 2016 11:09
>>>>>>> To: Stephen Taylor<stephen.taylor at imm.ox.ac.uk>;
>>>>>>> e.x.williams at dundee.ac.uk;j.a.moore at dundee.ac.uk
>>>>>>> Subject: Re: Zegami / IDR
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Stephen
>>>>>>>
>>>>>>> Here is a link to our workshop presentation but it describes how to
>>>>>>> get the data into omero rather than out.
>>>>>>>
>>>>>>> http://downloads.openmicroscopy.org/presentations/2016/Users-Meeting/Workshops/Metadata-at-Scale.pdf
>>>>>>>
>>>>>>> We will be adding the annotation.csv files as attachments to each
>>>>>>> screen over the next couple of weeks.
>>>>>>>
>>>>>>> Let us know if you have any more questions.
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> Eleanor
>>>>>>> On 07/06/2016 10:41, Stephen Taylor wrote:
>>>>>>>
>>>>>>> Hi Eleanor and Josh,
>>>>>>>
>>>>>>> Great to meet you both at the OME meeting last week.
>>>>>>>
>>>>>>>
>>>>>>> I'd be interested in trying get some data into Zegami from IDR. I
>>>>>>> missed your Unmeeting (ironically I was demoing Zegami) where I think
>>>>>>> you went into some of the technical details. Can you send me the link
>>>>>>> to your presentation please?
>>>>>>>
>>>>>>>
>>>>>>> Kind regards and thanks,
>>>>>>> Steve
>>>>>> _______________________________________________
>>>>>> ome-devel mailing list
>>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>>
>>>>> --
>>>>> Eleanor Williams PhD
>>>>>
>>>>> Data Annotator/Software Engineer
>>>>> Centre for Gene Regulation and Expression
>>>>> University of Dundee, UK
>>>>>
>>>>> The University of Dundee is a registered Scottish Charity, No:
>>>>> SC015096
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ome-devel mailing list
>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>>
>>>> --
>>>> Damir Sudar - Affiliate Scientist
>>>> Lawrence Berkeley Natl Laboratory / MBIB
>>>> One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
>>>> T: 510/486-5346 - F: 510/486-5586 - E:DSudar at lbl.gov
>>>> http://biosciences.lbl.gov/profiles/damir-sudar-2/
>>>>
>>>> Visiting Scientist, Oregon Health & Science University
>>>>
>>>>
>>>> _______________________________________________
>>>> ome-devel mailing list
>>>> ome-devel at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>
>>> --
>>> Eleanor Williams PhD
>>>
>>> Data Annotator/Software Engineer
>>> Centre for Gene Regulation and Expression
>>> University of Dundee, UK
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>
>>>
>>> _______________________________________________
>>> ome-devel mailing list
>>> ome-devel at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>> --
>> Damir Sudar - Affiliate Scientist
>> Lawrence Berkeley Natl Laboratory / MBIB
>> One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
>> T: 510/486-5346 - F: 510/486-5586 - E:DSudar at lbl.gov
>> http://biosciences.lbl.gov/profiles/damir-sudar-2/
>>
>> Visiting Scientist, Oregon Health & Science University
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> <mailto:ome-devel at lists.openmicroscopy.org.uk>
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
--
Damir Sudar - Affiliate Scientist
Lawrence Berkeley Natl Laboratory / MBIB
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
http://biosciences.lbl.gov/profiles/damir-sudar-2/
Visiting Scientist, Oregon Health & Science University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160704/f9af5343/attachment.html>
More information about the ome-devel
mailing list