[ome-devel] Fwd: [IDR Project] Zegami / IDR

Fri Jun 17 14:44:02 BST 2016

On Fri, Jun 17, 2016 at 3:05 PM, Eleanor Williams
<exwilliams at dundee.ac.uk> wrote:
> Hi Steve,
>
> You can et the current annotation.csv file for idr0002-heriche-condensation
> here
> https://github.com/IDR/idr-metadata/tree/master/idr0002-heriche-condensation/screenA
> if you want to try it out.  If you want to see how a larger file scales then
> idr0013-neumann-mitocheck/screenA is a good one to look at
> https://github.com/IDR/idr-metadata/tree/master/idr0013-neumann-mitocheck/screenA.
> Again I'll be updating these files to include species etc soon, but they
> should still be useful as they are.
>
> There are no references to image ids in these files.  Josh will write to you
> separately about how to link the two.

Taking the first row from
https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-condensation/screenA/idr0002-screenA-annotation.csv
the following snippet will give you the related image object:

    import omero
    from omero.rtypes import unwrap
    c = omero.client(ip)
    c.createSession(name, user)
    from omero.gateway import BlitzGateway
    conn = BlitzGateway(client_obj=c)
    plate = conn.getObject("Plate", 422)
    well = plate.getWellGrid()[0][0]
    image = well.getImage()
    map = image.getAnnotation()
    print map.getMapValueAsMap()

    {'Gene Identifier': 'ENSG00000117399',
     'Gene Identifier URL': 'http://www.ensembl.org/id/ENSG00000117399',
     'Gene Symbol': 'CDC20',
     'siRNA Identifier': 's2748'}

since well A1 is the first-row, first-column (i.e. [0][0]) of the well grid.

Cheers,
~Josh

> A plate viewer would be useful for seeing if there are any technical
> problems across the plate.
>
> Best regards
>
> Eleanor
>
>
>
>
> On 17/06/2016 10:03, Stephen Taylor wrote:
>
> Hi Eleanor,
>
>
>
> I think idr0002-heriche-condensation would be a good screen to start with.
> It is relatively small at 12 plates and has good annotation with control,
> quality control and phenotype values.  It can be found here
> http://idr-demo.openmicroscopy.org/webclient/?show=screen-102.
>
> These look good. Thanks!
>
>
>
> I am doing a bit of a stock-take of the annotation files I've produced so
> far and will be changing a few aspects over the next few weeks, mainly to
> include more of the information that is currently held in the study files.
> For example I'd like to get more of the sample attributes such as species
> and cell line into the annotation.csv file, and then into the omero tables
> and Map Annotations.  Sample attributes for screens were originally recorded
> in the study file because it didn't vary per screen, but now that we are
> also looking at non-screen data where this might vary it makes sense to
> always have it visible with the images.  Also we haven't got as far as
> exposing much of the study information in omero yet.
>
> I would really like to work with you to understand how you are going to use
> the experimental and analytic annotations and how we can help in providing
> the metadata you need and in the most usable format.  For example would it
> be useful if I indicate what is sample attribute information in the
> annotation.csv file by using something like a constistent header format e.g.
> Characterisics [Organism], Characteristics [Cell Line], Characteristics
> [Organism Part] etc, or is this not really necessary for your purposes and
> 'Organism', 'Cell Line' column headings would be enough?
>
>
>
> Key information would be OMERO id of the thumbnail (to potentially stream)
> and OMERO id so we can link back to the full image in the OMERO database
>
> So then we can construct URLs like this:
>
> http://idr-demo.openmicroscopy.org/webgateway/render_thumbnail/179717/96/
>
> http://idr-demo.openmicroscopy.org/webclient/img_detail/179715/
>
> Plate, Well id (or number) is also pretty useful. Do you think we would need
> to write a Plate Viewer to organise the tiles for the user?
>
> I would imagine in a lot of studies we won’t have control on how they are
> annotated. Zegami can read most data sources by writing an appropriate data
> loader plugin, so I think the issue is how fast can we get the metadata out
> of OMERO and what level of caching would be required on the Zegami side. For
> example, if the data is in available as a single flat table (such as
> .csv/.tsv) format it should be feasible to load a collection of this size
> dynamically (1152 wells) since I did something similar by reading a shared
> Google Spreadsheet. Very large metadata files will be a bigger problem.
>
> With regards to other information needed I think it depends on the
> biological question. For example, this experiment is published so the
> analysis has been done and summary statistics produced for each well
> (Quality Control, Median Deviation Fraction - Shorter Prophase, Median
> Deviation Fraction - Longer Prophase, Phenotype Reproducibility - Shorter
> Prophase etc) and these could all be imported in as a mixture of categorical
> and numeric data and then sorted and filtered. i.e. show me all the wells
> where Quality Control is “pass” and then sort on “Median Deviation Fraction
> - Longer Prophase:”. The scientist then can review the images to identify
> candidate wells, then link to the time series in OMERO to the well of
> interest. You could also feed in a whole bunch of other statistics to help
> verify the methods during development of the screen. For example, in this
> case the authors use CellCognition software to classify nuclei stages
> (interphase, early prophase, late prophase, prometaphase, metaphase) and
> you could verify how well the algorithms are doing by having a %nuclei stage
> for each well. This would allow you to look for wells that had more cells in
> prometaphase for example and check the analysis is correct. You could also
> add in various gene annotations, and other metadata that was useful.
>
>
>
> Finally, it was Balaji Ramalingam that you were talking to about streaming
> the thumbnails.  I'm sure he would be happy to discuss this further with
> you.
>
>
>
> Ok great.
>
> Please let me know if you have any further questions.
>
> Kind regards,
>
> Steve
>
>
>
> Best regards
>
> Eleanor
>
>
>
>
>
>
>
> On 15/06/2016 20:59, Josh Moore wrote:
>
> Forwarding this to the mailing list so that anyone who had expressed
>
> interested in http://zegami.com/ at this year's users' meeting can
>
> follow along.
>
>
>
> Cheers,
>
> ~Josh
>
>
>
> P.S. If you weren't able to attend #OME2016, a recap is now up on the
>
> blog http://blog.openmicroscopy.org/community/2016/06/15/user-meeting/
>
>
>
>
>
> ---------- Forwarded message ----------
>
> From: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>
>
> Date: Tue, Jun 14, 2016 at 8:17 AM
>
> Subject: Re: [IDR Project] Zegami / IDR
>
> Cc: Roger Noble <roger at zegami.com>
>
>
>
> Hi,
>
>
>
> I’ve been look at a few examples from the link Josh sent to me and
>
> there are quite a lot! What would be a good data set to get import
>
> into Zegami from IDR?  I am thinking a set that has decent metadata
>
> and a study that people are more familiar with that would make an
>
> interesting use case.
>
>
>
> Kind regards,
>
> Steve
>
>
>
>
>
> From: Stephen Taylor
>
> Sent: 10 June 2016 14:25
>
> Cc: Roger Noble <roger at zegami.com>; 'e.x.williams at dundee.ac.uk'
>
> <e.x.williams at dundee.ac.uk>
>
> Subject: RE: [IDR Project] Zegami / IDR
>
>
>
> Hi all,
>
>
>
> I am interested generally in getting data in and out of OMERO from
>
> both internal and external. Over a WAN the images are less of an issue
>
> since they are streamed on demand but large metadata on the other hand
>
> can get out of hand since it all has to be loaded completely into
>
> Zegami’s database.
>
>
>
> Currently I do “OMERO project” to “Zegami collection” using a Zegami
>
> export script (modified version of the OMERO batch export script).
>
> This takes a project or selection of images from OMERO and produces
>
> thumbnails and data in tab delimited Zegami format and then I manually
>
> import that into Zegami for processing. The OMERO id is preserved so I
>
> can link to the original image in OMERO using the OMERO web client. I
>
> only export certain metadata currently (filenames and OMERO tags) and
>
> my plan is to make this more comprehensive and allow MapAnnotations
>
> and Tables to be exported using this mechanism as well. I need to find
>
> out more how to query these data types.  Since we have internal
>
> projects at Oxford that need this I’d like to finish this and make
>
> that plugin freely available for other OMERO users to publish their
>
> data in Zegami.
>
>
>
>
>
> Longer term for IDR/Zegami
>
>
>
>
>
> One of your team (apologies I didn’t get his name but he was doing
>
> OMERO/ImageJ in the Unconference session) suggested we just stream in
>
> the thumbnail and just have 1 level (i.e. no zoom) and when you want
>
> to see the image just open it up in OMERO. This would not be as nice
>
> to use but may be sufficient for HCS and would allow you to use
>
> Zegami’s facet based query engine and would be probably little (no?)
>
> work for you guys. We need to test this to see what it looks like and
>
> it assumes OMERO can supply a unique thumbnail URL for each image and
>
> that the metadata will provide a primary key so that we can get to
>
> this image via a thumbnail id. To get at the metadata Josh and Eleanor
>
> said they were thinking of making the CSV downloadable for a
>
> collection, which could be loaded into Zegami on the fly (we have
>
> instances where we dynamically load metadata via URL already).
>
> However, major caveat is on how well this would perform over a WAN
>
> with a large amount of data.
>
>
>
> BTW we have developer docs for Zegami for those interested at:
>
>
>
> https://support.zegami.com/hc/en-us/categories/200350657-Developers
>
>
>
> and would welcome feedback.
>
>
>
> Kind regards and thanks,
>
>
>
> Steve
>
>
>
>
>
> From: Jason Swedlow (Staff) [mailto:j.r.swedlow at dundee.ac.uk]
>
> Sent: 08 June 2016 13:09
>
> Cc: Eleanor Williams (Staff) <e.x.williams at dundee.ac.uk>
>
> Subject: Re: [IDR Project] Zegami / IDR
>
>
>
>
>
> Hi Josh et al—
>
>
>
> My two cents— can we consider this use case from an external users’
>
> perspective, mindful that distribution of such a solution would
>
> definitely cause us to  consider how we ship thumbnails, handle
>
> straws, etc.
>
>
>
> Cheers,
>
> Jason
>
>
>
>
>
>
>
> On Wed, Jun 8, 2016 at 1:31 PM, Josh Moore <josh at glencoesoftware.com> wrote:
>
>
>
> Stephen,
>
>
>
>
>
> On Wed, Jun 8, 2016 at 9:52 AM, Stephen Taylor
>
> <stephen.taylor at imm.ox.ac.uk> wrote:
>
>
>
> Hi Josh,
>
>
>
>
>
> Yes! What would be the best way of bulk downloading the thumbnails for a
> screen or plate
>
>
>
> Interesting question. Guess the reverse question is: do you want to do
>
> this as an external user or as if you had access to the OMERO?
>
>
>
>
>
> ( I notice the "Run Script" button isn't active ).
>
>
>
> Correct. This server is stripped down in several ways as a public resource.
>
> ~J.
>
>
>
>
>
> Kind regards,
>
> Steve
>
>
>
>
>
> -----Original Message-----
>
> From: Josh Moore [mailto:josh at glencoesoftware.com]
>
> Sent: 08 June 2016 08:07
>
> To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>
>
> Cc: Eleanor Williams <exwilliams at dundee.ac.uk>;
>
> e.x.williams at dundee.ac.uk; j.a.moore at dundee.ac.uk
>
>
>
> Subject: Re: Zegami / IDR
>
>
>
>
>
> Hi Stephen,
>
>
>
>
>
> On Wed, Jun 8, 2016 at 8:40 AM, Stephen Taylor
>
> <stephen.taylor at imm.ox.ac.uk> wrote:
>
>
>
> Hi Eleanor,
>
>
>
> Thanks very useful. Is the web site publically available?
>
>
>
> You mean http://idr-demo.openmicroscopy.org/webclient ?
>
>
>
>
>
> Cheers,
>
> ~Josh
>
>
>
>
>
> Kind regards and thanks,
>
>
>
>
>
> Steve
>
>
>
>
>
> From: Eleanor Williams [mailto:exwilliams at dundee.ac.uk]
>
> Sent: 07 June 2016 11:09
>
> To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>;
>
> e.x.williams at dundee.ac.uk; j.a.moore at dundee.ac.uk
>
> Subject: Re: Zegami / IDR
>
>
>
>
>
>
>
> Hi Stephen
>
>
>
> Here is a link to our workshop presentation but it describes how to
>
> get the data into omero rather than out.
>
>
>
> http://downloads.openmicroscopy.org/presentations/2016/Users-Meeting/Workshops/Metadata-at-Scale.pdf
>
>
>
> We will be adding the annotation.csv files as attachments to each
>
> screen over the next couple of weeks.
>
>
>
> Let us know if you have any more questions.
>
>
>
> Best regards
>
>
>
> Eleanor
>
>
>
>
>
>
>
> On 07/06/2016 10:41, Stephen Taylor wrote:
>
>
>
> Hi Eleanor and Josh,
>
>
>
> Great to meet you both at the OME meeting last week.
>
>
>
>
>
> I'd be interested in trying get some data into Zegami from IDR. I
>
> missed your Unmeeting (ironically I was demoing Zegami) where I think
>
> you went into some of the technical details. Can you send me the link
>
> to your presentation please?
>
>
>
>
>
> Kind regards and thanks,
>
> Steve
>
> _______________________________________________
>
> ome-devel mailing list
>
> ome-devel at lists.openmicroscopy.org.uk
>
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
> --
>
> Eleanor Williams PhD
>
>
>
> Data Annotator/Software Engineer
>
> Centre for Gene Regulation and Expression
>
> University of Dundee, UK
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
> --
> Eleanor Williams PhD
>
> Data Annotator/Software Engineer
> Centre for Gene Regulation and Expression
> University of Dundee, UK
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>