[ome-devel] Fwd: [IDR Project] Zegami / IDR

Tue Jun 21 06:57:27 BST 2016

Hi Josh and Eleanor,

Thanks for these pointers. We are going to try a few things out and get back to you.

Kind regards and thanks,

Steve
=======================================
Head of Computational Biology Research Group 
Weatherall Institute of Molecular Medicine 
University of Oxford  
+44 1865 222640
www.cbrg.ox.ac.uk  

> -----Original Message-----
> From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
> On Behalf Of Josh Moore
> Sent: 17 June 2016 14:44
> To: OME External Developer List <ome-devel at lists.openmicroscopy.org.uk>
> Subject: Re: [ome-devel] Fwd: [IDR Project] Zegami / IDR
> 
> On Fri, Jun 17, 2016 at 3:05 PM, Eleanor Williams <exwilliams at dundee.ac.uk>
> wrote:
> > Hi Steve,
> >
> > You can et the current annotation.csv file for
> > idr0002-heriche-condensation here
> > https://github.com/IDR/idr-metadata/tree/master/idr0002-heriche-conden
> > sation/screenA if you want to try it out.  If you want to see how a
> > larger file scales then idr0013-neumann-mitocheck/screenA is a good
> > one to look at
> > https://github.com/IDR/idr-metadata/tree/master/idr0013-neumann-
> mitocheck/screenA.
> > Again I'll be updating these files to include species etc soon, but
> > they should still be useful as they are.
> >
> > There are no references to image ids in these files.  Josh will write
> > to you separately about how to link the two.
> 
> Taking the first row from
> https://github.com/IDR/idr-metadata/blob/master/idr0002-heriche-
> condensation/screenA/idr0002-screenA-annotation.csv
> the following snippet will give you the related image object:
> 
>     import omero
>     from omero.rtypes import unwrap
>     c = omero.client(ip)
>     c.createSession(name, user)
>     from omero.gateway import BlitzGateway
>     conn = BlitzGateway(client_obj=c)
>     plate = conn.getObject("Plate", 422)
>     well = plate.getWellGrid()[0][0]
>     image = well.getImage()
>     map = image.getAnnotation()
>     print map.getMapValueAsMap()
> 
>     {'Gene Identifier': 'ENSG00000117399',
>      'Gene Identifier URL': 'http://www.ensembl.org/id/ENSG00000117399',
>      'Gene Symbol': 'CDC20',
>      'siRNA Identifier': 's2748'}
> 
> since well A1 is the first-row, first-column (i.e. [0][0]) of the well grid.
> 
> Cheers,
> ~Josh
> 
> 
> 
> 
> 
> > A plate viewer would be useful for seeing if there are any technical
> > problems across the plate.
> >
> > Best regards
> >
> > Eleanor
> >
> >
> >
> >
> > On 17/06/2016 10:03, Stephen Taylor wrote:
> >
> > Hi Eleanor,
> >
> >
> >
> > I think idr0002-heriche-condensation would be a good screen to start with.
> > It is relatively small at 12 plates and has good annotation with
> > control, quality control and phenotype values.  It can be found here
> > http://idr-demo.openmicroscopy.org/webclient/?show=screen-102.
> >
> > These look good. Thanks!
> >
> >
> >
> > I am doing a bit of a stock-take of the annotation files I've produced
> > so far and will be changing a few aspects over the next few weeks,
> > mainly to include more of the information that is currently held in the study
> files.
> > For example I'd like to get more of the sample attributes such as
> > species and cell line into the annotation.csv file, and then into the
> > omero tables and Map Annotations.  Sample attributes for screens were
> > originally recorded in the study file because it didn't vary per
> > screen, but now that we are also looking at non-screen data where this
> > might vary it makes sense to always have it visible with the images.
> > Also we haven't got as far as exposing much of the study information in
> omero yet.
> >
> > I would really like to work with you to understand how you are going
> > to use the experimental and analytic annotations and how we can help
> > in providing the metadata you need and in the most usable format.  For
> > example would it be useful if I indicate what is sample attribute
> > information in the annotation.csv file by using something like a constistent
> header format e.g.
> > Characterisics [Organism], Characteristics [Cell Line],
> > Characteristics [Organism Part] etc, or is this not really necessary
> > for your purposes and 'Organism', 'Cell Line' column headings would be
> enough?
> >
> >
> >
> > Key information would be OMERO id of the thumbnail (to potentially
> > stream) and OMERO id so we can link back to the full image in the
> > OMERO database
> >
> > So then we can construct URLs like this:
> >
> > http://idr-
> demo.openmicroscopy.org/webgateway/render_thumbnail/179717/
> > 96/
> >
> > http://idr-demo.openmicroscopy.org/webclient/img_detail/179715/
> >
> > Plate, Well id (or number) is also pretty useful. Do you think we
> > would need to write a Plate Viewer to organise the tiles for the user?
> >
> > I would imagine in a lot of studies we won’t have control on how they
> > are annotated. Zegami can read most data sources by writing an
> > appropriate data loader plugin, so I think the issue is how fast can
> > we get the metadata out of OMERO and what level of caching would be
> > required on the Zegami side. For example, if the data is in available
> > as a single flat table (such as
> > .csv/.tsv) format it should be feasible to load a collection of this
> > size dynamically (1152 wells) since I did something similar by reading
> > a shared Google Spreadsheet. Very large metadata files will be a bigger
> problem.
> >
> > With regards to other information needed I think it depends on the
> > biological question. For example, this experiment is published so the
> > analysis has been done and summary statistics produced for each well
> > (Quality Control, Median Deviation Fraction - Shorter Prophase, Median
> > Deviation Fraction - Longer Prophase, Phenotype Reproducibility -
> > Shorter Prophase etc) and these could all be imported in as a mixture
> > of categorical and numeric data and then sorted and filtered. i.e.
> > show me all the wells where Quality Control is “pass” and then sort on
> > “Median Deviation Fraction
> > - Longer Prophase:”. The scientist then can review the images to
> > identify candidate wells, then link to the time series in OMERO to the
> > well of interest. You could also feed in a whole bunch of other
> > statistics to help verify the methods during development of the
> > screen. For example, in this case the authors use CellCognition
> > software to classify nuclei stages (interphase, early prophase, late
> > prophase, prometaphase, metaphase) and you could verify how well the
> > algorithms are doing by having a %nuclei stage for each well. This
> > would allow you to look for wells that had more cells in prometaphase
> > for example and check the analysis is correct. You could also add in various
> gene annotations, and other metadata that was useful.
> >
> >
> >
> > Finally, it was Balaji Ramalingam that you were talking to about
> > streaming the thumbnails.  I'm sure he would be happy to discuss this
> > further with you.
> >
> >
> >
> > Ok great.
> >
> > Please let me know if you have any further questions.
> >
> > Kind regards,
> >
> > Steve
> >
> >
> >
> > Best regards
> >
> > Eleanor
> >
> >
> >
> >
> >
> >
> >
> > On 15/06/2016 20:59, Josh Moore wrote:
> >
> > Forwarding this to the mailing list so that anyone who had expressed
> >
> > interested in http://zegami.com/ at this year's users' meeting can
> >
> > follow along.
> >
> >
> >
> > Cheers,
> >
> > ~Josh
> >
> >
> >
> > P.S. If you weren't able to attend #OME2016, a recap is now up on the
> >
> > blog http://blog.openmicroscopy.org/community/2016/06/15/user-meeting/
> >
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> >
> > From: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>
> >
> > Date: Tue, Jun 14, 2016 at 8:17 AM
> >
> > Subject: Re: [IDR Project] Zegami / IDR
> >
> > Cc: Roger Noble <roger at zegami.com>
> >
> >
> >
> > Hi,
> >
> >
> >
> > I’ve been look at a few examples from the link Josh sent to me and
> >
> > there are quite a lot! What would be a good data set to get import
> >
> > into Zegami from IDR?  I am thinking a set that has decent metadata
> >
> > and a study that people are more familiar with that would make an
> >
> > interesting use case.
> >
> >
> >
> > Kind regards,
> >
> > Steve
> >
> >
> >
> >
> >
> > From: Stephen Taylor
> >
> > Sent: 10 June 2016 14:25
> >
> > Cc: Roger Noble <roger at zegami.com>; 'e.x.williams at dundee.ac.uk'
> >
> > <e.x.williams at dundee.ac.uk>
> >
> > Subject: RE: [IDR Project] Zegami / IDR
> >
> >
> >
> > Hi all,
> >
> >
> >
> > I am interested generally in getting data in and out of OMERO from
> >
> > both internal and external. Over a WAN the images are less of an issue
> >
> > since they are streamed on demand but large metadata on the other hand
> >
> > can get out of hand since it all has to be loaded completely into
> >
> > Zegami’s database.
> >
> >
> >
> > Currently I do “OMERO project” to “Zegami collection” using a Zegami
> >
> > export script (modified version of the OMERO batch export script).
> >
> > This takes a project or selection of images from OMERO and produces
> >
> > thumbnails and data in tab delimited Zegami format and then I manually
> >
> > import that into Zegami for processing. The OMERO id is preserved so I
> >
> > can link to the original image in OMERO using the OMERO web client. I
> >
> > only export certain metadata currently (filenames and OMERO tags) and
> >
> > my plan is to make this more comprehensive and allow MapAnnotations
> >
> > and Tables to be exported using this mechanism as well. I need to find
> >
> > out more how to query these data types.  Since we have internal
> >
> > projects at Oxford that need this I’d like to finish this and make
> >
> > that plugin freely available for other OMERO users to publish their
> >
> > data in Zegami.
> >
> >
> >
> >
> >
> > Longer term for IDR/Zegami
> >
> >
> >
> >
> >
> > One of your team (apologies I didn’t get his name but he was doing
> >
> > OMERO/ImageJ in the Unconference session) suggested we just stream in
> >
> > the thumbnail and just have 1 level (i.e. no zoom) and when you want
> >
> > to see the image just open it up in OMERO. This would not be as nice
> >
> > to use but may be sufficient for HCS and would allow you to use
> >
> > Zegami’s facet based query engine and would be probably little (no?)
> >
> > work for you guys. We need to test this to see what it looks like and
> >
> > it assumes OMERO can supply a unique thumbnail URL for each image and
> >
> > that the metadata will provide a primary key so that we can get to
> >
> > this image via a thumbnail id. To get at the metadata Josh and Eleanor
> >
> > said they were thinking of making the CSV downloadable for a
> >
> > collection, which could be loaded into Zegami on the fly (we have
> >
> > instances where we dynamically load metadata via URL already).
> >
> > However, major caveat is on how well this would perform over a WAN
> >
> > with a large amount of data.
> >
> >
> >
> > BTW we have developer docs for Zegami for those interested at:
> >
> >
> >
> > https://support.zegami.com/hc/en-us/categories/200350657-Developers
> >
> >
> >
> > and would welcome feedback.
> >
> >
> >
> > Kind regards and thanks,
> >
> >
> >
> > Steve
> >
> >
> >
> >
> >
> > From: Jason Swedlow (Staff) [mailto:j.r.swedlow at dundee.ac.uk]
> >
> > Sent: 08 June 2016 13:09
> >
> > Cc: Eleanor Williams (Staff) <e.x.williams at dundee.ac.uk>
> >
> > Subject: Re: [IDR Project] Zegami / IDR
> >
> >
> >
> >
> >
> > Hi Josh et al—
> >
> >
> >
> > My two cents— can we consider this use case from an external users’
> >
> > perspective, mindful that distribution of such a solution would
> >
> > definitely cause us to  consider how we ship thumbnails, handle
> >
> > straws, etc.
> >
> >
> >
> > Cheers,
> >
> > Jason
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Jun 8, 2016 at 1:31 PM, Josh Moore <josh at glencoesoftware.com>
> wrote:
> >
> >
> >
> > Stephen,
> >
> >
> >
> >
> >
> > On Wed, Jun 8, 2016 at 9:52 AM, Stephen Taylor
> >
> > <stephen.taylor at imm.ox.ac.uk> wrote:
> >
> >
> >
> > Hi Josh,
> >
> >
> >
> >
> >
> > Yes! What would be the best way of bulk downloading the thumbnails for
> > a screen or plate
> >
> >
> >
> > Interesting question. Guess the reverse question is: do you want to do
> >
> > this as an external user or as if you had access to the OMERO?
> >
> >
> >
> >
> >
> > ( I notice the "Run Script" button isn't active ).
> >
> >
> >
> > Correct. This server is stripped down in several ways as a public resource.
> >
> > ~J.
> >
> >
> >
> >
> >
> > Kind regards,
> >
> > Steve
> >
> >
> >
> >
> >
> > -----Original Message-----
> >
> > From: Josh Moore [mailto:josh at glencoesoftware.com]
> >
> > Sent: 08 June 2016 08:07
> >
> > To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>
> >
> > Cc: Eleanor Williams <exwilliams at dundee.ac.uk>;
> >
> > e.x.williams at dundee.ac.uk; j.a.moore at dundee.ac.uk
> >
> >
> >
> > Subject: Re: Zegami / IDR
> >
> >
> >
> >
> >
> > Hi Stephen,
> >
> >
> >
> >
> >
> > On Wed, Jun 8, 2016 at 8:40 AM, Stephen Taylor
> >
> > <stephen.taylor at imm.ox.ac.uk> wrote:
> >
> >
> >
> > Hi Eleanor,
> >
> >
> >
> > Thanks very useful. Is the web site publically available?
> >
> >
> >
> > You mean http://idr-demo.openmicroscopy.org/webclient ?
> >
> >
> >
> >
> >
> > Cheers,
> >
> > ~Josh
> >
> >
> >
> >
> >
> > Kind regards and thanks,
> >
> >
> >
> >
> >
> > Steve
> >
> >
> >
> >
> >
> > From: Eleanor Williams [mailto:exwilliams at dundee.ac.uk]
> >
> > Sent: 07 June 2016 11:09
> >
> > To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk>;
> >
> > e.x.williams at dundee.ac.uk; j.a.moore at dundee.ac.uk
> >
> > Subject: Re: Zegami / IDR
> >
> >
> >
> >
> >
> >
> >
> > Hi Stephen
> >
> >
> >
> > Here is a link to our workshop presentation but it describes how to
> >
> > get the data into omero rather than out.
> >
> >
> >
> > http://downloads.openmicroscopy.org/presentations/2016/Users-
> Meeting/W
> > orkshops/Metadata-at-Scale.pdf
> >
> >
> >
> > We will be adding the annotation.csv files as attachments to each
> >
> > screen over the next couple of weeks.
> >
> >
> >
> > Let us know if you have any more questions.
> >
> >
> >
> > Best regards
> >
> >
> >
> > Eleanor
> >
> >
> >
> >
> >
> >
> >
> > On 07/06/2016 10:41, Stephen Taylor wrote:
> >
> >
> >
> > Hi Eleanor and Josh,
> >
> >
> >
> > Great to meet you both at the OME meeting last week.
> >
> >
> >
> >
> >
> > I'd be interested in trying get some data into Zegami from IDR. I
> >
> > missed your Unmeeting (ironically I was demoing Zegami) where I think
> >
> > you went into some of the technical details. Can you send me the link
> >
> > to your presentation please?
> >
> >
> >
> >
> >
> > Kind regards and thanks,
> >
> > Steve
> >
> > _______________________________________________
> >
> > ome-devel mailing list
> >
> > ome-devel at lists.openmicroscopy.org.uk
> >
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> >
> >
> > --
> >
> > Eleanor Williams PhD
> >
> >
> >
> > Data Annotator/Software Engineer
> >
> > Centre for Gene Regulation and Expression
> >
> > University of Dundee, UK
> >
> >
> > The University of Dundee is a registered Scottish Charity, No:
> > SC015096
> >
> >
> >
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> >
> > --
> > Eleanor Williams PhD
> >
> > Data Annotator/Software Engineer
> > Centre for Gene Regulation and Expression University of Dundee, UK
> >
> >
> > The University of Dundee is a registered Scottish Charity, No: SC015096
> >
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel