[ome-devel] Fwd: [IDR Project] Zegami / IDR

Fri Jun 17 14:05:55 BST 2016

Hi Steve,

You can et the current annotation.csv file for idr0002-heriche-condensation here https://github.com/IDR/idr-metadata/tree/master/idr0002-heriche-condensation/screenA if you want to try it out.  If you want to see how a larger file scales then idr0013-neumann-mitocheck/screenA is a good one to look at https://github.com/IDR/idr-metadata/tree/master/idr0013-neumann-mitocheck/screenA. Again I'll be updating these files to include species etc soon, but they should still be useful as they are.

There are no references to image ids in these files.  Josh will write to you separately about how to link the two.

A plate viewer would be useful for seeing if there are any technical problems across the plate.

Best regards

Eleanor

On 17/06/2016 10:03, Stephen Taylor wrote:
Hi Eleanor,

I think idr0002-heriche-condensation would be a good screen to start with.  It is relatively small at 12 plates and has good annotation with control, quality control and phenotype values.  It can be found here http://idr-demo.openmicroscopy.org/webclient/?show=screen-102.

These look good. Thanks!

I am doing a bit of a stock-take of the annotation files I've produced so far and will be changing a few aspects over the next few weeks, mainly to include more of the information that is currently held in the study files.  For example I'd like to get more of the sample attributes such as species and cell line into the annotation.csv file, and then into the omero tables and Map Annotations.  Sample attributes for screens were originally recorded in the study file because it didn't vary per screen, but now that we are also looking at non-screen data where this might vary it makes sense to always have it visible with the images.  Also we haven't got as far as exposing much of the study information in omero yet.

I would really like to work with you to understand how you are going to use the experimental and analytic annotations and how we can help in providing the metadata you need and in the most usable format.  For example would it be useful if I indicate what is sample attribute information in the annotation.csv file by using something like a constistent header format e.g. Characterisics [Organism], Characteristics [Cell Line], Characteristics [Organism Part] etc, or is this not really necessary for your purposes and 'Organism', 'Cell Line' column headings would be enough?

Key information would be OMERO id of the thumbnail (to potentially stream) and OMERO id so we can link back to the full image in the OMERO database

So then we can construct URLs like this:

<http://idr-demo.openmicroscopy.org/webgateway/render_thumbnail/179717/96/>http://idr-demo.openmicroscopy.org/webgateway/render_thumbnail/179717/96/

<http://idr-demo.openmicroscopy.org/webclient/img_detail/179715/>http://idr-demo.openmicroscopy.org/webclient/img_detail/179715/

Plate, Well id (or number) is also pretty useful. Do you think we would need to write a Plate Viewer to organise the tiles for the user?

I would imagine in a lot of studies we won’t have control on how they are annotated. Zegami can read most data sources by writing an appropriate data loader plugin, so I think the issue is how fast can we get the metadata out of OMERO and what level of caching would be required on the Zegami side. For example, if the data is in available as a single flat table (such as .csv/.tsv) format it should be feasible to load a collection of this size dynamically (1152 wells) since I did something similar by reading a shared Google Spreadsheet. Very large metadata files will be a bigger problem.

With regards to other information needed I think it depends on the biological question. For example, this experiment is published so the analysis has been done and summary statistics produced for each well (Quality Control, Median Deviation Fraction - Shorter Prophase, Median Deviation Fraction - Longer Prophase, Phenotype Reproducibility - Shorter Prophase etc) and these could all be imported in as a mixture of categorical and numeric data and then sorted and filtered. i.e. show me all the wells where Quality Control is “pass” and then sort on “Median Deviation Fraction - Longer Prophase:”. The scientist then can review the images to identify candidate wells, then link to the time series in OMERO to the well of interest. You could also feed in a whole bunch of other statistics to help verify the methods during development of the screen. For example, in this case the authors use CellCognition software to classify nuclei stages (interphase, early prophase, late prophase, prometaphase, metaphase) and  you could verify how well the algorithms are doing by having a %nuclei stage for each well. This would allow you to look for wells that had more cells in prometaphase for example and check the analysis is correct. You could also add in various gene annotations, and other metadata that was useful.

Finally, it was Balaji Ramalingam<https://www.openmicroscopy.org/site/about/development-teams/jason/balaji.png/view> that you were talking to about streaming the thumbnails.  I'm sure he would be happy to discuss this further with you.

Ok great.

Please let me know if you have any further questions.

Kind regards,

Steve

Best regards

Eleanor

On 15/06/2016 20:59, Josh Moore wrote:

Forwarding this to the mailing list so that anyone who had expressed

interested in http://zegami.com/ at this year's users' meeting can

follow along.

Cheers,

~Josh

P.S. If you weren't able to attend #OME2016, a recap is now up on the

blog http://blog.openmicroscopy.org/community/2016/06/15/user-meeting/

---------- Forwarded message ----------

From: Stephen Taylor <stephen.taylor at imm.ox.ac.uk><mailto:stephen.taylor at imm.ox.ac.uk>

Date: Tue, Jun 14, 2016 at 8:17 AM

Subject: Re: [IDR Project] Zegami / IDR

Cc: Roger Noble <roger at zegami.com><mailto:roger at zegami.com>

Hi,

I’ve been look at a few examples from the link Josh sent to me and

there are quite a lot! What would be a good data set to get import

into Zegami from IDR?  I am thinking a set that has decent metadata

and a study that people are more familiar with that would make an

interesting use case.

Kind regards,

Steve

From: Stephen Taylor

Sent: 10 June 2016 14:25

Cc: Roger Noble <roger at zegami.com><mailto:roger at zegami.com>; 'e.x.williams at dundee.ac.uk<mailto:e.x.williams at dundee.ac.uk>'

<e.x.williams at dundee.ac.uk><mailto:e.x.williams at dundee.ac.uk>

Subject: RE: [IDR Project] Zegami / IDR

Hi all,

I am interested generally in getting data in and out of OMERO from

both internal and external. Over a WAN the images are less of an issue

since they are streamed on demand but large metadata on the other hand

can get out of hand since it all has to be loaded completely into

Zegami’s database.

Currently I do “OMERO project” to “Zegami collection” using a Zegami

export script (modified version of the OMERO batch export script).

This takes a project or selection of images from OMERO and produces

thumbnails and data in tab delimited Zegami format and then I manually

import that into Zegami for processing. The OMERO id is preserved so I

can link to the original image in OMERO using the OMERO web client. I

only export certain metadata currently (filenames and OMERO tags) and

my plan is to make this more comprehensive and allow MapAnnotations

and Tables to be exported using this mechanism as well. I need to find

out more how to query these data types.  Since we have internal

projects at Oxford that need this I’d like to finish this and make

that plugin freely available for other OMERO users to publish their

data in Zegami.

Longer term for IDR/Zegami

One of your team (apologies I didn’t get his name but he was doing

OMERO/ImageJ in the Unconference session) suggested we just stream in

the thumbnail and just have 1 level (i.e. no zoom) and when you want

to see the image just open it up in OMERO. This would not be as nice

to use but may be sufficient for HCS and would allow you to use

Zegami’s facet based query engine and would be probably little (no?)

work for you guys. We need to test this to see what it looks like and

it assumes OMERO can supply a unique thumbnail URL for each image and

that the metadata will provide a primary key so that we can get to

this image via a thumbnail id. To get at the metadata Josh and Eleanor

said they were thinking of making the CSV downloadable for a

collection, which could be loaded into Zegami on the fly (we have

instances where we dynamically load metadata via URL already).

However, major caveat is on how well this would perform over a WAN

with a large amount of data.

BTW we have developer docs for Zegami for those interested at:

https://support.zegami.com/hc/en-us/categories/200350657-Developers

and would welcome feedback.

Kind regards and thanks,

Steve

From: Jason Swedlow (Staff) [mailto:j.r.swedlow at dundee.ac.uk]

Sent: 08 June 2016 13:09

Cc: Eleanor Williams (Staff) <e.x.williams at dundee.ac.uk><mailto:e.x.williams at dundee.ac.uk>

Subject: Re: [IDR Project] Zegami / IDR

Hi Josh et al—

My two cents— can we consider this use case from an external users’

perspective, mindful that distribution of such a solution would

definitely cause us to  consider how we ship thumbnails, handle

straws, etc.

Cheers,

Jason

On Wed, Jun 8, 2016 at 1:31 PM, Josh Moore <josh at glencoesoftware.com><mailto:josh at glencoesoftware.com> wrote:

Stephen,

On Wed, Jun 8, 2016 at 9:52 AM, Stephen Taylor

<stephen.taylor at imm.ox.ac.uk><mailto:stephen.taylor at imm.ox.ac.uk> wrote:

Hi Josh,

Yes! What would be the best way of bulk downloading the thumbnails for a screen or plate

Interesting question. Guess the reverse question is: do you want to do

this as an external user or as if you had access to the OMERO?

( I notice the "Run Script" button isn't active ).

Correct. This server is stripped down in several ways as a public resource.

~J.

Kind regards,

Steve

-----Original Message-----

From: Josh Moore [mailto:josh at glencoesoftware.com]

Sent: 08 June 2016 08:07

To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk><mailto:stephen.taylor at imm.ox.ac.uk>

Cc: Eleanor Williams <exwilliams at dundee.ac.uk><mailto:exwilliams at dundee.ac.uk>;

e.x.williams at dundee.ac.uk<mailto:e.x.williams at dundee.ac.uk>; j.a.moore at dundee.ac.uk<mailto:j.a.moore at dundee.ac.uk>

Subject: Re: Zegami / IDR

Hi Stephen,

On Wed, Jun 8, 2016 at 8:40 AM, Stephen Taylor

<stephen.taylor at imm.ox.ac.uk><mailto:stephen.taylor at imm.ox.ac.uk> wrote:

Hi Eleanor,

Thanks very useful. Is the web site publically available?

You mean http://idr-demo.openmicroscopy.org/webclient ?

Cheers,

~Josh

Kind regards and thanks,

Steve

From: Eleanor Williams [mailto:exwilliams at dundee.ac.uk]

Sent: 07 June 2016 11:09

To: Stephen Taylor <stephen.taylor at imm.ox.ac.uk><mailto:stephen.taylor at imm.ox.ac.uk>;

e.x.williams at dundee.ac.uk<mailto:e.x.williams at dundee.ac.uk>; j.a.moore at dundee.ac.uk<mailto:j.a.moore at dundee.ac.uk>

Subject: Re: Zegami / IDR

Hi Stephen

Here is a link to our workshop presentation but it describes how to

get the data into omero rather than out.

http://downloads.openmicroscopy.org/presentations/2016/Users-Meeting/Workshops/Metadata-at-Scale.pdf

We will be adding the annotation.csv files as attachments to each

screen over the next couple of weeks.

Let us know if you have any more questions.

Best regards

Eleanor

On 07/06/2016 10:41, Stephen Taylor wrote:

Hi Eleanor and Josh,

Great to meet you both at the OME meeting last week.

I'd be interested in trying get some data into Zegami from IDR. I

missed your Unmeeting (ironically I was demoing Zegami) where I think

you went into some of the technical details. Can you send me the link

to your presentation please?

Kind regards and thanks,

Steve

_______________________________________________

ome-devel mailing list

ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>

http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

--

Eleanor Williams PhD

Data Annotator/Software Engineer

Centre for Gene Regulation and Expression

University of Dundee, UK

The University of Dundee is a registered Scottish Charity, No: SC015096

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

--
Eleanor Williams PhD

Data Annotator/Software Engineer
Centre for Gene Regulation and Expression
University of Dundee, UK

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20160617/2e649689/attachment.html>