[ome-devel] OMERO URLs of original images from a plate

Mon Feb 13 15:55:24 GMT 2017

Hi Will,

thanks for your advice! I also just heard that it would be possible to have the analysis machine see (read-only) the same NFS share where OMERO is storing data. So it would be enough to dig out the paths to the original files (maybe adjust by mount point), no need to worry about building zip or downloads in general. It'll probably take a while to have the NFS set up, and I don't even have the analysis VM yet, but I'll start from the Python scripting side.

Regards,
Harri
________________________________________
From: ome-devel <ome-devel-bounces at lists.openmicroscopy.org.uk> on behalf of William Moore (Staff) <W.Moore at dundee.ac.uk>
Sent: 13 February 2017 12:36:54
To: OME External Developer List
Subject: Re: [ome-devel] OMERO URLs of original images from a plate

Hi Harri,

 In general I would imagine that downloading a whole plate of data and processing it locally is not the best solution.

 - By default, the OMERO web and Insight clients don’t allow you to download a whole plate of data because of the burden this puts on the server. In the case of web, the web code tries to create a single zip file (since the web can’t access your file system to download many separate files). However, for SPW data of any reasonable size, creating the zip will typically take longer than your web server (nginx/apache) timeout (1 minute) so you’d need to configure this to be a longer time.

 - If you download the SPW data and write a script to parse it, you have to treat every different SPW format differently. Whereas using the OMERO API, all different SPW formats would be accessed the same way.

 - I’m not so familiar with the way that CellProfiler accesses data from OMERO, but I imagine the strategy works for most users and that "several parallel downloads of individual images from the OMERO server” is not a problem.

 - When accessing single channels and images of a Plate, you will not be unnecessarily downloading any extra images or channels that you don’t need.

To enable download of Plates, see the configuration of omero.policy.binary_access:
 https://www.openmicroscopy.org/site/support/omero5.2/sysadmins/config.html#omero-policy-binary-access
E.g: to allow plate download:
$ bin/omero config set omero.policy.binary_access "+read,+write,+image,+plate”
(you will probably need to restart OMERO after this).

Then you should be able to download the plate from any image ID in the plate (every image is linked to all the SPW files).
This url will build and download the plate zip for the specified image:
/webgateway/archived_files/download/?image=62845&zipname=myData.zip

In your original strategy of building a list of image IDs from a plate,
I would suggest a python OMERO.script, which can be called from the web or Insight client to create a .csv file to download.

Using the python Blitz Gateway it’s quite easy to list images for a plate:

plate = conn.getObject("Plate", 5017)

for well in plate.listChildren():
     print "Well: ", well.id
     for well_sample in well.listChildren():
         print "  ws:", well_sample.id
         image = well_sample.getImage()
         print "  image:", image.id, image.name

A .csv file could be attached to the Plate object and returned by the script.
See “How to create a file annotation…” at https://www.openmicroscopy.org/site/support/omero5.2/developers/Python.html

See also “Introduction to OMERO scripts” at https://www.openmicroscopy.org/site/support/omero5.2/developers/scripts/index.html
If you need any help turning this into an OMERO.script let us know.

Hope this helps, whichever strategy you find works best,

 Regards,

   Will.

On 12 Feb 2017, at 06:28, Jäälinoja, Harri Tapio <harri.jaalinoja at helsinki.fi<mailto:harri.jaalinoja at helsinki.fi>> wrote:

Hi Ola,

thanks a lot for your reply! With merged images I mean the ones you see in the plate viewer, not individual C01 per channel, but a multichannel image.

Anyway, after sleeping over this a couple of nights, I started to wonder if what I was trying to do was a good idea to begin with. Let's say I have CellProfiler on a cluster, and I start to process an image list with "omero::iid=nnn" entries. That would mean several parallel downloads of individual images from the OMERO server, right? I noticed there's a link in the OMERO plate viewer to download the entire plate in a an archive. Can I find the archive URL programmatically? Then I could just download that once, unzip and sort out the paths in a script. And exclude images with an error tag found with your second example.

I'll try this tomorrow. Thanks again!

Best,
Harri

________________________________________
From: ome-devel <ome-devel-bounces at lists.openmicroscopy.org.uk<mailto:ome-devel-bounces at lists.openmicroscopy.org.uk>> on behalf of Aleksandra Tarkowska (Staff) <A.Tarkowska at dundee.ac.uk<mailto:A.Tarkowska at dundee.ac.uk>>
Sent: 10 February 2017 17:31:57
To: OME External Developer List
Subject: Re: [ome-devel] OMERO URLs of original images from a plate

Hi

What do you mean by merged images? Did you have a chance to check HCS structures (Screen, Plate and Well) of the OME Model in http://www.openmicroscopy.org/site/support/ome-model/developers/model-overview.html?

If you query DB directly to see how that is structured you could try something like

SQL:
select w.id as well_id, ws.well_index as field, i.id as image_id
from image as i
   join wellsample as ws on (ws.image=i.id)
   join well as w on (ws.well=w.id)
   join plate as p on (w.plate=p.id)
   join screenplatelink as spl on (spl.child=p.id)
where spl.parent = :screen_id

Using omero api you could try HQL query and filter by tag value:

ParametersI param = new ParametersI();
param.map.put("sid", omero.rtypes.rlong(102));

StringBuffer sb = new StringBuffer();
sb.append("select new map( w.id as wellid, index(ws) as field, i.id as imgid) ");
sb.append("from Well w ");
sb.append("join w.wellSamples ws ");
sb.append("join ws.image i ");
sb.append("join i.annotationLinks ial ");
sb.append("join ial.child tag ");
sb.append("join w.plate pl ");
sb.append("join pl.screenLinks sl ");
sb.append("join sl.parent s ");
sb.append("where s.id = :sid “);
sb.append("and tag.class = TagAnnotation and tag.textValue = 'error' “);

for (final List<RType> result : queryService.projection(sb.toString(), param)) {
   System.out.println(omero.rtypes.unwrap(result.get(0)));
}

depends on which version of OMERO.jar you are using annotation discriminator may be /basic/text/tag/ (this was fixed in 5.3-develop to map the class TagAnnotation)

Hope that helps

Kind regards

Ola
Software Engineer
Open Microscopy Environment
University of Dundee

On 9 Feb 2017, at 15:36, Jäälinoja, Harri Tapio <harri.jaalinoja at helsinki.fi<mailto:harri.jaalinoja at helsinki.fi><mailto:harri.jaalinoja at helsinki.fi>> wrote:

Hi,

this is continuation of my previous question about browsing a plate with the Java API. I've uploaded a couple of plates, in the database it looks like this:

select id,name from image order by id;

id  |                              name
------+-----------------------------------------------------------------
  1 | LMU-CELLINSIGHT_170111150001_B03f00d0.C01 [Well B03, Field #00]
...
1920 | LMU-CELLINSIGHT_150428120001_B05f00d0.C01 [Well B05, Field #15]
(1920 rows)

So the database only has the merged images of each field in well. The original files are on the OMERO server, but it seems it's not possible to find them with a URL like "omero::iid=123"?

What I have in mind is to pick a plateID and build a corresponding CSV formatted image list for CellProfiler LoadData module (see https://github.com/CellProfiler/CellProfiler/wiki/OMERO:-Accessing-images-from-CellProfiler). Something like:

URL_DAPI,URL_GFP,URL_CY5
omero::iid=123,omero::iid=124,omero::iid=125

(Ideally the image list would exclude fields tagged by the user as imaging errors.)
Is this possible at the moment? If not, would it a reasonable request?

Best,
Harri

---
Harri Jäälinoja
Light Microscopy Unit
Institute of Biotechnology
University of Helsinki
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk><mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

The University of Dundee is a registered Scottish Charity, No: SC015096