[ome-devel] Can get raw images (vs. compressed) for several datasets?

Tue May 16 20:27:37 BST 2017

Hey Josh - sorry for the delayed reply.  Passing on a couple of questions
from the engineer on our team that's working on this:

1. You mentioned that "One short-term option for accessing all the plate
data would be for you to make use of our jupyter instance" (
https://idr.openmicroscopy.org <https://idr.openmicroscopy.org/jupyter>/
jupyter <https://idr.openmicroscopy.org/jupyter>). I tried accessing that
using my personal GitHub account (cmclean) and alternatively received 403:
Forbidden and 500: Internal Server Error codes. Is there something else I
need to do to get access to the Jupyter instance?

2. You mentioned an alternative would be to use the connection() method in
the idr-notebooks/library/idr.py library, as depicted in
https://github.com/IDR/idr-notebooks/blob/master/notebooks/Getting_Started.
ipynb. I tried doing this after performing the installation notes at
https://www.openmicroscopy.org/site/support/omero5.3/developers/Python.html
(including
all of the OMERO.server installation requirements e.g. ICE). However, when
I try to connect to the server (host: localhost, username: omero, password:
omero) I get the following error:

ConnectionRefusedException: Ice.ConnectionRefusedException:
Connection refused

It seems to me that I still need to connect to a valid OMERO server that
holds the data, not the empty server on localhost (which I'm not sure why I
cannot connect to but that seems to be a different problem)... so I still
have the same question of " it looks like a valid username and password are
required to open a connection to your server.  How do we go about getting
one?"

Thanks for your advice Josh.

On Thu, May 4, 2017 at 3:21 AM, Josh Moore <josh at glencoesoftware.com> wrote:

> Hi Jon,
>
> On Thu, May 4, 2017 at 6:42 AM, Jon Hazard <jhazard at google.com> wrote:
> > Hey Josh - it would be far better for us if we could get access to full
> vs.
> > a partial datasets.
>
> Definitely understood, and it's something we're keen to enable as well.
>
>
> > Given that, to verify that I'm understanding how to
> > retrieve raw data correctly, could you confirm that something like the
> > following code is your suggestion for using the Python API to download
> raw
> > image data:
> >
> > from PIL import Image
> > # Do the following for each imageId we have in the dataset.
> > image = conn.getObject("Image", imageId)
> > for z in range(image.getSizeZ()):
> >   for c in range(image.getSizeC()):
> >     for t in range(image.getSizeT()):
> >       pixels = image.getPrimaryPixels()
> >       plane = pixels.getPlane(z, c, t)  # Get a numpy array.
> >       im = Image.fromArray(plane)  # Using PIL.Imagey array.
> >
> >       im.save("image.tiff")
>
> You might want to take a look at pixels.getPlanes to simplify the loop:
>
>  |  getPlanes(self, zctList)
>  |      Returns generator of numpy 2D planes from this set of pixels for a
>  |      list of Z, C, T indexes.
>  |
>  |      :param zctList:     A list of indexes: [(z,c,t), ]
>
>
> > Also, it looks like a valid username and password are required to open a
> > connection to your server.  How do we go about getting one?
>
> The `idr` library provides a `connection` method, e.g. from
> https://github.com/IDR/idr-notebooks/blob/master/
> notebooks/Getting_Started.ipynb
>
>     from idr import connection
>     conn = connection()
>
> --
>
> Cheers,
> ~Josh
>
>
> > On Wed, May 3, 2017 at 3:11 PM, Josh Moore <josh at glencoesoftware.com>
> wrote:
> >>
> >> Hi Jon,
> >>
> >> On Mon, May 1, 2017 at 11:50 PM, Jon Hazard <jhazard at google.com> wrote:
> >> > Hi Josh - replies to your questions inline below
> >> >
> >> > On Mon, May 1, 2017 at 1:19 PM, Josh Moore <josh at glencoesoftware.com>
> >> > wrote:
> >> >>
> >> >> (List members: sorry for the re-send)
> >> >>
> >> >> On Fri, Apr 28, 2017 at 8:07 PM, Jon Hazard <jhazard at google.com>
> wrote:
> >> >> > Hello - first, a hearty thanks to OME for making this data
> available!
> >> >>
> >> >> Gladly! And glad to you're interesting in using it.
> >> >>
> >> >>
> >> >> > My team is interested in the images from two datasets:
> >> >> >
> >> >> > idr0033-rohban-pathways/screenA
> >> >> >
> >> >> > and
> >> >> >
> >> >> > idr0028-pascualvargas-rhogtpases
> >> >> >
> >> >> > It looks like the images for at least the first data set are
> >> >> > compressed.
> >> >>
> >> >> How do you mean compressed? (Can you send a link?) The original data
> >> >> has been uploaded to the IDR but certainly for viewing, rendering is
> >> >> performed server-side before sending to the client.
> >> >
> >> >
> >> > To clarify, I meant that the images I can obtain are lower-resolution
> >> > JPGs
> >> > rather than the original-quality TIFF files. This is true whether I
> >> > export
> >> > through either the individual image download (by clicking the Down
> Arrow
> >> > as
> >> > shown in the first section of the export help page
> >> > http://help.openmicroscopy.org/export.html) or by using the public
> web
> >> > API
> >> > (e.g. by issuing the following command: curl
> >> >
> >> > "http://idr-demo.openmicroscopy.org/webclient/
> render_image/3194301/?c=1&q=1"
> >> >> image.jpg).
> >>
> >> I understand: there's not yet a webclient/ or webgateway/ yet that
> >> would return raw image planes. Our other APIs (e.g. Python, Java, C++)
> >> do. See below.
> >>
> >>
> >> >> > It would help us a lot to be able to work with the raw,
> uncompressed
> >> >> > images.
> >> >> > Are those available?
> >> >>
> >> >> Can you explain more about what you are trying to achieve? We're
> >> >> working on having full download enabled from EBI but until then we
> >> >> should be able to get you access to something more significant.
> >> >
> >> >
> >> > I would like to have access to the original-quality images for all
> >> > images in
> >> > the two datasets I mentioned previously:
> >> >
> >> >> idr0028-pascualvargas-rhogtpases and idr0033-rohban-pathways/screenA
> >> >
> >> > That is great that the full downloads will eventually be supported
> >> > through
> >> > EBI. Is there an expected timeframe for that capability to come
> online?
> >>
> >> I've checked on the status, but there's not yet a concrete date. My
> >> guess would be that it'll be on the order of weeks rather than months.
> >>
> >>
> >> > Assuming that there is some non-trivial infrastructure required there,
> >> > is
> >> > there any other way that we could have access to those two datasets
> in a
> >> > more timely manner?
> >>
> >> The OME project regularly makes sample data available for download [1]
> >> . To at least unblock you initially, I'd propose putting one plate for
> >> each study up on the downloads site (10GB and 40GB respectively), but
> >> we'd certainly like to avoid mirroring the dozens of TBs from IDR via
> >> that mechanism. One short-term option for accessing all the plate data
> >> would be for you to make use of our jupyter instance[2]. See
> >> https://github.com/IDR/idr-notebooks for examples.
> >>
> >> All the best,
> >> ~Josh
> >>
> >> [1] https://downloads.openmicroscopy.org/images/
> >> [2] https://idr.openmicroscopy.org/jupyter
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20170516/e207e436/attachment.html>