[ome-devel] Can get raw images (vs. compressed) for several datasets?

Tue May 16 21:02:44 BST 2017

Hi Jon & Cory,

On Tue, May 16, 2017 at 9:27 PM, Jon Hazard <jhazard at google.com> wrote:
> Hey Josh - sorry for the delayed reply.  Passing on a couple of questions
> from the engineer on our team that's working on this:
>
> 1. You mentioned that "One short-term option for accessing all the plate
> data would be for you to make use of our jupyter instance"
> (https://idr.openmicroscopy.org/jupyter). I tried accessing that using my
> personal GitHub account (cmclean) and alternatively received 403: Forbidden
> and 500: Internal Server Error codes. Is there something else I need to do
> to get access to the Jupyter instance?

With our latest release, membership in a GitHub org (idr-contrib) is
required for accessing the resource. I've invited cmclean; Cory will
need to accept the invitation and set membership status to "public"
for the oauth plugin to function.

> 2. You mentioned an alternative would be to use the connection() method in
> the idr-notebooks/library/idr.py library, as depicted in
> https://github.com/IDR/idr-notebooks/blob/master/notebooks/Getting_Started.ipynb.
> I tried doing this after performing the installation notes at
> https://www.openmicroscopy.org/site/support/omero5.3/developers/Python.html
> (including all of the OMERO.server installation requirements e.g. ICE).
> However, when I try to connect to the server (host: localhost, username:
> omero, password: omero) I get the following error:
>
> ConnectionRefusedException: Ice.ConnectionRefusedException:
> Connection refused

Those servers are not accessible externally and are only intended for
use within the EBI Embassy Cloud.

> It seems to me that I still need to connect to a valid OMERO server that
> holds the data, not the empty server on localhost (which I'm not sure why I
> cannot connect to but that seems to be a different problem)... so I still
> have the same question of " it looks like a valid username and password are
> required to open a connection to your server.  How do we go about getting
> one?"

The connection in idr.py is the correct one, but only works within the
Jupyter environment.

Sorry for any confusion.
~Josh.

p.s. incidentally EBI has been in touch about opening Aspera
downloads. Hopefully I'll have connection info for you to test soon.

> Thanks for your advice Josh.
>
>
> On Thu, May 4, 2017 at 3:21 AM, Josh Moore <josh at glencoesoftware.com> wrote:
>>
>> Hi Jon,
>>
>> On Thu, May 4, 2017 at 6:42 AM, Jon Hazard <jhazard at google.com> wrote:
>> > Hey Josh - it would be far better for us if we could get access to full
>> > vs.
>> > a partial datasets.
>>
>> Definitely understood, and it's something we're keen to enable as well.
>>
>>
>> > Given that, to verify that I'm understanding how to
>> > retrieve raw data correctly, could you confirm that something like the
>> > following code is your suggestion for using the Python API to download
>> > raw
>> > image data:
>> >
>> > from PIL import Image
>> > # Do the following for each imageId we have in the dataset.
>> > image = conn.getObject("Image", imageId)
>> > for z in range(image.getSizeZ()):
>> >   for c in range(image.getSizeC()):
>> >     for t in range(image.getSizeT()):
>> >       pixels = image.getPrimaryPixels()
>> >       plane = pixels.getPlane(z, c, t)  # Get a numpy array.
>> >       im = Image.fromArray(plane)  # Using PIL.Imagey array.
>> >
>> >       im.save("image.tiff")
>>
>> You might want to take a look at pixels.getPlanes to simplify the loop:
>>
>>  |  getPlanes(self, zctList)
>>  |      Returns generator of numpy 2D planes from this set of pixels for a
>>  |      list of Z, C, T indexes.
>>  |
>>  |      :param zctList:     A list of indexes: [(z,c,t), ]
>>
>>
>> > Also, it looks like a valid username and password are required to open a
>> > connection to your server.  How do we go about getting one?
>>
>> The `idr` library provides a `connection` method, e.g. from
>>
>> https://github.com/IDR/idr-notebooks/blob/master/notebooks/Getting_Started.ipynb
>>
>>     from idr import connection
>>     conn = connection()
>>
>> --
>>
>> Cheers,
>> ~Josh
>>
>>
>> > On Wed, May 3, 2017 at 3:11 PM, Josh Moore <josh at glencoesoftware.com>
>> > wrote:
>> >>
>> >> Hi Jon,
>> >>
>> >> On Mon, May 1, 2017 at 11:50 PM, Jon Hazard <jhazard at google.com> wrote:
>> >> > Hi Josh - replies to your questions inline below
>> >> >
>> >> > On Mon, May 1, 2017 at 1:19 PM, Josh Moore <josh at glencoesoftware.com>
>> >> > wrote:
>> >> >>
>> >> >> (List members: sorry for the re-send)
>> >> >>
>> >> >> On Fri, Apr 28, 2017 at 8:07 PM, Jon Hazard <jhazard at google.com>
>> >> >> wrote:
>> >> >> > Hello - first, a hearty thanks to OME for making this data
>> >> >> > available!
>> >> >>
>> >> >> Gladly! And glad to you're interesting in using it.
>> >> >>
>> >> >>
>> >> >> > My team is interested in the images from two datasets:
>> >> >> >
>> >> >> > idr0033-rohban-pathways/screenA
>> >> >> >
>> >> >> > and
>> >> >> >
>> >> >> > idr0028-pascualvargas-rhogtpases
>> >> >> >
>> >> >> > It looks like the images for at least the first data set are
>> >> >> > compressed.
>> >> >>
>> >> >> How do you mean compressed? (Can you send a link?) The original data
>> >> >> has been uploaded to the IDR but certainly for viewing, rendering is
>> >> >> performed server-side before sending to the client.
>> >> >
>> >> >
>> >> > To clarify, I meant that the images I can obtain are lower-resolution
>> >> > JPGs
>> >> > rather than the original-quality TIFF files. This is true whether I
>> >> > export
>> >> > through either the individual image download (by clicking the Down
>> >> > Arrow
>> >> > as
>> >> > shown in the first section of the export help page
>> >> > http://help.openmicroscopy.org/export.html) or by using the public
>> >> > web
>> >> > API
>> >> > (e.g. by issuing the following command: curl
>> >> >
>> >> >
>> >> > "http://idr-demo.openmicroscopy.org/webclient/render_image/3194301/?c=1&q=1"
>> >> >> image.jpg).
>> >>
>> >> I understand: there's not yet a webclient/ or webgateway/ yet that
>> >> would return raw image planes. Our other APIs (e.g. Python, Java, C++)
>> >> do. See below.
>> >>
>> >>
>> >> >> > It would help us a lot to be able to work with the raw,
>> >> >> > uncompressed
>> >> >> > images.
>> >> >> > Are those available?
>> >> >>
>> >> >> Can you explain more about what you are trying to achieve? We're
>> >> >> working on having full download enabled from EBI but until then we
>> >> >> should be able to get you access to something more significant.
>> >> >
>> >> >
>> >> > I would like to have access to the original-quality images for all
>> >> > images in
>> >> > the two datasets I mentioned previously:
>> >> >
>> >> >> idr0028-pascualvargas-rhogtpases and idr0033-rohban-pathways/screenA
>> >> >
>> >> > That is great that the full downloads will eventually be supported
>> >> > through
>> >> > EBI. Is there an expected timeframe for that capability to come
>> >> > online?
>> >>
>> >> I've checked on the status, but there's not yet a concrete date. My
>> >> guess would be that it'll be on the order of weeks rather than months.
>> >>
>> >>
>> >> > Assuming that there is some non-trivial infrastructure required
>> >> > there,
>> >> > is
>> >> > there any other way that we could have access to those two datasets
>> >> > in a
>> >> > more timely manner?
>> >>
>> >> The OME project regularly makes sample data available for download [1]
>> >> . To at least unblock you initially, I'd propose putting one plate for
>> >> each study up on the downloads site (10GB and 40GB respectively), but
>> >> we'd certainly like to avoid mirroring the dozens of TBs from IDR via
>> >> that mechanism. One short-term option for accessing all the plate data
>> >> would be for you to make use of our jupyter instance[2]. See
>> >> https://github.com/IDR/idr-notebooks for examples.
>> >>
>> >> All the best,
>> >> ~Josh
>> >>
>> >> [1] https://downloads.openmicroscopy.org/images/
>> >> [2] https://idr.openmicroscopy.org/jupyter
>> >
>> >
>
>