[ome-devel] Can get raw images (vs. compressed) for several datasets?

[Josh Moore]

Hi Jon,

On Thu, May 4, 2017 at 6:42 AM, Jon Hazard <jhazard at google.com> wrote:
> Hey Josh - it would be far better for us if we could get access to full vs.
> a partial datasets.

Definitely understood, and it's something we're keen to enable as well.


> Given that, to verify that I'm understanding how to
> retrieve raw data correctly, could you confirm that something like the
> following code is your suggestion for using the Python API to download raw
> image data:
>
> from PIL import Image
> # Do the following for each imageId we have in the dataset.
> image = conn.getObject("Image", imageId)
> for z in range(image.getSizeZ()):
>   for c in range(image.getSizeC()):
>     for t in range(image.getSizeT()):
>       pixels = image.getPrimaryPixels()
>       plane = pixels.getPlane(z, c, t)  # Get a numpy array.
>       im = Image.fromArray(plane)  # Using PIL.Imagey array.
>
>       im.save("image.tiff")

You might want to take a look at pixels.getPlanes to simplify the loop:

 |  getPlanes(self, zctList)
 |      Returns generator of numpy 2D planes from this set of pixels for a
 |      list of Z, C, T indexes.
 |
 |      :param zctList:     A list of indexes: [(z,c,t), ]


> Also, it looks like a valid username and password are required to open a
> connection to your server.  How do we go about getting one?

The `idr` library provides a `connection` method, e.g. from
https://github.com/IDR/idr-notebooks/blob/master/notebooks/Getting_Started.ipynb

    from idr import connection
    conn = connection()

--

Cheers,
~Josh


> On Wed, May 3, 2017 at 3:11 PM, Josh Moore <josh at glencoesoftware.com> wrote:
>>
>> Hi Jon,
>>
>> On Mon, May 1, 2017 at 11:50 PM, Jon Hazard <jhazard at google.com> wrote:
>> > Hi Josh - replies to your questions inline below
>> >
>> > On Mon, May 1, 2017 at 1:19 PM, Josh Moore <josh at glencoesoftware.com>
>> > wrote:
>> >>
>> >> (List members: sorry for the re-send)
>> >>
>> >> On Fri, Apr 28, 2017 at 8:07 PM, Jon Hazard <jhazard at google.com> wrote:
>> >> > Hello - first, a hearty thanks to OME for making this data available!
>> >>
>> >> Gladly! And glad to you're interesting in using it.
>> >>
>> >>
>> >> > My team is interested in the images from two datasets:
>> >> >
>> >> > idr0033-rohban-pathways/screenA
>> >> >
>> >> > and
>> >> >
>> >> > idr0028-pascualvargas-rhogtpases
>> >> >
>> >> > It looks like the images for at least the first data set are
>> >> > compressed.
>> >>
>> >> How do you mean compressed? (Can you send a link?) The original data
>> >> has been uploaded to the IDR but certainly for viewing, rendering is
>> >> performed server-side before sending to the client.
>> >
>> >
>> > To clarify, I meant that the images I can obtain are lower-resolution
>> > JPGs
>> > rather than the original-quality TIFF files. This is true whether I
>> > export
>> > through either the individual image download (by clicking the Down Arrow
>> > as
>> > shown in the first section of the export help page
>> > http://help.openmicroscopy.org/export.html) or by using the public web
>> > API
>> > (e.g. by issuing the following command: curl
>> >
>> > "http://idr-demo.openmicroscopy.org/webclient/render_image/3194301/?c=1&q=1"
>> >> image.jpg).
>>
>> I understand: there's not yet a webclient/ or webgateway/ yet that
>> would return raw image planes. Our other APIs (e.g. Python, Java, C++)
>> do. See below.
>>
>>
>> >> > It would help us a lot to be able to work with the raw, uncompressed
>> >> > images.
>> >> > Are those available?
>> >>
>> >> Can you explain more about what you are trying to achieve? We're
>> >> working on having full download enabled from EBI but until then we
>> >> should be able to get you access to something more significant.
>> >
>> >
>> > I would like to have access to the original-quality images for all
>> > images in
>> > the two datasets I mentioned previously:
>> >
>> >> idr0028-pascualvargas-rhogtpases and idr0033-rohban-pathways/screenA
>> >
>> > That is great that the full downloads will eventually be supported
>> > through
>> > EBI. Is there an expected timeframe for that capability to come online?
>>
>> I've checked on the status, but there's not yet a concrete date. My
>> guess would be that it'll be on the order of weeks rather than months.
>>
>>
>> > Assuming that there is some non-trivial infrastructure required there,
>> > is
>> > there any other way that we could have access to those two datasets in a
>> > more timely manner?
>>
>> The OME project regularly makes sample data available for download [1]
>> . To at least unblock you initially, I'd propose putting one plate for
>> each study up on the downloads site (10GB and 40GB respectively), but
>> we'd certainly like to avoid mirroring the dozens of TBs from IDR via
>> that mechanism. One short-term option for accessing all the plate data
>> would be for you to make use of our jupyter instance[2]. See
>> https://github.com/IDR/idr-notebooks for examples.
>>
>> All the best,
>> ~Josh
>>
>> [1] https://downloads.openmicroscopy.org/images/
>> [2] https://idr.openmicroscopy.org/jupyter
>
>