[ome-devel] smart way to open a time series of one pixel?

Fri Feb 13 16:56:57 GMT 2015

Hi Ian,
No your code is great. 
One step that takes long for large image files is r.setID(). I guess this cannot be sped up because the reader may need to go through all planes to create the metadata (may depend on the file type?)
Melissa suggested to get the data by handing openBytes a pointer to an array, so it doesn't need to be initialized for each call. You don't do that either, and I believe this is because it's against the philosophy in Matlab, so it is hard to do (a few google hits looked very complicated, so I left it). But if it is possible, I expect that your code could be sped up as well.

Concerning reshaping, as I only need to identify pixels at the moment, and not form an image, I can leave the image data as a 1D array instead of reshaping it to 2D. But I expect that this is different for you.
Best, Heinrich

> -----Original Message-----
> From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
> On Behalf Of Munro, Ian
> Sent: 13 February, 2015 11:44 AM
> To: OME External Developer List
> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
> 
> Hi Heinrich
> 
> Sounds as if I was less helpful than I hoped. Ah well!
> Are you saying that my code reshapes more than required as well ?
> 
> Ian
> 
> 
> On 13 Feb 2015, at 16:36, Grabmayr, Heinrich
> <Heinrich.Grabmayr at ph.tum.de> wrote:
> 
> > Hi Ian,
> >
> > Sorry to get back to you that late. Thanks very much for your code. I tried
> it, and its performance was sometimes slightly better, sometimes not;
> depending on file size and how much to read out of each plane. One big
> thing I noticed was that I was reshaping an array one time more than
> necessary, and that helped to speed up the process to about 2/3 the time.
> >
> > Cheers
> >   Heinrich
> >
> >
> >> -----Original Message-----
> >> From: ome-devel
> >> [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
> >> On Behalf Of Munro, Ian
> >> Sent: 6 February, 2015 5:44 PM
> >> To: OME External Developer List
> >> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
> >>
> >> Hi Heinrich
> >>
> >> In my Matlab  code I am attempting to do something similiar (i.e.
> >> load many planes as as fast as possible where I can assume that  each
> >> plane is the same in terms of size etc).
> >> My code to do this is at  https://github.com/openmicroscopy/Imperial-
> >> FLIMfit/blob/master/FLIMfitFrontEnd/Classes/%40flim_data_series/load_
> >> fli
> >> m_cube.m.
> >> (see around line 280)
> >>
> >> This is based on the code form the  Matlab plug-in available for bio-
> formats.
> >>
> >> Regards
> >>
> >> Ian
> >>
> >>
> >> On 6 Feb 2015, at 22:29, Grabmayr, Heinrich
> >> <Heinrich.Grabmayr at ph.tum.de> wrote:
> >>
> >>> Hi Melissa,
> >>>
> >>>>> I have huge movies (20k to 60k frames) in file formats from
> >>>>> various camera suppliers that can all be opened using bioformats.
> >>>>> I would like to access the time series of indivindual pixels. At
> >>>>> the moment, I do this in matlab by using r = bfGetReader and then
> >>>>> using the
> >>>> loci.formats.ChannelSeparator's function to call
> >>>> r.openBytes<http://downloads.openmicroscopy.org/bio-
> >>>>
> >>
> formats/5.0.6/api/loci/formats/ChannelSeparator.html#openBytes%28int,
> >>>> % 20int,%20int,%20int,%20int%29>(int no, int x, int y, int w, int
> >>>> h) with w=h=1 and looping through all the planes. This takes ages.
> >>>> I was wondering whether there might be a smarter way to access the
> >>>> data I am interested in?
> >>>>
> >>>> I would suggest starting by allocating a single buffer, and then
> >>>> calling openBytes(int, byte[], int, int, int, int) repeatedly
> >>>> instead of openBytes(int, int, int, int, int).  The latter will
> >>>> allocate a new byte array to contain the
> >>>> pixel(s) each time it is called; I would expect this to have a
> >>>> noticeable performance impact when called thousands of times.
> >>>
> >>> Thanks for this suggestion. Are you familiar with MATLAB, and how to
> >> implement this? I have used the approach mentioned because I do not
> >> know how to pass a reference to an array in matlab.
> >>>
> >>> I investigated the timing just now:
> >>> Using one call of openBytes(int, int, int, int, int) takes 0.175 ms.
> >>> This call, and the necessary type casting to be able to store it
> >>> into a matlab array takes 0.989 ms (although now I come to think
> >>> about it,
> >> maybe I should try typecasting the whole array after loading) Opening
> >> the file using r=bfGetReader(id) takes 40ms And opening the file and
> >> loading
> >> 4000 frames take 16s. In this case (.fits file), an acquisition is
> >> stored in 15 bunches of 2GB-files, so the whole data set is loaded
> >> after 4 minutes. I do this for several data sets, and that is why I
> >> say it takes very long. Broken down, these times do not seem to be
> >> bad to me. Maybe I just have incredibly much data to load.
> >>> While at the subject, the .cxd files are stored as single 30GB
> >>> files. When I
> >> load these, r=bfGetReader takes 5-10 minutes to call. Shouldn't the
> >> metadata information from one plane be enough, so that the time to
> >> get the reader wouldn|t  have to depend on the file siye?
> >>>
> >>>>
> >>>> If reusing a single buffer doesn't help noticeably, it would be
> >>>> helpful to know more about exactly which formats you are working
> >>>> with, as well as roughly how long (in minutes) it takes to open all
> >>>> of the
> >> pixels.
> >>>
> >>> I am using .fits files saved by the Andor software Solis, .cxd from
> >> Hamamatsu LiveImage, and .pcoraw from PCO's CamWare.
> >>>
> >>>>
> >>> Thanks,
> >>>   Heinrich
> >>>
> >>> _______________________________________________
> >>> ome-devel mailing list
> >>> ome-devel at lists.openmicroscopy.org.uk
> >>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >>
> >> _______________________________________________
> >> ome-devel mailing list
> >> ome-devel at lists.openmicroscopy.org.uk
> >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> 
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel