[ome-devel] smart way to open a time series of one pixel?

[Grabmayr, Heinrich]

Hi Ian,

Sorry to get back to you that late. Thanks very much for your code. I tried it, and its performance was sometimes slightly better, sometimes not; depending on file size and how much to read out of each plane. One big thing I noticed was that I was reshaping an array one time more than necessary, and that helped to speed up the process to about 2/3 the time.

Cheers
   Heinrich


> -----Original Message-----
> From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
> On Behalf Of Munro, Ian
> Sent: 6 February, 2015 5:44 PM
> To: OME External Developer List
> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
> 
> Hi Heinrich
> 
> In my Matlab  code I am attempting to do something similiar (i.e. load many
> planes as as fast as possible where I can assume that  each plane is the
> same in terms of size etc).
> My code to do this is at  https://github.com/openmicroscopy/Imperial-
> FLIMfit/blob/master/FLIMfitFrontEnd/Classes/%40flim_data_series/load_fli
> m_cube.m.
> (see around line 280)
> 
> This is based on the code form the  Matlab plug-in available for bio-formats.
> 
> Regards
> 
> Ian
> 
> 
> On 6 Feb 2015, at 22:29, Grabmayr, Heinrich
> <Heinrich.Grabmayr at ph.tum.de> wrote:
> 
> > Hi Melissa,
> >
> >>> I have huge movies (20k to 60k frames) in file formats from various
> >>> camera suppliers that can all be opened using bioformats. I would
> >>> like to access the time series of indivindual pixels. At the moment,
> >>> I do this in matlab by using r = bfGetReader and then using the
> >> loci.formats.ChannelSeparator's function to call
> >> r.openBytes<http://downloads.openmicroscopy.org/bio-
> >>
> formats/5.0.6/api/loci/formats/ChannelSeparator.html#openBytes%28int,
> >> % 20int,%20int,%20int,%20int%29>(int no, int x, int y, int w, int h)
> >> with w=h=1 and looping through all the planes. This takes ages. I was
> >> wondering whether there might be a smarter way to access the data I
> >> am interested in?
> >>
> >> I would suggest starting by allocating a single buffer, and then
> >> calling openBytes(int, byte[], int, int, int, int) repeatedly instead
> >> of openBytes(int, int, int, int, int).  The latter will allocate a
> >> new byte array to contain the
> >> pixel(s) each time it is called; I would expect this to have a
> >> noticeable performance impact when called thousands of times.
> >
> > Thanks for this suggestion. Are you familiar with MATLAB, and how to
> implement this? I have used the approach mentioned because I do not know
> how to pass a reference to an array in matlab.
> >
> > I investigated the timing just now:
> > Using one call of openBytes(int, int, int, int, int) takes 0.175 ms.
> > This call, and the necessary type casting to be able to store it into
> > a matlab array takes 0.989 ms (although now I come to think about it,
> maybe I should try typecasting the whole array after loading) Opening the
> file using r=bfGetReader(id) takes 40ms And opening the file and loading
> 4000 frames take 16s. In this case (.fits file), an acquisition is stored in 15
> bunches of 2GB-files, so the whole data set is loaded after 4 minutes. I do
> this for several data sets, and that is why I say it takes very long. Broken
> down, these times do not seem to be bad to me. Maybe I just have
> incredibly much data to load.
> > While at the subject, the .cxd files are stored as single 30GB files. When I
> load these, r=bfGetReader takes 5-10 minutes to call. Shouldn't the
> metadata information from one plane be enough, so that the time to get the
> reader wouldn|t  have to depend on the file siye?
> >
> >>
> >> If reusing a single buffer doesn't help noticeably, it would be
> >> helpful to know more about exactly which formats you are working
> >> with, as well as roughly how long (in minutes) it takes to open all of the
> pixels.
> >
> > I am using .fits files saved by the Andor software Solis, .cxd from
> Hamamatsu LiveImage, and .pcoraw from PCO's CamWare.
> >
> >>
> > Thanks,
> >    Heinrich
> >
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> 
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel