[ome-devel] smart way to open a time series of one pixel?

[Munro, Ian]

Hi Heinrich

In my Matlab  code I am attempting to do something similiar (i.e. load many planes as as fast as possible where I can assume that  each plane 
is the same in terms of size etc).
My code to do this is at  https://github.com/openmicroscopy/Imperial-FLIMfit/blob/master/FLIMfitFrontEnd/Classes/%40flim_data_series/load_flim_cube.m.
(see around line 280)  

This is based on the code form the  Matlab plug-in available for bio-formats.

Regards

Ian


On 6 Feb 2015, at 22:29, Grabmayr, Heinrich <Heinrich.Grabmayr at ph.tum.de> wrote:

> Hi Melissa,
> 
>>> I have huge movies (20k to 60k frames) in file formats from various
>>> camera suppliers that can all be opened using bioformats. I would like
>>> to access the time series of indivindual pixels. At the moment, I do
>>> this in matlab by using r = bfGetReader and then using the
>> loci.formats.ChannelSeparator's function to call
>> r.openBytes<http://downloads.openmicroscopy.org/bio-
>> formats/5.0.6/api/loci/formats/ChannelSeparator.html#openBytes%28int,%
>> 20int,%20int,%20int,%20int%29>(int no, int x, int y, int w, int h) with w=h=1
>> and looping through all the planes. This takes ages. I was wondering
>> whether there might be a smarter way to access the data I am interested
>> in?
>> 
>> I would suggest starting by allocating a single buffer, and then calling
>> openBytes(int, byte[], int, int, int, int) repeatedly instead of openBytes(int,
>> int, int, int, int).  The latter will allocate a new byte array to contain the
>> pixel(s) each time it is called; I would expect this to have a noticeable
>> performance impact when called thousands of times.
> 
> Thanks for this suggestion. Are you familiar with MATLAB, and how to implement this? I have used the approach mentioned because I do not know how to pass a reference to an array in matlab.
> 
> I investigated the timing just now:
> Using one call of openBytes(int, int, int, int, int) takes 0.175 ms. This call, and the necessary type casting to be able to store it into a matlab array takes 0.989 ms (although now I come to think about it, maybe I should try typecasting the whole array after loading)
> Opening the file using r=bfGetReader(id) takes 40ms
> And opening the file and loading 4000 frames take 16s. In this case (.fits file), an acquisition is stored in 15 bunches of 2GB-files, so the whole data set is loaded after 4 minutes. I do this for several data sets, and that is why I say it takes very long. Broken down, these times do not seem to be bad to me. Maybe I just have incredibly much data to load.
> While at the subject, the .cxd files are stored as single 30GB files. When I load these, r=bfGetReader takes 5-10 minutes to call. Shouldn't the metadata information from one plane be enough, so that the time to get the reader wouldn|t  have to depend on the file siye?
> 
>> 
>> If reusing a single buffer doesn't help noticeably, it would be helpful to know
>> more about exactly which formats you are working with, as well as roughly
>> how long (in minutes) it takes to open all of the pixels.
> 
> I am using .fits files saved by the Andor software Solis, .cxd from Hamamatsu LiveImage, and .pcoraw from PCO's CamWare.
> 
>> 
> Thanks, 
>    Heinrich
> 
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel