[ome-devel] smart way to open a time series of one pixel?

[Munro, Ian]

Hi Heinrich

Sounds as if I was less helpful than I hoped. Ah well!
Are you saying that my code reshapes more than required as well ?

Ian 


On 13 Feb 2015, at 16:36, Grabmayr, Heinrich <Heinrich.Grabmayr at ph.tum.de> wrote:

> Hi Ian,
> 
> Sorry to get back to you that late. Thanks very much for your code. I tried it, and its performance was sometimes slightly better, sometimes not; depending on file size and how much to read out of each plane. One big thing I noticed was that I was reshaping an array one time more than necessary, and that helped to speed up the process to about 2/3 the time.
> 
> Cheers
>   Heinrich
> 
> 
>> -----Original Message-----
>> From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
>> On Behalf Of Munro, Ian
>> Sent: 6 February, 2015 5:44 PM
>> To: OME External Developer List
>> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
>> 
>> Hi Heinrich
>> 
>> In my Matlab  code I am attempting to do something similiar (i.e. load many
>> planes as as fast as possible where I can assume that  each plane is the
>> same in terms of size etc).
>> My code to do this is at  https://github.com/openmicroscopy/Imperial-
>> FLIMfit/blob/master/FLIMfitFrontEnd/Classes/%40flim_data_series/load_fli
>> m_cube.m.
>> (see around line 280)
>> 
>> This is based on the code form the  Matlab plug-in available for bio-formats.
>> 
>> Regards
>> 
>> Ian
>> 
>> 
>> On 6 Feb 2015, at 22:29, Grabmayr, Heinrich
>> <Heinrich.Grabmayr at ph.tum.de> wrote:
>> 
>>> Hi Melissa,
>>> 
>>>>> I have huge movies (20k to 60k frames) in file formats from various
>>>>> camera suppliers that can all be opened using bioformats. I would
>>>>> like to access the time series of indivindual pixels. At the moment,
>>>>> I do this in matlab by using r = bfGetReader and then using the
>>>> loci.formats.ChannelSeparator's function to call
>>>> r.openBytes<http://downloads.openmicroscopy.org/bio-
>>>> 
>> formats/5.0.6/api/loci/formats/ChannelSeparator.html#openBytes%28int,
>>>> % 20int,%20int,%20int,%20int%29>(int no, int x, int y, int w, int h)
>>>> with w=h=1 and looping through all the planes. This takes ages. I was
>>>> wondering whether there might be a smarter way to access the data I
>>>> am interested in?
>>>> 
>>>> I would suggest starting by allocating a single buffer, and then
>>>> calling openBytes(int, byte[], int, int, int, int) repeatedly instead
>>>> of openBytes(int, int, int, int, int).  The latter will allocate a
>>>> new byte array to contain the
>>>> pixel(s) each time it is called; I would expect this to have a
>>>> noticeable performance impact when called thousands of times.
>>> 
>>> Thanks for this suggestion. Are you familiar with MATLAB, and how to
>> implement this? I have used the approach mentioned because I do not know
>> how to pass a reference to an array in matlab.
>>> 
>>> I investigated the timing just now:
>>> Using one call of openBytes(int, int, int, int, int) takes 0.175 ms.
>>> This call, and the necessary type casting to be able to store it into
>>> a matlab array takes 0.989 ms (although now I come to think about it,
>> maybe I should try typecasting the whole array after loading) Opening the
>> file using r=bfGetReader(id) takes 40ms And opening the file and loading
>> 4000 frames take 16s. In this case (.fits file), an acquisition is stored in 15
>> bunches of 2GB-files, so the whole data set is loaded after 4 minutes. I do
>> this for several data sets, and that is why I say it takes very long. Broken
>> down, these times do not seem to be bad to me. Maybe I just have
>> incredibly much data to load.
>>> While at the subject, the .cxd files are stored as single 30GB files. When I
>> load these, r=bfGetReader takes 5-10 minutes to call. Shouldn't the
>> metadata information from one plane be enough, so that the time to get the
>> reader wouldn|t  have to depend on the file siye?
>>> 
>>>> 
>>>> If reusing a single buffer doesn't help noticeably, it would be
>>>> helpful to know more about exactly which formats you are working
>>>> with, as well as roughly how long (in minutes) it takes to open all of the
>> pixels.
>>> 
>>> I am using .fits files saved by the Andor software Solis, .cxd from
>> Hamamatsu LiveImage, and .pcoraw from PCO's CamWare.
>>> 
>>>> 
>>> Thanks,
>>>   Heinrich
>>> 
>>> _______________________________________________
>>> ome-devel mailing list
>>> ome-devel at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>> 
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel