[ome-devel] smart way to open a time series of one pixel?

Munro, Ian i.munro at imperial.ac.uk
Fri Feb 13 17:02:04 GMT 2015 

Hi Heinrich.

Thanks for the tips. Much appreciated  I’ll look into the passing a pointer if I get a chance it’s certainly not obvious 
how to do that in Matlab.

Regards

Ian 



On 13 Feb 2015, at 16:56, Grabmayr, Heinrich <Heinrich.Grabmayr at ph.tum.de> wrote:

> Hi Ian,
> No your code is great. 
> One step that takes long for large image files is r.setID(). I guess this cannot be sped up because the reader may need to go through all planes to create the metadata (may depend on the file type?)
> Melissa suggested to get the data by handing openBytes a pointer to an array, so it doesn't need to be initialized for each call. You don't do that either, and I believe this is because it's against the philosophy in Matlab, so it is hard to do (a few google hits looked very complicated, so I left it). But if it is possible, I expect that your code could be sped up as well.
> 
> Concerning reshaping, as I only need to identify pixels at the moment, and not form an image, I can leave the image data as a 1D array instead of reshaping it to 2D. But I expect that this is different for you.
> Best, Heinrich
> 
>> -----Original Message-----
>> From: ome-devel [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
>> On Behalf Of Munro, Ian
>> Sent: 13 February, 2015 11:44 AM
>> To: OME External Developer List
>> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
>> 
>> Hi Heinrich
>> 
>> Sounds as if I was less helpful than I hoped. Ah well!
>> Are you saying that my code reshapes more than required as well ?
>> 
>> Ian
>> 
>> 
>> On 13 Feb 2015, at 16:36, Grabmayr, Heinrich
>> <Heinrich.Grabmayr at ph.tum.de> wrote:
>> 
>>> Hi Ian,
>>> 
>>> Sorry to get back to you that late. Thanks very much for your code. I tried
>> it, and its performance was sometimes slightly better, sometimes not;
>> depending on file size and how much to read out of each plane. One big
>> thing I noticed was that I was reshaping an array one time more than
>> necessary, and that helped to speed up the process to about 2/3 the time.
>>> 
>>> Cheers
>>>  Heinrich
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: ome-devel
>>>> [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk]
>>>> On Behalf Of Munro, Ian
>>>> Sent: 6 February, 2015 5:44 PM
>>>> To: OME External Developer List
>>>> Subject: Re: [ome-devel] smart way to open a time series of one pixel?
>>>> 
>>>> Hi Heinrich
>>>> 
>>>> In my Matlab  code I am attempting to do something similiar (i.e.
>>>> load many planes as as fast as possible where I can assume that  each
>>>> plane is the same in terms of size etc).
>>>> My code to do this is at  https://github.com/openmicroscopy/Imperial-
>>>> FLIMfit/blob/master/FLIMfitFrontEnd/Classes/%40flim_data_series/load_
>>>> fli
>>>> m_cube.m.
>>>> (see around line 280)
>>>> 
>>>> This is based on the code form the  Matlab plug-in available for bio-
>> formats.
>>>> 
>>>> Regards
>>>> 
>>>> Ian
>>>> 
>>>> 
>>>> On 6 Feb 2015, at 22:29, Grabmayr, Heinrich
>>>> <Heinrich.Grabmayr at ph.tum.de> wrote:
>>>> 
>>>>> Hi Melissa,
>>>>> 
>>>>>>> I have huge movies (20k to 60k frames) in file formats from
>>>>>>> various camera suppliers that can all be opened using bioformats.
>>>>>>> I would like to access the time series of indivindual pixels. At
>>>>>>> the moment, I do this in matlab by using r = bfGetReader and then
>>>>>>> using the
>>>>>> loci.formats.ChannelSeparator's function to call
>>>>>> r.openBytes<http://downloads.openmicroscopy.org/bio-
>>>>>> 
>>>> 
>> formats/5.0.6/api/loci/formats/ChannelSeparator.html#openBytes%28int,
>>>>>> % 20int,%20int,%20int,%20int%29>(int no, int x, int y, int w, int
>>>>>> h) with w=h=1 and looping through all the planes. This takes ages.
>>>>>> I was wondering whether there might be a smarter way to access the
>>>>>> data I am interested in?
>>>>>> 
>>>>>> I would suggest starting by allocating a single buffer, and then
>>>>>> calling openBytes(int, byte[], int, int, int, int) repeatedly
>>>>>> instead of openBytes(int, int, int, int, int).  The latter will
>>>>>> allocate a new byte array to contain the
>>>>>> pixel(s) each time it is called; I would expect this to have a
>>>>>> noticeable performance impact when called thousands of times.
>>>>> 
>>>>> Thanks for this suggestion. Are you familiar with MATLAB, and how to
>>>> implement this? I have used the approach mentioned because I do not
>>>> know how to pass a reference to an array in matlab.
>>>>> 
>>>>> I investigated the timing just now:
>>>>> Using one call of openBytes(int, int, int, int, int) takes 0.175 ms.
>>>>> This call, and the necessary type casting to be able to store it
>>>>> into a matlab array takes 0.989 ms (although now I come to think
>>>>> about it,
>>>> maybe I should try typecasting the whole array after loading) Opening
>>>> the file using r=bfGetReader(id) takes 40ms And opening the file and
>>>> loading
>>>> 4000 frames take 16s. In this case (.fits file), an acquisition is
>>>> stored in 15 bunches of 2GB-files, so the whole data set is loaded
>>>> after 4 minutes. I do this for several data sets, and that is why I
>>>> say it takes very long. Broken down, these times do not seem to be
>>>> bad to me. Maybe I just have incredibly much data to load.
>>>>> While at the subject, the .cxd files are stored as single 30GB
>>>>> files. When I
>>>> load these, r=bfGetReader takes 5-10 minutes to call. Shouldn't the
>>>> metadata information from one plane be enough, so that the time to
>>>> get the reader wouldn|t  have to depend on the file siye?
>>>>> 
>>>>>> 
>>>>>> If reusing a single buffer doesn't help noticeably, it would be
>>>>>> helpful to know more about exactly which formats you are working
>>>>>> with, as well as roughly how long (in minutes) it takes to open all
>>>>>> of the
>>>> pixels.
>>>>> 
>>>>> I am using .fits files saved by the Andor software Solis, .cxd from
>>>> Hamamatsu LiveImage, and .pcoraw from PCO's CamWare.
>>>>> 
>>>>>> 
>>>>> Thanks,
>>>>>  Heinrich
>>>>> 
>>>>> _______________________________________________
>>>>> ome-devel mailing list
>>>>> ome-devel at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>>> 
>>>> _______________________________________________
>>>> ome-devel mailing list
>>>> ome-devel at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>> _______________________________________________
>>> ome-devel mailing list
>>> ome-devel at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>> 
>> _______________________________________________
>> ome-devel mailing list
>> ome-devel at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

More information about the ome-devel mailing list