[ome-users] Loading a subset of a stack

[Melissa Linkert]

Hi Thomas,

> > We are reviewing a fix for this now: https://github.com/openmicroscopy/bioformats/pull/1687
> > This will mean that disabling file grouping works as intended - the initialization time will be faster (~20x in local tests), but the only planes available will be those in the selected TIFF file.
> 
> Just to be sure, this will disable the loading of several positions in the same dataset, but for dataset where one position must be split in several stacks (since micromanager still doesn’t handle bigtiff), all files will still be concatenated?…

Not quite - all file grouping will be disabled, so grouping of multiple
files in a single position will not occur.  Each file from the position
would need to be initialized separately.

> > I would expect that to be included in the upcoming 5.1.0 release.  
> 
> Any time horizon?

I would expect within the next couple of weeks.

> Is there anything I can help testing regarding this issue?

At the moment, the best thing is to wait for 5.1.0 - if upgrading does
not acceptably improve performance, then we may ask for more help in
testing.

Regards,
-Melissa

On Tue, Mar 24, 2015 at 08:58:14AM +0100, Thomas Julou wrote:
> Hi,
> 
> > We are reviewing a fix for this now: https://github.com/openmicroscopy/bioformats/pull/1687
> > This will mean that disabling file grouping works as intended - the initialization time will be faster (~20x in local tests), but the only planes available will be those in the selected TIFF file.
> 
> Just to be sure, this will disable the loading of several positions in the same dataset, but for dataset where one position must be split in several stacks (since micromanager still doesn’t handle bigtiff), all files will still be concatenated?…
> 
> > I would expect that to be included in the upcoming 5.1.0 release.  
> 
> Any time horizon?
> Is there anything I can help testing regarding this issue?
> 
> Thanks for your help. Best,
> Thomas
> 
> 
> > On Wed, Mar 18, 2015 at 04:07:44PM +0100, Thomas Julou wrote:
> >> Hello,
> >> 
> >>> I heard a lot of good in the past about the legendary support for ome. I’m glad to experience it for real :)
> >> 
> >> I’m happy I gave you a good week end start ;)
> >> 
> >>>> It looks like the performance of this reader is unlikely to be improved
> >>>> further. (…)
> >>>> 
> >>>> You said that it was taking a very long time to initialise.  Could you
> >>>> possibly let us know any quantitative measurements of that time?  And do
> >>>> you know how many files/directories are in the dataset in question?
> >>> 
> >>> 
> >>> I’ll try next week to disable file grouping, and also to benchmark import with different options next week.
> >> 
> >> Here you go:
> >> I did the tests with a directory that contains the following files (time lapse at 5 positions acquired with micro-manager)
> >> 
> >> 595 20150304_exp_settings.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos0_1.ome.tif
> >> 2.7G 20150304_glu_multiexp_1_MMStack_Pos0_2.ome.tif
> >> 20M 20150304_glu_multiexp_1_MMStack_Pos0_metadata.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos0.ome.tif
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos1_1.ome.tif
> >> 2.7G 20150304_glu_multiexp_1_MMStack_Pos1_2.ome.tif
> >> 20M 20150304_glu_multiexp_1_MMStack_Pos1_metadata.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos1.ome.tif
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos2_1.ome.tif
> >> 2.7G 20150304_glu_multiexp_1_MMStack_Pos2_2.ome.tif
> >> 20M 20150304_glu_multiexp_1_MMStack_Pos2_metadata.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos2.ome.tif
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos3_1.ome.tif
> >> 2.7G 20150304_glu_multiexp_1_MMStack_Pos3_2.ome.tif
> >> 20M 20150304_glu_multiexp_1_MMStack_Pos3_metadata.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos3.ome.tif
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos4_1.ome.tif
> >> 2.7G 20150304_glu_multiexp_1_MMStack_Pos4_2.ome.tif
> >> 20M 20150304_glu_multiexp_1_MMStack_Pos4_metadata.txt
> >> 4.0G 20150304_glu_multiexp_1_MMStack_Pos4.ome.tif
> >> 
> >> I tried to read the first 10 frames of the first stack (≈11gb; split into 3 files by micro-manager) using ImageReader() as described early in this thread (see attached file).
> >> with default options: « 494.016s to create a reader. »
> >> with no grouping: « 489.204s to create a reader. » 
> >> with no grouping, minimum metadata: « 474.095s to create a reader. » 
> >> 
> >> I guess the little speed up is due to repeated file access: if I use default options once again at the end I get « 460.052s to create a reader. »
> >> 
> >> Whether I disable grouping or not, I always get 5 messages «  Reading IFDs; Populating metadata »; so I guess that grouping is actually not disabled. correct?
> >> I suspect a function disabling grouping of « series » might be what I’m after here, though I don’t know which one exactly…
> >> 
> >> This is maybe not so slow if you consider that it’s parsing all metadata; but in my case, what I would like would be to simply access the data of the first frames, don’t care about what happens later.
> >> 
> >> Any hint to help us handle these datasets in a faster way will be very appreciated.
> >> Best,
> >> 
> >> Thomas
> >> 
> >> --
> >> Thomas Julou  |  Computational & Systems Biology  |  Biozentrum – University of Basel  |  Klingelbergstrasse 50/70 CH-4056 Basel  |  +41 (0)61 267 16 21
> >> 
> >> 
> >> 
> > 
> >> _______________________________________________
> >> ome-users mailing list
> >> ome-users at lists.openmicroscopy.org.uk
> >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>