[ome-devel] Zeiss 710

Tue Oct 12 20:48:13 BST 2010

Hi Rubén,

> @Curtis: Actually we face this problem with very large datasets instead. One
> of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +
> 5.5 M headers. This kind of storage can not be afforded by some groups.

My guess is that a large portion of this metadata is occupied by
TiffData elements.  One idea would be to update Bio-Formats' OME-TIFF writer
to be a bit smarter about how TiffData elements are written, which may
substantially shrink the size of the metadata in each file without
requiring a schema change.

I do agree with Curtis that we should at least consider making a change to the
OME-XML schema, but in the meantime that may be a partial solution.
However, as mentioned before, it would help to see the exact OME-XML
that you are working with.  We wouldn't want to implement a solution
only to find out that it doesn't solve your problem.

> @Melissa: Do you think that a tool that could group or split datasets could
> also deal with the Leica Matrix Screener OME.TIF. My tool will assume that
> all the headers are the same, but I know that is not like that. Therefore I
> think that unification of the headers is a good idea, in the future.

It's possible, but I as mentioned before I really think you would be
better off waiting for proper Leica Matrix support in Bio-Formats.

Regards,
-Melissa

On Tue, Oct 12, 2010 at 02:20:28PM -0500, Curtis Rueden wrote:
> Hi Rubén,
> 
> @Curtis: Actually we face this problem with very large datasets instead. One
> > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +
> > 5.5 M headers. This kind of storage can not be afforded by some groups.
> >
> 
> Thanks, that explanation helps a lot!
> 
> It sounds like this is a use case for either:
> 
> A) Putting the metadata in only the first .ome.tif file; or
> B) Putting the metadata in a companion file.
> 
> Each OME-TIFF would still need a small amount of metadata (particularly the
> UUID associated with it) but the bulk of it could in principle be stripped
> out.
> 
> I am all in favor of either of these options being supported in the next
> OME-XML schema, but in the past there has been strong opposition, due to
> concern about metadata being lost by a partial copy. Perhaps this concrete
> use case justifies reversing our decision on this? Anyone have any other
> ideas?
> 
> To split each serie in different ome.tif datasets seems like our solution.
> > Do you think its convenient? Are there tools for this? I am working in one
> > already.
> >
> 
> You could definitely do that, but it seems a shame. You would be
> compromising the modeling of your data, just to work around an unfortunate
> implementation detail of OME-TIFF. Better would be for us to provide a
> better solution with the next schema release.
> 
> -Curtis
> 
> On Tue, Oct 12, 2010 at 1:32 PM, Rubén Muñoz <ruben.munoz at embl.de> wrote:
> 
> > Hi Curtis and Melissa,
> >
> > On Oct 12, 2010, at 5:41 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
> >
> > Hi Rubén & Melissa,
> >
> > > Is there a way of excluding the image/pixel list in all the OME.TIF with
> >> bfconvert?
> >>
> >> Perhaps I am misunderstanding the question, but are you asking about
> >> excluding the Image and Pixels elements from the OME-XML metadata?
> >
> >
> > Yes that's my question.
> >
> > If
> >> so, then I'm afraid that that is really not possible - both elements
> >> provide critical metadata for reading the pixel data.
> >>
> >
> >
> > Thats how I understand the specification. All the files are
> > cross-referenced in the OME.TIF headers by Pixel elements, independently of
> > the number of files in the set...
> >
> > Note: We store all the planes & timepoins & channels in different files for
> > convenience, that's not a matter of discussion anymore here, since our
> > software doesn't always operate on stacks.
> >
> > This could conceivably happen with extremely small image planes. For
> > example, an 8x8 image at 8-bit resolution only occupies 64 bytes, but the
> > metadata would be at least ~180 bytes; e.g.:
> >
> > <Image ID="Image:xxxx" Name="Image xxxx"><Pixels DimensionOrder="XYCZT"
> > ID="Pixels:xxxx" SizeC="1" SizeT="1" SizeX="8" SizeY="8" SizeZ="1"
> > Type="uint8"><TiffData/></Pixels></Image>
> >
> > I can't think of any other case where the metadata size exceeds the pixel
> > size. Rubén, can you elaborate on what is going on? What are the dimensions
> > of your images? How many screens, plates, wells, and what is the image
> > resolution?
> >
> >
> > @Curtis: Actually we face this problem with very large datasets instead.
> > One of our time-lapse acquisitions has 23000 ome.tif files, each is 8M =
> > 2.5M + 5.5 M headers. This kind of storage can not be afforded by some
> > groups.
> >
> > My colleagues plan to make bigger experiments, and I we are facing storage
> > issues. And bioformats needs more memory to convert large datasets.
> >
> > To split each serie in different ome.tif datasets seems like our solution.
> > Do you think its convenient? Are there tools for this? I am working in one
> > already.
> >
> > @Melissa: Do you think that a tool that could group or split datasets could
> > also deal with the Leica Matrix Screener OME.TIF. My tool will assume that
> > all the headers are the same, but I know that is not like that. Therefore I
> > think that unification of the headers is a good idea, in the future.
> >
> > Thanks for your replies,
> >
> > Rubén
> >
> >
> > -Curtis
> >
> > On Tue, Oct 12, 2010 at 10:23 AM, Melissa Linkert <<melissa at glencoesoftware.com>
> > melissa at glencoesoftware.com> wrote:
> >
> >> Hi Rubén,
> >>
> >> > Is there a way of excluding the image/pixel list in all the OME.TIF with
> >> bfconvert?
> >>
> >> Perhaps I am misunderstanding the question, but are you asking about
> >> excluding the Image and Pixels elements from the OME-XML metadata?  If
> >> so, then I'm afraid that that is really not possible - both elements
> >> provide critical metadata for reading the pixel data.
> >>
> >> > For large multi-file sets the metadata is bigger than the pixel values
> >> and this prevents us to use OME.TIF in High Throughput Screens.
> >>
> >> That is a bit strange, but I have a few ideas as to what might be
> >> happening there.  I imagine the OME-TIFF file is prohibitively large,
> >> but if you could send just the OME-XML metadata that would be a huge
> >> help.  You can extract the metadata like this:
> >>
> >>  # if all of the Bio-Formats command-line tools are installed
> >>  $ tiffcomment /path/to/file.ome.tiff
> >>
> >>  # if 'tiffcomment' is not installed
> >>  $ java loci.formats.tools.TiffComment /path/to/file.ome.tiff
> >>
> >> > I'd like to help with the Leica Matrix Screener and ZEN importers as
> >> much as possible.
> >>
> >> Thank you for the offer.  At the moment, there is not much that can be
> >> done with the Zeiss LSM/Zen reader.  It will be at least a few weeks
> >> before I can do anything with the Leica Matrix reader, but you are more
> >> than welcome to start on a reader in the meantime if you like.  There
> >> are three main pieces of work to be done for this reader:
> >>
> >>  1) Make sure that all of the files are grouped together correctly, no
> >>  matter which file in the dataset is selected by the user.
> >>
> >>  2) Make sure that all of the pixel data and 'core' metadata is read
> >>  correctly.
> >>
> >>  3) Make sure that all of the OME-XML metadata from each of the
> >>  constituent OME-TIFFs is parsed and unified.
> >>
> >> My plan is to roughtly model the Leica Matrix reader after
> >> loci.formats.in.MIASReader and loci.formats.in.InCellReader, with an
> >> internal loci.formats.in.OMETiffReader doing much of the work for (2)
> >> and (3).
> >>
> >> Alternatively, you can wait a bit until I've had time to write an initial
> >> version, and then help with the testing and polishing.
> >>
> >> Regards,
> >> -Melissa
> >>
> >> On Mon, Oct 11, 2010 at 09:51:56PM +0200, Rubén Muñoz wrote:
> >> > Hi Melissa,
> >> > Is there a way of excluding the image/pixel list in all the OME.TIF with
> >> bfconvert?
> >> > For large multi-file sets the metadata is bigger than the pixel values
> >> and this prevents us to use OME.TIF in High Throughput Screens. I'd like to
> >> help with the Leica Matrix Screener and ZEN importers as much as possible.
> >> > Best regards,
> >> > Rubén
> >> >
> >> > On Oct 11, 2010, at 8:22 PM, Melissa Linkert <<melissa at glencoesoftware.com>
> >> melissa at glencoesoftware.com> wrote:
> >> >
> >> > > Hi Rubén,
> >> > >
> >> > >> That's good news. I am amazed by how the project evolves from simpler
> >> to more sophisticated needs. Actually bfconvert was designed to convert I
> >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
> >> to one-big OME.TIF and then enrich the metadata.
> >> > >
> >> > > If possible, I would recommend waiting for this ticket to be closed:
> >> > >
> >> > > <http://dev.loci.wisc.edu/trac/software/ticket/563>
> >> http://dev.loci.wisc.edu/trac/software/ticket/563
> >> > >
> >> > > ...at which point you can just do 'bfconvert
> >> leica-matrix-file.ome.tiff output.ome.tiff' without any additional magic.
> >> > >
> >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> >> metadata with EditTiffComment for that? Would a tool like EditTiffComment
> >> that modifies the metadata of many files accordingly be usefull?
> >> > >
> >> > > You could use one of our TIFF comment editing utilities:
> >> > >
> >> > > * java loci.formats.tools.TiffComment -edit /path/to/file
> >> > > * java loci.formats.tools.EditTiffG /path/to/file
> >> > >
> >> > > But please be very, very cautious.  Again, the above ticket really is
> >> > > the best solution, so if you can I would recommend either waiting for
> >> > > that ticket to be closed or (if you have a lot of free time) helping
> >> to implement it.
> >> > >
> >> > > Regards,
> >> > > -Melissa
> >> > >
> >> > > On Wed, Oct 06, 2010 at 02:46:50PM +0200, Rubén Muñoz wrote:
> >> > >> Hi Melissa,
> >> > >>
> >> > >> On Oct 6, 2010, at 1:09 AM, Melissa Linkert wrote:
> >> > >>
> >> > >>> Hi Rubén,
> >> > >>>
> >> > >>>> Therefore tried this with bfconvert, even when appending to
> >> existing OME.TIF is not supported yet.
> >> > >>>
> >> > >>> Appending to existing OME-TIFF files is, in theory, supported.
> >>  However,
> >> > >>> the MetadataRetrieve object used by OMETiffWriter must be correctly
> >> set
> >> > >>> up, which unfortunately will not happen by repeatedly calling
> >> > >>> bfconvert.
> >> > >>
> >> > >> That's good news. I am amazed by how the project evolves from simpler
> >> to more sophisticated needs. Actually bfconvert was designed to convert I
> >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
> >> to one-big OME.TIF and then enrich the metadata.
> >> > >>
> >> > >>>
> >> > >>>> However wIth the latest revision of bioformats 7034 when I output
> >> the bfconvert to the same file consecutively, the file is always getting
> >> bigger in size.
> >> > >>>
> >> > >>> Yes, I see the same behavior.  It is ticketed here:
> >> > >>>
> >> > >>> <http://dev.loci.wisc.edu/trac/software/ticket/578>
> >> http://dev.loci.wisc.edu/trac/software/ticket/578
> >> > >>>
> >> > >>> And, as usual, you have been CC'd.  I am a bit curious though: why
> >> run
> >> > >>> bfconvert repeatedly with the same output file?  Are you trying to
> >> > >>> convert multiple datasets to a single OME-TIFF file?  If I know the
> >> use
> >> > >>> case, then maybe there is a work-around until the above ticket is
> >> closed.
> >> > >>
> >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> >> metadata with EditTiffComment for that? Would a tool like EditTiffComment
> >> that modifies the metadata of many files accordingly be usefull?
> >> > >>
> >> > >> Another topic, I guess you remember about the Zeiss ZEN software,
> >>  multiple positions in the same LSM file are now supported.
> >> > >> I recently found out that it also produces multi-file LSM datasets
> >> (without .mdb). I am getting one of these. I wonder how these files now
> >> about each other, but could be that they don't.
> >> > >>
> >> > >> Best regards,
> >> > >>
> >> > >> Rubén
> >> > >>
> >> > >>
> >> > >>
> >> _______________________________________________
> >> ome-devel mailing list
> >>  <ome-devel at lists.openmicroscopy.org.uk>
> >> ome-devel at lists.openmicroscopy.org.uk
> >>  <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
> >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >>
> >
> >