[ome-devel] Zeiss 710

[Curtis Rueden]

Hi Melissa,

My guess is that a large portion of this metadata is occupied by
> TiffData elements.


Indeed.


> One idea would be to update Bio-Formats' OME-TIFF writer
> to be a bit smarter about how TiffData elements are written, which may
> substantially shrink the size of the metadata in each file without
> requiring a schema change.
>

In general I agree that making the OME-TIFF writer smarter is a good thing.

However, in the case of one plane per TIFF file, the specification requires
a separate TiffData element per plane, to indicate the UUID of each file. So
I don't think you'll be able to achieve much savings there...

-Curtis

On Tue, Oct 12, 2010 at 2:48 PM, Melissa Linkert <
melissa at glencoesoftware.com> wrote:

> Hi Rubén,
>
> > @Curtis: Actually we face this problem with very large datasets instead.
> One
> > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M
> +
> > 5.5 M headers. This kind of storage can not be afforded by some groups.
>
> My guess is that a large portion of this metadata is occupied by
> TiffData elements.  One idea would be to update Bio-Formats' OME-TIFF
> writer
> to be a bit smarter about how TiffData elements are written, which may
> substantially shrink the size of the metadata in each file without
> requiring a schema change.
>
> I do agree with Curtis that we should at least consider making a change to
> the
> OME-XML schema, but in the meantime that may be a partial solution.
> However, as mentioned before, it would help to see the exact OME-XML
> that you are working with.  We wouldn't want to implement a solution
> only to find out that it doesn't solve your problem.
>
> > @Melissa: Do you think that a tool that could group or split datasets
> could
> > also deal with the Leica Matrix Screener OME.TIF. My tool will assume
> that
> > all the headers are the same, but I know that is not like that. Therefore
> I
> > think that unification of the headers is a good idea, in the future.
>
> It's possible, but I as mentioned before I really think you would be
> better off waiting for proper Leica Matrix support in Bio-Formats.
>
> Regards,
> -Melissa
>
> On Tue, Oct 12, 2010 at 02:20:28PM -0500, Curtis Rueden wrote:
> > Hi Rubén,
> >
> > @Curtis: Actually we face this problem with very large datasets instead.
> One
> > > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M =
> 2.5M +
> > > 5.5 M headers. This kind of storage can not be afforded by some groups.
> > >
> >
> > Thanks, that explanation helps a lot!
> >
> > It sounds like this is a use case for either:
> >
> > A) Putting the metadata in only the first .ome.tif file; or
> > B) Putting the metadata in a companion file.
> >
> > Each OME-TIFF would still need a small amount of metadata (particularly
> the
> > UUID associated with it) but the bulk of it could in principle be
> stripped
> > out.
> >
> > I am all in favor of either of these options being supported in the next
> > OME-XML schema, but in the past there has been strong opposition, due to
> > concern about metadata being lost by a partial copy. Perhaps this
> concrete
> > use case justifies reversing our decision on this? Anyone have any other
> > ideas?
> >
> > To split each serie in different ome.tif datasets seems like our
> solution.
> > > Do you think its convenient? Are there tools for this? I am working in
> one
> > > already.
> > >
> >
> > You could definitely do that, but it seems a shame. You would be
> > compromising the modeling of your data, just to work around an
> unfortunate
> > implementation detail of OME-TIFF. Better would be for us to provide a
> > better solution with the next schema release.
> >
> > -Curtis
> >
> > On Tue, Oct 12, 2010 at 1:32 PM, Rubén Muñoz <ruben.munoz at embl.de>
> wrote:
> >
> > > Hi Curtis and Melissa,
> > >
> > > On Oct 12, 2010, at 5:41 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
> > >
> > > Hi Rubén & Melissa,
> > >
> > > > Is there a way of excluding the image/pixel list in all the OME.TIF
> with
> > >> bfconvert?
> > >>
> > >> Perhaps I am misunderstanding the question, but are you asking about
> > >> excluding the Image and Pixels elements from the OME-XML metadata?
> > >
> > >
> > > Yes that's my question.
> > >
> > > If
> > >> so, then I'm afraid that that is really not possible - both elements
> > >> provide critical metadata for reading the pixel data.
> > >>
> > >
> > >
> > > Thats how I understand the specification. All the files are
> > > cross-referenced in the OME.TIF headers by Pixel elements,
> independently of
> > > the number of files in the set...
> > >
> > > Note: We store all the planes & timepoins & channels in different files
> for
> > > convenience, that's not a matter of discussion anymore here, since our
> > > software doesn't always operate on stacks.
> > >
> > > This could conceivably happen with extremely small image planes. For
> > > example, an 8x8 image at 8-bit resolution only occupies 64 bytes, but
> the
> > > metadata would be at least ~180 bytes; e.g.:
> > >
> > > <Image ID="Image:xxxx" Name="Image xxxx"><Pixels DimensionOrder="XYCZT"
> > > ID="Pixels:xxxx" SizeC="1" SizeT="1" SizeX="8" SizeY="8" SizeZ="1"
> > > Type="uint8"><TiffData/></Pixels></Image>
> > >
> > > I can't think of any other case where the metadata size exceeds the
> pixel
> > > size. Rubén, can you elaborate on what is going on? What are the
> dimensions
> > > of your images? How many screens, plates, wells, and what is the image
> > > resolution?
> > >
> > >
> > > @Curtis: Actually we face this problem with very large datasets
> instead.
> > > One of our time-lapse acquisitions has 23000 ome.tif files, each is 8M
> =
> > > 2.5M + 5.5 M headers. This kind of storage can not be afforded by some
> > > groups.
> > >
> > > My colleagues plan to make bigger experiments, and I we are facing
> storage
> > > issues. And bioformats needs more memory to convert large datasets.
> > >
> > > To split each serie in different ome.tif datasets seems like our
> solution.
> > > Do you think its convenient? Are there tools for this? I am working in
> one
> > > already.
> > >
> > > @Melissa: Do you think that a tool that could group or split datasets
> could
> > > also deal with the Leica Matrix Screener OME.TIF. My tool will assume
> that
> > > all the headers are the same, but I know that is not like that.
> Therefore I
> > > think that unification of the headers is a good idea, in the future.
> > >
> > > Thanks for your replies,
> > >
> > > Rubén
> > >
> > >
> > > -Curtis
> > >
> > > On Tue, Oct 12, 2010 at 10:23 AM, Melissa Linkert <<
> melissa at glencoesoftware.com>
> > > melissa at glencoesoftware.com> wrote:
> > >
> > >> Hi Rubén,
> > >>
> > >> > Is there a way of excluding the image/pixel list in all the OME.TIF
> with
> > >> bfconvert?
> > >>
> > >> Perhaps I am misunderstanding the question, but are you asking about
> > >> excluding the Image and Pixels elements from the OME-XML metadata?  If
> > >> so, then I'm afraid that that is really not possible - both elements
> > >> provide critical metadata for reading the pixel data.
> > >>
> > >> > For large multi-file sets the metadata is bigger than the pixel
> values
> > >> and this prevents us to use OME.TIF in High Throughput Screens.
> > >>
> > >> That is a bit strange, but I have a few ideas as to what might be
> > >> happening there.  I imagine the OME-TIFF file is prohibitively large,
> > >> but if you could send just the OME-XML metadata that would be a huge
> > >> help.  You can extract the metadata like this:
> > >>
> > >>  # if all of the Bio-Formats command-line tools are installed
> > >>  $ tiffcomment /path/to/file.ome.tiff
> > >>
> > >>  # if 'tiffcomment' is not installed
> > >>  $ java loci.formats.tools.TiffComment /path/to/file.ome.tiff
> > >>
> > >> > I'd like to help with the Leica Matrix Screener and ZEN importers as
> > >> much as possible.
> > >>
> > >> Thank you for the offer.  At the moment, there is not much that can be
> > >> done with the Zeiss LSM/Zen reader.  It will be at least a few weeks
> > >> before I can do anything with the Leica Matrix reader, but you are
> more
> > >> than welcome to start on a reader in the meantime if you like.  There
> > >> are three main pieces of work to be done for this reader:
> > >>
> > >>  1) Make sure that all of the files are grouped together correctly, no
> > >>  matter which file in the dataset is selected by the user.
> > >>
> > >>  2) Make sure that all of the pixel data and 'core' metadata is read
> > >>  correctly.
> > >>
> > >>  3) Make sure that all of the OME-XML metadata from each of the
> > >>  constituent OME-TIFFs is parsed and unified.
> > >>
> > >> My plan is to roughtly model the Leica Matrix reader after
> > >> loci.formats.in.MIASReader and loci.formats.in.InCellReader, with an
> > >> internal loci.formats.in.OMETiffReader doing much of the work for (2)
> > >> and (3).
> > >>
> > >> Alternatively, you can wait a bit until I've had time to write an
> initial
> > >> version, and then help with the testing and polishing.
> > >>
> > >> Regards,
> > >> -Melissa
> > >>
> > >> On Mon, Oct 11, 2010 at 09:51:56PM +0200, Rubén Muñoz wrote:
> > >> > Hi Melissa,
> > >> > Is there a way of excluding the image/pixel list in all the OME.TIF
> with
> > >> bfconvert?
> > >> > For large multi-file sets the metadata is bigger than the pixel
> values
> > >> and this prevents us to use OME.TIF in High Throughput Screens. I'd
> like to
> > >> help with the Leica Matrix Screener and ZEN importers as much as
> possible.
> > >> > Best regards,
> > >> > Rubén
> > >> >
> > >> > On Oct 11, 2010, at 8:22 PM, Melissa Linkert <<
> melissa at glencoesoftware.com>
> > >> melissa at glencoesoftware.com> wrote:
> > >> >
> > >> > > Hi Rubén,
> > >> > >
> > >> > >> That's good news. I am amazed by how the project evolves from
> simpler
> > >> to more sophisticated needs. Actually bfconvert was designed to
> convert I
> > >> guess, but I was trying to transform Leica Matrix Screener multiple
> OME.TIF
> > >> to one-big OME.TIF and then enrich the metadata.
> > >> > >
> > >> > > If possible, I would recommend waiting for this ticket to be
> closed:
> > >> > >
> > >> > > <http://dev.loci.wisc.edu/trac/software/ticket/563>
> > >> http://dev.loci.wisc.edu/trac/software/ticket/563
> > >> > >
> > >> > > ...at which point you can just do 'bfconvert
> > >> leica-matrix-file.ome.tiff output.ome.tiff' without any additional
> magic.
> > >> > >
> > >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> > >> metadata with EditTiffComment for that? Would a tool like
> EditTiffComment
> > >> that modifies the metadata of many files accordingly be usefull?
> > >> > >
> > >> > > You could use one of our TIFF comment editing utilities:
> > >> > >
> > >> > > * java loci.formats.tools.TiffComment -edit /path/to/file
> > >> > > * java loci.formats.tools.EditTiffG /path/to/file
> > >> > >
> > >> > > But please be very, very cautious.  Again, the above ticket really
> is
> > >> > > the best solution, so if you can I would recommend either waiting
> for
> > >> > > that ticket to be closed or (if you have a lot of free time)
> helping
> > >> to implement it.
> > >> > >
> > >> > > Regards,
> > >> > > -Melissa
> > >> > >
> > >> > > On Wed, Oct 06, 2010 at 02:46:50PM +0200, Rubén Muñoz wrote:
> > >> > >> Hi Melissa,
> > >> > >>
> > >> > >> On Oct 6, 2010, at 1:09 AM, Melissa Linkert wrote:
> > >> > >>
> > >> > >>> Hi Rubén,
> > >> > >>>
> > >> > >>>> Therefore tried this with bfconvert, even when appending to
> > >> existing OME.TIF is not supported yet.
> > >> > >>>
> > >> > >>> Appending to existing OME-TIFF files is, in theory, supported.
> > >>  However,
> > >> > >>> the MetadataRetrieve object used by OMETiffWriter must be
> correctly
> > >> set
> > >> > >>> up, which unfortunately will not happen by repeatedly calling
> > >> > >>> bfconvert.
> > >> > >>
> > >> > >> That's good news. I am amazed by how the project evolves from
> simpler
> > >> to more sophisticated needs. Actually bfconvert was designed to
> convert I
> > >> guess, but I was trying to transform Leica Matrix Screener multiple
> OME.TIF
> > >> to one-big OME.TIF and then enrich the metadata.
> > >> > >>
> > >> > >>>
> > >> > >>>> However wIth the latest revision of bioformats 7034 when I
> output
> > >> the bfconvert to the same file consecutively, the file is always
> getting
> > >> bigger in size.
> > >> > >>>
> > >> > >>> Yes, I see the same behavior.  It is ticketed here:
> > >> > >>>
> > >> > >>> <http://dev.loci.wisc.edu/trac/software/ticket/578>
> > >> http://dev.loci.wisc.edu/trac/software/ticket/578
> > >> > >>>
> > >> > >>> And, as usual, you have been CC'd.  I am a bit curious though:
> why
> > >> run
> > >> > >>> bfconvert repeatedly with the same output file?  Are you trying
> to
> > >> > >>> convert multiple datasets to a single OME-TIFF file?  If I know
> the
> > >> use
> > >> > >>> case, then maybe there is a work-around until the above ticket
> is
> > >> closed.
> > >> > >>
> > >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> > >> metadata with EditTiffComment for that? Would a tool like
> EditTiffComment
> > >> that modifies the metadata of many files accordingly be usefull?
> > >> > >>
> > >> > >> Another topic, I guess you remember about the Zeiss ZEN software,
> > >>  multiple positions in the same LSM file are now supported.
> > >> > >> I recently found out that it also produces multi-file LSM
> datasets
> > >> (without .mdb). I am getting one of these. I wonder how these files
> now
> > >> about each other, but could be that they don't.
> > >> > >>
> > >> > >> Best regards,
> > >> > >>
> > >> > >> Rubén
> > >> > >>
> > >> > >>
> > >> > >>
> > >> _______________________________________________
> > >> ome-devel mailing list
> > >>  <ome-devel at lists.openmicroscopy.org.uk>
> > >> ome-devel at lists.openmicroscopy.org.uk
> > >>  <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
> > >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> > >>
> > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20101012/455de588/attachment-0001.html>