[ome-devel] Zeiss 710

[Rubén Muñoz]

Hi Melissa and Curtis, 
I had a somehow funny idea which would be to include one TiffData element in each file with the next file in the dataset only. The last one would reference the first one closing the circle. I would like to ask you:
1) is it crazy?
2) is it difficult to implement?
3) it definitivelly optimizes the disk storage but, will it require more disk accesses for conversion and integrity checks?

Best regards.
Rubén


On Oct 12, 2010, at 9:57 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:

> Hi Melissa,
> 
> My guess is that a large portion of this metadata is occupied by
> TiffData elements.
> 
> Indeed.
>  
> One idea would be to update Bio-Formats' OME-TIFF writer
> to be a bit smarter about how TiffData elements are written, which may
> substantially shrink the size of the metadata in each file without
> requiring a schema change.
> 
> In general I agree that making the OME-TIFF writer smarter is a good thing.
> 
> However, in the case of one plane per TIFF file, the specification requires a separate TiffData element per plane, to indicate the UUID of each file. So I don't think you'll be able to achieve much savings there...
> 
> -Curtis
> 
> On Tue, Oct 12, 2010 at 2:48 PM, Melissa Linkert <melissa at glencoesoftware.com> wrote:
> Hi Rubén,
> 
> > @Curtis: Actually we face this problem with very large datasets instead. One
> > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +
> > 5.5 M headers. This kind of storage can not be afforded by some groups.
> 
> My guess is that a large portion of this metadata is occupied by
> TiffData elements.  One idea would be to update Bio-Formats' OME-TIFF writer
> to be a bit smarter about how TiffData elements are written, which may
> substantially shrink the size of the metadata in each file without
> requiring a schema change.
> 
> I do agree with Curtis that we should at least consider making a change to the
> OME-XML schema, but in the meantime that may be a partial solution.
> However, as mentioned before, it would help to see the exact OME-XML
> that you are working with.  We wouldn't want to implement a solution
> only to find out that it doesn't solve your problem.
> 
> > @Melissa: Do you think that a tool that could group or split datasets could
> > also deal with the Leica Matrix Screener OME.TIF. My tool will assume that
> > all the headers are the same, but I know that is not like that. Therefore I
> > think that unification of the headers is a good idea, in the future.
> 
> It's possible, but I as mentioned before I really think you would be
> better off waiting for proper Leica Matrix support in Bio-Formats.
> 
> Regards,
> -Melissa
> 
> On Tue, Oct 12, 2010 at 02:20:28PM -0500, Curtis Rueden wrote:
> > Hi Rubén,
> >
> > @Curtis: Actually we face this problem with very large datasets instead. One
> > > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +
> > > 5.5 M headers. This kind of storage can not be afforded by some groups.
> > >
> >
> > Thanks, that explanation helps a lot!
> >
> > It sounds like this is a use case for either:
> >
> > A) Putting the metadata in only the first .ome.tif file; or
> > B) Putting the metadata in a companion file.
> >
> > Each OME-TIFF would still need a small amount of metadata (particularly the
> > UUID associated with it) but the bulk of it could in principle be stripped
> > out.
> >
> > I am all in favor of either of these options being supported in the next
> > OME-XML schema, but in the past there has been strong opposition, due to
> > concern about metadata being lost by a partial copy. Perhaps this concrete
> > use case justifies reversing our decision on this? Anyone have any other
> > ideas?
> >
> > To split each serie in different ome.tif datasets seems like our solution.
> > > Do you think its convenient? Are there tools for this? I am working in one
> > > already.
> > >
> >
> > You could definitely do that, but it seems a shame. You would be
> > compromising the modeling of your data, just to work around an unfortunate
> > implementation detail of OME-TIFF. Better would be for us to provide a
> > better solution with the next schema release.
> >
> > -Curtis
> >
> > On Tue, Oct 12, 2010 at 1:32 PM, Rubén Muñoz <ruben.munoz at embl.de> wrote:
> >
> > > Hi Curtis and Melissa,
> > >
> > > On Oct 12, 2010, at 5:41 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
> > >
> > > Hi Rubén & Melissa,
> > >
> > > > Is there a way of excluding the image/pixel list in all the OME.TIF with
> > >> bfconvert?
> > >>
> > >> Perhaps I am misunderstanding the question, but are you asking about
> > >> excluding the Image and Pixels elements from the OME-XML metadata?
> > >
> > >
> > > Yes that's my question.
> > >
> > > If
> > >> so, then I'm afraid that that is really not possible - both elements
> > >> provide critical metadata for reading the pixel data.
> > >>
> > >
> > >
> > > Thats how I understand the specification. All the files are
> > > cross-referenced in the OME.TIF headers by Pixel elements, independently of
> > > the number of files in the set...
> > >
> > > Note: We store all the planes & timepoins & channels in different files for
> > > convenience, that's not a matter of discussion anymore here, since our
> > > software doesn't always operate on stacks.
> > >
> > > This could conceivably happen with extremely small image planes. For
> > > example, an 8x8 image at 8-bit resolution only occupies 64 bytes, but the
> > > metadata would be at least ~180 bytes; e.g.:
> > >
> > > <Image ID="Image:xxxx" Name="Image xxxx"><Pixels DimensionOrder="XYCZT"
> > > ID="Pixels:xxxx" SizeC="1" SizeT="1" SizeX="8" SizeY="8" SizeZ="1"
> > > Type="uint8"><TiffData/></Pixels></Image>
> > >
> > > I can't think of any other case where the metadata size exceeds the pixel
> > > size. Rubén, can you elaborate on what is going on? What are the dimensions
> > > of your images? How many screens, plates, wells, and what is the image
> > > resolution?
> > >
> > >
> > > @Curtis: Actually we face this problem with very large datasets instead.
> > > One of our time-lapse acquisitions has 23000 ome.tif files, each is 8M =
> > > 2.5M + 5.5 M headers. This kind of storage can not be afforded by some
> > > groups.
> > >
> > > My colleagues plan to make bigger experiments, and I we are facing storage
> > > issues. And bioformats needs more memory to convert large datasets.
> > >
> > > To split each serie in different ome.tif datasets seems like our solution.
> > > Do you think its convenient? Are there tools for this? I am working in one
> > > already.
> > >
> > > @Melissa: Do you think that a tool that could group or split datasets could
> > > also deal with the Leica Matrix Screener OME.TIF. My tool will assume that
> > > all the headers are the same, but I know that is not like that. Therefore I
> > > think that unification of the headers is a good idea, in the future.
> > >
> > > Thanks for your replies,
> > >
> > > Rubén
> > >
> > >
> > > -Curtis
> > >
> > > On Tue, Oct 12, 2010 at 10:23 AM, Melissa Linkert <<melissa at glencoesoftware.com>
> > > melissa at glencoesoftware.com> wrote:
> > >
> > >> Hi Rubén,
> > >>
> > >> > Is there a way of excluding the image/pixel list in all the OME.TIF with
> > >> bfconvert?
> > >>
> > >> Perhaps I am misunderstanding the question, but are you asking about
> > >> excluding the Image and Pixels elements from the OME-XML metadata?  If
> > >> so, then I'm afraid that that is really not possible - both elements
> > >> provide critical metadata for reading the pixel data.
> > >>
> > >> > For large multi-file sets the metadata is bigger than the pixel values
> > >> and this prevents us to use OME.TIF in High Throughput Screens.
> > >>
> > >> That is a bit strange, but I have a few ideas as to what might be
> > >> happening there.  I imagine the OME-TIFF file is prohibitively large,
> > >> but if you could send just the OME-XML metadata that would be a huge
> > >> help.  You can extract the metadata like this:
> > >>
> > >>  # if all of the Bio-Formats command-line tools are installed
> > >>  $ tiffcomment /path/to/file.ome.tiff
> > >>
> > >>  # if 'tiffcomment' is not installed
> > >>  $ java loci.formats.tools.TiffComment /path/to/file.ome.tiff
> > >>
> > >> > I'd like to help with the Leica Matrix Screener and ZEN importers as
> > >> much as possible.
> > >>
> > >> Thank you for the offer.  At the moment, there is not much that can be
> > >> done with the Zeiss LSM/Zen reader.  It will be at least a few weeks
> > >> before I can do anything with the Leica Matrix reader, but you are more
> > >> than welcome to start on a reader in the meantime if you like.  There
> > >> are three main pieces of work to be done for this reader:
> > >>
> > >>  1) Make sure that all of the files are grouped together correctly, no
> > >>  matter which file in the dataset is selected by the user.
> > >>
> > >>  2) Make sure that all of the pixel data and 'core' metadata is read
> > >>  correctly.
> > >>
> > >>  3) Make sure that all of the OME-XML metadata from each of the
> > >>  constituent OME-TIFFs is parsed and unified.
> > >>
> > >> My plan is to roughtly model the Leica Matrix reader after
> > >> loci.formats.in.MIASReader and loci.formats.in.InCellReader, with an
> > >> internal loci.formats.in.OMETiffReader doing much of the work for (2)
> > >> and (3).
> > >>
> > >> Alternatively, you can wait a bit until I've had time to write an initial
> > >> version, and then help with the testing and polishing.
> > >>
> > >> Regards,
> > >> -Melissa
> > >>
> > >> On Mon, Oct 11, 2010 at 09:51:56PM +0200, Rubén Muñoz wrote:
> > >> > Hi Melissa,
> > >> > Is there a way of excluding the image/pixel list in all the OME.TIF with
> > >> bfconvert?
> > >> > For large multi-file sets the metadata is bigger than the pixel values
> > >> and this prevents us to use OME.TIF in High Throughput Screens. I'd like to
> > >> help with the Leica Matrix Screener and ZEN importers as much as possible.
> > >> > Best regards,
> > >> > Rubén
> > >> >
> > >> > On Oct 11, 2010, at 8:22 PM, Melissa Linkert <<melissa at glencoesoftware.com>
> > >> melissa at glencoesoftware.com> wrote:
> > >> >
> > >> > > Hi Rubén,
> > >> > >
> > >> > >> That's good news. I am amazed by how the project evolves from simpler
> > >> to more sophisticated needs. Actually bfconvert was designed to convert I
> > >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
> > >> to one-big OME.TIF and then enrich the metadata.
> > >> > >
> > >> > > If possible, I would recommend waiting for this ticket to be closed:
> > >> > >
> > >> > > <http://dev.loci.wisc.edu/trac/software/ticket/563>
> > >> http://dev.loci.wisc.edu/trac/software/ticket/563
> > >> > >
> > >> > > ...at which point you can just do 'bfconvert
> > >> leica-matrix-file.ome.tiff output.ome.tiff' without any additional magic.
> > >> > >
> > >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> > >> metadata with EditTiffComment for that? Would a tool like EditTiffComment
> > >> that modifies the metadata of many files accordingly be usefull?
> > >> > >
> > >> > > You could use one of our TIFF comment editing utilities:
> > >> > >
> > >> > > * java loci.formats.tools.TiffComment -edit /path/to/file
> > >> > > * java loci.formats.tools.EditTiffG /path/to/file
> > >> > >
> > >> > > But please be very, very cautious.  Again, the above ticket really is
> > >> > > the best solution, so if you can I would recommend either waiting for
> > >> > > that ticket to be closed or (if you have a lot of free time) helping
> > >> to implement it.
> > >> > >
> > >> > > Regards,
> > >> > > -Melissa
> > >> > >
> > >> > > On Wed, Oct 06, 2010 at 02:46:50PM +0200, Rubén Muñoz wrote:
> > >> > >> Hi Melissa,
> > >> > >>
> > >> > >> On Oct 6, 2010, at 1:09 AM, Melissa Linkert wrote:
> > >> > >>
> > >> > >>> Hi Rubén,
> > >> > >>>
> > >> > >>>> Therefore tried this with bfconvert, even when appending to
> > >> existing OME.TIF is not supported yet.
> > >> > >>>
> > >> > >>> Appending to existing OME-TIFF files is, in theory, supported.
> > >>  However,
> > >> > >>> the MetadataRetrieve object used by OMETiffWriter must be correctly
> > >> set
> > >> > >>> up, which unfortunately will not happen by repeatedly calling
> > >> > >>> bfconvert.
> > >> > >>
> > >> > >> That's good news. I am amazed by how the project evolves from simpler
> > >> to more sophisticated needs. Actually bfconvert was designed to convert I
> > >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
> > >> to one-big OME.TIF and then enrich the metadata.
> > >> > >>
> > >> > >>>
> > >> > >>>> However wIth the latest revision of bioformats 7034 when I output
> > >> the bfconvert to the same file consecutively, the file is always getting
> > >> bigger in size.
> > >> > >>>
> > >> > >>> Yes, I see the same behavior.  It is ticketed here:
> > >> > >>>
> > >> > >>> <http://dev.loci.wisc.edu/trac/software/ticket/578>
> > >> http://dev.loci.wisc.edu/trac/software/ticket/578
> > >> > >>>
> > >> > >>> And, as usual, you have been CC'd.  I am a bit curious though: why
> > >> run
> > >> > >>> bfconvert repeatedly with the same output file?  Are you trying to
> > >> > >>> convert multiple datasets to a single OME-TIFF file?  If I know the
> > >> use
> > >> > >>> case, then maybe there is a work-around until the above ticket is
> > >> closed.
> > >> > >>
> > >> > >> To give dimensionally to a dataset is my goal. Could one edit the
> > >> metadata with EditTiffComment for that? Would a tool like EditTiffComment
> > >> that modifies the metadata of many files accordingly be usefull?
> > >> > >>
> > >> > >> Another topic, I guess you remember about the Zeiss ZEN software,
> > >>  multiple positions in the same LSM file are now supported.
> > >> > >> I recently found out that it also produces multi-file LSM datasets
> > >> (without .mdb). I am getting one of these. I wonder how these files now
> > >> about each other, but could be that they don't.
> > >> > >>
> > >> > >> Best regards,
> > >> > >>
> > >> > >> Rubén
> > >> > >>
> > >> > >>
> > >> > >>
> > >> _______________________________________________
> > >> ome-devel mailing list
> > >>  <ome-devel at lists.openmicroscopy.org.uk>
> > >> ome-devel at lists.openmicroscopy.org.uk
> > >>  <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
> > >> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> > >>
> > >
> > >
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20101013/48205d94/attachment-0001.html>