Hi Melissa,<br><br><blockquote style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;" class="gmail_quote">My guess is that a large portion of this metadata is occupied by<br>
TiffData elements.</blockquote><div><br>Indeed.<br> </div><blockquote style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;" class="gmail_quote">One idea would be to update Bio-Formats' OME-TIFF writer<br>
to be a bit smarter about how TiffData elements are written, which may<br>
substantially shrink the size of the metadata in each file without<br>
requiring a schema change.<br></blockquote><br>In general I agree that making the OME-TIFF writer smarter is a good thing.<br><br>However, in the case of one plane per TIFF file, the specification requires a separate TiffData element per plane, to indicate the UUID of each file. So I don't think you'll be able to achieve much savings there...<br>
<br>-Curtis<br><br><div class="gmail_quote">On Tue, Oct 12, 2010 at 2:48 PM, Melissa Linkert <span dir="ltr"><<a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">Hi Rubén,<br>
<br>
> @Curtis: Actually we face this problem with very large datasets instead. One<br>
> of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +<br>
> 5.5 M headers. This kind of storage can not be afforded by some groups.<br>
<br>
</div>My guess is that a large portion of this metadata is occupied by<br>
TiffData elements. One idea would be to update Bio-Formats' OME-TIFF writer<br>
to be a bit smarter about how TiffData elements are written, which may<br>
substantially shrink the size of the metadata in each file without<br>
requiring a schema change.<br>
<br>
I do agree with Curtis that we should at least consider making a change to the<br>
OME-XML schema, but in the meantime that may be a partial solution.<br>
However, as mentioned before, it would help to see the exact OME-XML<br>
that you are working with. We wouldn't want to implement a solution<br>
only to find out that it doesn't solve your problem.<br>
<div class="im"><br>
> @Melissa: Do you think that a tool that could group or split datasets could<br>
> also deal with the Leica Matrix Screener OME.TIF. My tool will assume that<br>
> all the headers are the same, but I know that is not like that. Therefore I<br>
> think that unification of the headers is a good idea, in the future.<br>
<br>
</div>It's possible, but I as mentioned before I really think you would be<br>
better off waiting for proper Leica Matrix support in Bio-Formats.<br>
<br>
Regards,<br>
<font color="#888888">-Melissa<br>
</font><div><div></div><div class="h5"><br>
On Tue, Oct 12, 2010 at 02:20:28PM -0500, Curtis Rueden wrote:<br>
> Hi Rubén,<br>
><br>
> @Curtis: Actually we face this problem with very large datasets instead. One<br>
> > of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +<br>
> > 5.5 M headers. This kind of storage can not be afforded by some groups.<br>
> ><br>
><br>
> Thanks, that explanation helps a lot!<br>
><br>
> It sounds like this is a use case for either:<br>
><br>
> A) Putting the metadata in only the first .ome.tif file; or<br>
> B) Putting the metadata in a companion file.<br>
><br>
> Each OME-TIFF would still need a small amount of metadata (particularly the<br>
> UUID associated with it) but the bulk of it could in principle be stripped<br>
> out.<br>
><br>
> I am all in favor of either of these options being supported in the next<br>
> OME-XML schema, but in the past there has been strong opposition, due to<br>
> concern about metadata being lost by a partial copy. Perhaps this concrete<br>
> use case justifies reversing our decision on this? Anyone have any other<br>
> ideas?<br>
><br>
> To split each serie in different ome.tif datasets seems like our solution.<br>
> > Do you think its convenient? Are there tools for this? I am working in one<br>
> > already.<br>
> ><br>
><br>
> You could definitely do that, but it seems a shame. You would be<br>
> compromising the modeling of your data, just to work around an unfortunate<br>
> implementation detail of OME-TIFF. Better would be for us to provide a<br>
> better solution with the next schema release.<br>
><br>
> -Curtis<br>
><br>
> On Tue, Oct 12, 2010 at 1:32 PM, Rubén Muñoz <<a href="mailto:ruben.munoz@embl.de">ruben.munoz@embl.de</a>> wrote:<br>
><br>
> > Hi Curtis and Melissa,<br>
> ><br>
> > On Oct 12, 2010, at 5:41 PM, Curtis Rueden <<a href="mailto:ctrueden@wisc.edu">ctrueden@wisc.edu</a>> wrote:<br>
> ><br>
> > Hi Rubén & Melissa,<br>
> ><br>
> > > Is there a way of excluding the image/pixel list in all the OME.TIF with<br>
> >> bfconvert?<br>
> >><br>
> >> Perhaps I am misunderstanding the question, but are you asking about<br>
> >> excluding the Image and Pixels elements from the OME-XML metadata?<br>
> ><br>
> ><br>
> > Yes that's my question.<br>
> ><br>
> > If<br>
> >> so, then I'm afraid that that is really not possible - both elements<br>
> >> provide critical metadata for reading the pixel data.<br>
> >><br>
> ><br>
> ><br>
> > Thats how I understand the specification. All the files are<br>
> > cross-referenced in the OME.TIF headers by Pixel elements, independently of<br>
> > the number of files in the set...<br>
> ><br>
> > Note: We store all the planes & timepoins & channels in different files for<br>
> > convenience, that's not a matter of discussion anymore here, since our<br>
> > software doesn't always operate on stacks.<br>
> ><br>
> > This could conceivably happen with extremely small image planes. For<br>
> > example, an 8x8 image at 8-bit resolution only occupies 64 bytes, but the<br>
> > metadata would be at least ~180 bytes; e.g.:<br>
> ><br>
> > <Image ID="Image:xxxx" Name="Image xxxx"><Pixels DimensionOrder="XYCZT"<br>
> > ID="Pixels:xxxx" SizeC="1" SizeT="1" SizeX="8" SizeY="8" SizeZ="1"<br>
> > Type="uint8"><TiffData/></Pixels></Image><br>
> ><br>
> > I can't think of any other case where the metadata size exceeds the pixel<br>
> > size. Rubén, can you elaborate on what is going on? What are the dimensions<br>
> > of your images? How many screens, plates, wells, and what is the image<br>
> > resolution?<br>
> ><br>
> ><br>
> > @Curtis: Actually we face this problem with very large datasets instead.<br>
> > One of our time-lapse acquisitions has 23000 ome.tif files, each is 8M =<br>
> > 2.5M + 5.5 M headers. This kind of storage can not be afforded by some<br>
> > groups.<br>
> ><br>
> > My colleagues plan to make bigger experiments, and I we are facing storage<br>
> > issues. And bioformats needs more memory to convert large datasets.<br>
> ><br>
> > To split each serie in different ome.tif datasets seems like our solution.<br>
> > Do you think its convenient? Are there tools for this? I am working in one<br>
> > already.<br>
> ><br>
> > @Melissa: Do you think that a tool that could group or split datasets could<br>
> > also deal with the Leica Matrix Screener OME.TIF. My tool will assume that<br>
> > all the headers are the same, but I know that is not like that. Therefore I<br>
> > think that unification of the headers is a good idea, in the future.<br>
> ><br>
> > Thanks for your replies,<br>
> ><br>
> > Rubén<br>
> ><br>
> ><br>
> > -Curtis<br>
> ><br>
> > On Tue, Oct 12, 2010 at 10:23 AM, Melissa Linkert <<<a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>><br>
> > <a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>> wrote:<br>
> ><br>
> >> Hi Rubén,<br>
> >><br>
> >> > Is there a way of excluding the image/pixel list in all the OME.TIF with<br>
> >> bfconvert?<br>
> >><br>
> >> Perhaps I am misunderstanding the question, but are you asking about<br>
> >> excluding the Image and Pixels elements from the OME-XML metadata? If<br>
> >> so, then I'm afraid that that is really not possible - both elements<br>
> >> provide critical metadata for reading the pixel data.<br>
> >><br>
> >> > For large multi-file sets the metadata is bigger than the pixel values<br>
> >> and this prevents us to use OME.TIF in High Throughput Screens.<br>
> >><br>
> >> That is a bit strange, but I have a few ideas as to what might be<br>
> >> happening there. I imagine the OME-TIFF file is prohibitively large,<br>
> >> but if you could send just the OME-XML metadata that would be a huge<br>
> >> help. You can extract the metadata like this:<br>
> >><br>
> >> # if all of the Bio-Formats command-line tools are installed<br>
> >> $ tiffcomment /path/to/file.ome.tiff<br>
> >><br>
> >> # if 'tiffcomment' is not installed<br>
> >> $ java loci.formats.tools.TiffComment /path/to/file.ome.tiff<br>
> >><br>
> >> > I'd like to help with the Leica Matrix Screener and ZEN importers as<br>
> >> much as possible.<br>
> >><br>
> >> Thank you for the offer. At the moment, there is not much that can be<br>
> >> done with the Zeiss LSM/Zen reader. It will be at least a few weeks<br>
> >> before I can do anything with the Leica Matrix reader, but you are more<br>
> >> than welcome to start on a reader in the meantime if you like. There<br>
> >> are three main pieces of work to be done for this reader:<br>
> >><br>
> >> 1) Make sure that all of the files are grouped together correctly, no<br>
> >> matter which file in the dataset is selected by the user.<br>
> >><br>
> >> 2) Make sure that all of the pixel data and 'core' metadata is read<br>
> >> correctly.<br>
> >><br>
> >> 3) Make sure that all of the OME-XML metadata from each of the<br>
> >> constituent OME-TIFFs is parsed and unified.<br>
> >><br>
> >> My plan is to roughtly model the Leica Matrix reader after<br>
> >> loci.formats.in.MIASReader and loci.formats.in.InCellReader, with an<br>
> >> internal loci.formats.in.OMETiffReader doing much of the work for (2)<br>
> >> and (3).<br>
> >><br>
> >> Alternatively, you can wait a bit until I've had time to write an initial<br>
> >> version, and then help with the testing and polishing.<br>
> >><br>
> >> Regards,<br>
> >> -Melissa<br>
> >><br>
> >> On Mon, Oct 11, 2010 at 09:51:56PM +0200, Rubén Muñoz wrote:<br>
> >> > Hi Melissa,<br>
> >> > Is there a way of excluding the image/pixel list in all the OME.TIF with<br>
> >> bfconvert?<br>
> >> > For large multi-file sets the metadata is bigger than the pixel values<br>
> >> and this prevents us to use OME.TIF in High Throughput Screens. I'd like to<br>
> >> help with the Leica Matrix Screener and ZEN importers as much as possible.<br>
> >> > Best regards,<br>
> >> > Rubén<br>
> >> ><br>
> >> > On Oct 11, 2010, at 8:22 PM, Melissa Linkert <<<a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>><br>
> >> <a href="mailto:melissa@glencoesoftware.com">melissa@glencoesoftware.com</a>> wrote:<br>
> >> ><br>
> >> > > Hi Rubén,<br>
> >> > ><br>
> >> > >> That's good news. I am amazed by how the project evolves from simpler<br>
> >> to more sophisticated needs. Actually bfconvert was designed to convert I<br>
> >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF<br>
> >> to one-big OME.TIF and then enrich the metadata.<br>
> >> > ><br>
> >> > > If possible, I would recommend waiting for this ticket to be closed:<br>
> >> > ><br>
> >> > > <<a href="http://dev.loci.wisc.edu/trac/software/ticket/563" target="_blank">http://dev.loci.wisc.edu/trac/software/ticket/563</a>><br>
> >> <a href="http://dev.loci.wisc.edu/trac/software/ticket/563" target="_blank">http://dev.loci.wisc.edu/trac/software/ticket/563</a><br>
> >> > ><br>
> >> > > ...at which point you can just do 'bfconvert<br>
> >> leica-matrix-file.ome.tiff output.ome.tiff' without any additional magic.<br>
> >> > ><br>
> >> > >> To give dimensionally to a dataset is my goal. Could one edit the<br>
> >> metadata with EditTiffComment for that? Would a tool like EditTiffComment<br>
> >> that modifies the metadata of many files accordingly be usefull?<br>
> >> > ><br>
> >> > > You could use one of our TIFF comment editing utilities:<br>
> >> > ><br>
> >> > > * java loci.formats.tools.TiffComment -edit /path/to/file<br>
> >> > > * java loci.formats.tools.EditTiffG /path/to/file<br>
> >> > ><br>
> >> > > But please be very, very cautious. Again, the above ticket really is<br>
> >> > > the best solution, so if you can I would recommend either waiting for<br>
> >> > > that ticket to be closed or (if you have a lot of free time) helping<br>
> >> to implement it.<br>
> >> > ><br>
> >> > > Regards,<br>
> >> > > -Melissa<br>
> >> > ><br>
> >> > > On Wed, Oct 06, 2010 at 02:46:50PM +0200, Rubén Muñoz wrote:<br>
> >> > >> Hi Melissa,<br>
> >> > >><br>
> >> > >> On Oct 6, 2010, at 1:09 AM, Melissa Linkert wrote:<br>
> >> > >><br>
> >> > >>> Hi Rubén,<br>
> >> > >>><br>
> >> > >>>> Therefore tried this with bfconvert, even when appending to<br>
> >> existing OME.TIF is not supported yet.<br>
> >> > >>><br>
> >> > >>> Appending to existing OME-TIFF files is, in theory, supported.<br>
> >> However,<br>
> >> > >>> the MetadataRetrieve object used by OMETiffWriter must be correctly<br>
> >> set<br>
> >> > >>> up, which unfortunately will not happen by repeatedly calling<br>
> >> > >>> bfconvert.<br>
> >> > >><br>
> >> > >> That's good news. I am amazed by how the project evolves from simpler<br>
> >> to more sophisticated needs. Actually bfconvert was designed to convert I<br>
> >> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF<br>
> >> to one-big OME.TIF and then enrich the metadata.<br>
> >> > >><br>
> >> > >>><br>
> >> > >>>> However wIth the latest revision of bioformats 7034 when I output<br>
> >> the bfconvert to the same file consecutively, the file is always getting<br>
> >> bigger in size.<br>
> >> > >>><br>
> >> > >>> Yes, I see the same behavior. It is ticketed here:<br>
> >> > >>><br>
> >> > >>> <<a href="http://dev.loci.wisc.edu/trac/software/ticket/578" target="_blank">http://dev.loci.wisc.edu/trac/software/ticket/578</a>><br>
> >> <a href="http://dev.loci.wisc.edu/trac/software/ticket/578" target="_blank">http://dev.loci.wisc.edu/trac/software/ticket/578</a><br>
> >> > >>><br>
> >> > >>> And, as usual, you have been CC'd. I am a bit curious though: why<br>
> >> run<br>
> >> > >>> bfconvert repeatedly with the same output file? Are you trying to<br>
> >> > >>> convert multiple datasets to a single OME-TIFF file? If I know the<br>
> >> use<br>
> >> > >>> case, then maybe there is a work-around until the above ticket is<br>
> >> closed.<br>
> >> > >><br>
> >> > >> To give dimensionally to a dataset is my goal. Could one edit the<br>
> >> metadata with EditTiffComment for that? Would a tool like EditTiffComment<br>
> >> that modifies the metadata of many files accordingly be usefull?<br>
> >> > >><br>
> >> > >> Another topic, I guess you remember about the Zeiss ZEN software,<br>
> >> multiple positions in the same LSM file are now supported.<br>
> >> > >> I recently found out that it also produces multi-file LSM datasets<br>
> >> (without .mdb). I am getting one of these. I wonder how these files now<br>
> >> about each other, but could be that they don't.<br>
> >> > >><br>
> >> > >> Best regards,<br>
> >> > >><br>
> >> > >> Rubén<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> _______________________________________________<br>
> >> ome-devel mailing list<br>
> >> <<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
> >> <a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
> >> <<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a>><br>
> >> <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
> >><br>
> ><br>
> ><br>
</div></div></blockquote></div><br>