[ome-devel] Zeiss 710

Tue Oct 12 20:20:28 BST 2010

Hi Rubén,

@Curtis: Actually we face this problem with very large datasets instead. One
> of our time-lapse acquisitions has 23000 ome.tif files, each is 8M = 2.5M +
> 5.5 M headers. This kind of storage can not be afforded by some groups.
>

Thanks, that explanation helps a lot!

It sounds like this is a use case for either:

A) Putting the metadata in only the first .ome.tif file; or
B) Putting the metadata in a companion file.

Each OME-TIFF would still need a small amount of metadata (particularly the
UUID associated with it) but the bulk of it could in principle be stripped
out.

I am all in favor of either of these options being supported in the next
OME-XML schema, but in the past there has been strong opposition, due to
concern about metadata being lost by a partial copy. Perhaps this concrete
use case justifies reversing our decision on this? Anyone have any other
ideas?

To split each serie in different ome.tif datasets seems like our solution.
> Do you think its convenient? Are there tools for this? I am working in one
> already.
>

You could definitely do that, but it seems a shame. You would be
compromising the modeling of your data, just to work around an unfortunate
implementation detail of OME-TIFF. Better would be for us to provide a
better solution with the next schema release.

-Curtis

On Tue, Oct 12, 2010 at 1:32 PM, Rubén Muñoz <ruben.munoz at embl.de> wrote:

> Hi Curtis and Melissa,
>
> On Oct 12, 2010, at 5:41 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
>
> Hi Rubén & Melissa,
>
> > Is there a way of excluding the image/pixel list in all the OME.TIF with
>> bfconvert?
>>
>> Perhaps I am misunderstanding the question, but are you asking about
>> excluding the Image and Pixels elements from the OME-XML metadata?
>
>
> Yes that's my question.
>
> If
>> so, then I'm afraid that that is really not possible - both elements
>> provide critical metadata for reading the pixel data.
>>
>
>
> Thats how I understand the specification. All the files are
> cross-referenced in the OME.TIF headers by Pixel elements, independently of
> the number of files in the set...
>
> Note: We store all the planes & timepoins & channels in different files for
> convenience, that's not a matter of discussion anymore here, since our
> software doesn't always operate on stacks.
>
> This could conceivably happen with extremely small image planes. For
> example, an 8x8 image at 8-bit resolution only occupies 64 bytes, but the
> metadata would be at least ~180 bytes; e.g.:
>
> <Image ID="Image:xxxx" Name="Image xxxx"><Pixels DimensionOrder="XYCZT"
> ID="Pixels:xxxx" SizeC="1" SizeT="1" SizeX="8" SizeY="8" SizeZ="1"
> Type="uint8"><TiffData/></Pixels></Image>
>
> I can't think of any other case where the metadata size exceeds the pixel
> size. Rubén, can you elaborate on what is going on? What are the dimensions
> of your images? How many screens, plates, wells, and what is the image
> resolution?
>
>
> @Curtis: Actually we face this problem with very large datasets instead.
> One of our time-lapse acquisitions has 23000 ome.tif files, each is 8M =
> 2.5M + 5.5 M headers. This kind of storage can not be afforded by some
> groups.
>
> My colleagues plan to make bigger experiments, and I we are facing storage
> issues. And bioformats needs more memory to convert large datasets.
>
> To split each serie in different ome.tif datasets seems like our solution.
> Do you think its convenient? Are there tools for this? I am working in one
> already.
>
> @Melissa: Do you think that a tool that could group or split datasets could
> also deal with the Leica Matrix Screener OME.TIF. My tool will assume that
> all the headers are the same, but I know that is not like that. Therefore I
> think that unification of the headers is a good idea, in the future.
>
> Thanks for your replies,
>
> Rubén
>
>
> -Curtis
>
> On Tue, Oct 12, 2010 at 10:23 AM, Melissa Linkert <<melissa at glencoesoftware.com>
> melissa at glencoesoftware.com> wrote:
>
>> Hi Rubén,
>>
>> > Is there a way of excluding the image/pixel list in all the OME.TIF with
>> bfconvert?
>>
>> Perhaps I am misunderstanding the question, but are you asking about
>> excluding the Image and Pixels elements from the OME-XML metadata?  If
>> so, then I'm afraid that that is really not possible - both elements
>> provide critical metadata for reading the pixel data.
>>
>> > For large multi-file sets the metadata is bigger than the pixel values
>> and this prevents us to use OME.TIF in High Throughput Screens.
>>
>> That is a bit strange, but I have a few ideas as to what might be
>> happening there.  I imagine the OME-TIFF file is prohibitively large,
>> but if you could send just the OME-XML metadata that would be a huge
>> help.  You can extract the metadata like this:
>>
>>  # if all of the Bio-Formats command-line tools are installed
>>  $ tiffcomment /path/to/file.ome.tiff
>>
>>  # if 'tiffcomment' is not installed
>>  $ java loci.formats.tools.TiffComment /path/to/file.ome.tiff
>>
>> > I'd like to help with the Leica Matrix Screener and ZEN importers as
>> much as possible.
>>
>> Thank you for the offer.  At the moment, there is not much that can be
>> done with the Zeiss LSM/Zen reader.  It will be at least a few weeks
>> before I can do anything with the Leica Matrix reader, but you are more
>> than welcome to start on a reader in the meantime if you like.  There
>> are three main pieces of work to be done for this reader:
>>
>>  1) Make sure that all of the files are grouped together correctly, no
>>  matter which file in the dataset is selected by the user.
>>
>>  2) Make sure that all of the pixel data and 'core' metadata is read
>>  correctly.
>>
>>  3) Make sure that all of the OME-XML metadata from each of the
>>  constituent OME-TIFFs is parsed and unified.
>>
>> My plan is to roughtly model the Leica Matrix reader after
>> loci.formats.in.MIASReader and loci.formats.in.InCellReader, with an
>> internal loci.formats.in.OMETiffReader doing much of the work for (2)
>> and (3).
>>
>> Alternatively, you can wait a bit until I've had time to write an initial
>> version, and then help with the testing and polishing.
>>
>> Regards,
>> -Melissa
>>
>> On Mon, Oct 11, 2010 at 09:51:56PM +0200, Rubén Muñoz wrote:
>> > Hi Melissa,
>> > Is there a way of excluding the image/pixel list in all the OME.TIF with
>> bfconvert?
>> > For large multi-file sets the metadata is bigger than the pixel values
>> and this prevents us to use OME.TIF in High Throughput Screens. I'd like to
>> help with the Leica Matrix Screener and ZEN importers as much as possible.
>> > Best regards,
>> > Rubén
>> >
>> > On Oct 11, 2010, at 8:22 PM, Melissa Linkert <<melissa at glencoesoftware.com>
>> melissa at glencoesoftware.com> wrote:
>> >
>> > > Hi Rubén,
>> > >
>> > >> That's good news. I am amazed by how the project evolves from simpler
>> to more sophisticated needs. Actually bfconvert was designed to convert I
>> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
>> to one-big OME.TIF and then enrich the metadata.
>> > >
>> > > If possible, I would recommend waiting for this ticket to be closed:
>> > >
>> > > <http://dev.loci.wisc.edu/trac/software/ticket/563>
>> http://dev.loci.wisc.edu/trac/software/ticket/563
>> > >
>> > > ...at which point you can just do 'bfconvert
>> leica-matrix-file.ome.tiff output.ome.tiff' without any additional magic.
>> > >
>> > >> To give dimensionally to a dataset is my goal. Could one edit the
>> metadata with EditTiffComment for that? Would a tool like EditTiffComment
>> that modifies the metadata of many files accordingly be usefull?
>> > >
>> > > You could use one of our TIFF comment editing utilities:
>> > >
>> > > * java loci.formats.tools.TiffComment -edit /path/to/file
>> > > * java loci.formats.tools.EditTiffG /path/to/file
>> > >
>> > > But please be very, very cautious.  Again, the above ticket really is
>> > > the best solution, so if you can I would recommend either waiting for
>> > > that ticket to be closed or (if you have a lot of free time) helping
>> to implement it.
>> > >
>> > > Regards,
>> > > -Melissa
>> > >
>> > > On Wed, Oct 06, 2010 at 02:46:50PM +0200, Rubén Muñoz wrote:
>> > >> Hi Melissa,
>> > >>
>> > >> On Oct 6, 2010, at 1:09 AM, Melissa Linkert wrote:
>> > >>
>> > >>> Hi Rubén,
>> > >>>
>> > >>>> Therefore tried this with bfconvert, even when appending to
>> existing OME.TIF is not supported yet.
>> > >>>
>> > >>> Appending to existing OME-TIFF files is, in theory, supported.
>>  However,
>> > >>> the MetadataRetrieve object used by OMETiffWriter must be correctly
>> set
>> > >>> up, which unfortunately will not happen by repeatedly calling
>> > >>> bfconvert.
>> > >>
>> > >> That's good news. I am amazed by how the project evolves from simpler
>> to more sophisticated needs. Actually bfconvert was designed to convert I
>> guess, but I was trying to transform Leica Matrix Screener multiple OME.TIF
>> to one-big OME.TIF and then enrich the metadata.
>> > >>
>> > >>>
>> > >>>> However wIth the latest revision of bioformats 7034 when I output
>> the bfconvert to the same file consecutively, the file is always getting
>> bigger in size.
>> > >>>
>> > >>> Yes, I see the same behavior.  It is ticketed here:
>> > >>>
>> > >>> <http://dev.loci.wisc.edu/trac/software/ticket/578>
>> http://dev.loci.wisc.edu/trac/software/ticket/578
>> > >>>
>> > >>> And, as usual, you have been CC'd.  I am a bit curious though: why
>> run
>> > >>> bfconvert repeatedly with the same output file?  Are you trying to
>> > >>> convert multiple datasets to a single OME-TIFF file?  If I know the
>> use
>> > >>> case, then maybe there is a work-around until the above ticket is
>> closed.
>> > >>
>> > >> To give dimensionally to a dataset is my goal. Could one edit the
>> metadata with EditTiffComment for that? Would a tool like EditTiffComment
>> that modifies the metadata of many files accordingly be usefull?
>> > >>
>> > >> Another topic, I guess you remember about the Zeiss ZEN software,
>>  multiple positions in the same LSM file are now supported.
>> > >> I recently found out that it also produces multi-file LSM datasets
>> (without .mdb). I am getting one of these. I wonder how these files now
>> about each other, but could be that they don't.
>> > >>
>> > >> Best regards,
>> > >>
>> > >> Rubén
>> > >>
>> > >>
>> > >>
>> _______________________________________________
>> ome-devel mailing list
>>  <ome-devel at lists.openmicroscopy.org.uk>
>> ome-devel at lists.openmicroscopy.org.uk
>>  <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel>
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20101012/d0409597/attachment-0001.html>