<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<span class="">Hi Kai, Mario, Christian and others,<br class="">
</span><span class=""><br class="">
</span><span class="">As of 2016, the only native OME file format available for storing HCS data remains OME-TIFF. In order to reduce the number of TIFF files, it is possible to either use the BigTIFF extension [1] or use multi-image TIFFs for instance by storing
one TIFF per will containing 5D images. Interestingly, this is an approach undertaken by some manufacturers like Olympus ScanR [2].<br class="">
</span><span class=""><br class="">
</span><span class="">The usage of HDF file formats has certainly been raised both in the public mailing lists [3,4] and several conferences and at recent OME Users Meetings. This topic is still under investigation on our side and no concrete roadmap has been
assigned.<br class="">
</span><span class=""><br class="">
</span><span class="">Based on our experience with designing file formats, drafting a specification is necessary but not sufficient. As an example, we are still dealing weekly with invalid OME-TIFF produced by the community. In our experience, reference writing,
reading and validation tools need to be provided in addition that support and document any format that will be used by the scientific community.<br class="">
</span><span class=""><br class="">
</span><span class="">This is exactly why our recent efforts have been focused on building a reference implementation for writing OME-TIFF in C++. This library is now up-to-date with the latest model and we will use it as a driver for any necessary extension
support more complex heterogeneous metadata but also new container formats like HDF that offer solutions for HCS, multi-resolution, and multi-view.<br class="">
</span><span class=""><br class="">
</span><span class="">Our hope is to partner with others who are working on scalable container formats and provide a common method of discovering & defining the metadata in them. Suggestions and collaborations are most welcome.<br class="">
</span><span class=""><br class="">
</span><span class="">Best,<br class="">
</span><span class="">Sebastien<br class="">
</span><span class=""><br class="">
</span><span class=""><br class="">
</span><span class="">[1] <a href="https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/file-structure.html" class="">https://www.openmicroscopy.org/site/support/ome-model/ome-tiff/file-structure.html</a></span>
<div class=""><span class="">[2] <a href="https://www.openmicroscopy.org/community/viewtopic.php?f=13&t=8084&start=10#p17530" class="">https://www.openmicroscopy.org/community/viewtopic.php?f=13&t=8084&start=10#p17530</a></span></div>
<div class=""><span class="">[3] <a href="http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2009-February/001158.html" class="">http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2009-February/001158.html</a></span></div>
<div class=""><span class="">[4] <a href="http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2012-May/thread.html#2220" class="">http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2012-May/thread.html#2220</a></span></div>
<div class=""><span class=""><br class="">
</span>
<div>
<blockquote type="cite" class="">
<div class="">On 13 Oct 2016, at 13:46, Christian Carsten Sachs <<a href="mailto:c.sachs@fz-juelich.de" class="">c.sachs@fz-juelich.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Hi Mario, hi Kai,<br class="">
<br class="">
joining the discussion as I am also very interested in a HDF5 based<br class="">
container format.<br class="">
<br class="">
Given the data is to be assembled outside of an existing microscopy<br class="">
software, I don't think it makes much sense to 'emulate' the structure<br class="">
of a particular proprietary software's HDF format.<br class="">
As you state it, a new reader would be the way to go.<br class="">
<br class="">
There used to be some discussion about putting OME metadata in HDF5<br class="">
containers, i.e. creating OME-HDF (analogously to OME-TIFF) files; with<br class="">
discussions on the mailing list i.e. in 2009 [1] and 2012 [2].<br class="">
<br class="">
Therefore I wonder if someone from the OME team could elaborate whether<br class="">
progress has been made in this direction. I'd be very interested in<br class="">
developments in that direction.<br class="">
<br class="">
Best regards,<br class="">
Christian Sachs<br class="">
<br class="">
[1]<br class="">
<a href="http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2009-February/001152.html" class="">http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2009-February/001152.html</a><br class="">
[2]<br class="">
http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2012-May/002220.html<br class="">
<br class="">
On 10/13/2016 02:37 PM, Mario Emmenlauer wrote:<br class="">
<blockquote type="cite" class=""><br class="">
Dear Kai,<br class="">
<br class="">
On 13.10.2016 14:01, Kai Schleicher wrote:<br class="">
<blockquote type="cite" class="">One limitation for us is that whatever container format we chose must be<br class="">
*read**by the Bio-formats*.<br class="">
</blockquote>
<br class="">
This is a very valid point. I'm also only aware of CellH5 and Imaris Format.<br class="">
But I would not see this as a big limitation. HDF5 support is already in<br class="">
Bio-Formats, so adding support for a specific "formatting" of HDF5 is rather<br class="">
easy. If you choose any HDF5 based format, even if its not currently supported<br class="">
in Bio-Formats, adding support will be relatively easy. But that's just me :)<br class="">
<br class="">
But can you elaborate what your use-case is? Lets take an example, you have<br class="">
a container for a plate with 10.368 images. Do you just want to read an image<br class="">
with a known name? Or do you need a user interface to select the image from<br class="">
the container, for example in Fiji? If a user interface is needed, how<br class="">
elaborate does it need to be, to pick the images? Would a scrollable listing<br class="">
be sufficient, or should it rather be more comfortable, like a plate layout,<br class="">
or something more fancy?<br class="">
<br class="">
Cheers,<br class="">
<br class="">
Mario<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Writing the same format by the Bio-formats would be great too, but we could<br class="">
probably work around that.<br class="">
<br class="">
This limitation is only met by CellH5 if I am not mistaken. Are there<br class="">
alternatives for the Bio-formates?<br class="">
<br class="">
Thanks for your input and cheers,<br class="">
Kai<br class="">
<br class="">
<br class="">
<br class="">
On 10/13/2016 11:30 AM, Mario Emmenlauer wrote:<br class="">
<blockquote type="cite" class="">Dear Kai,<br class="">
<br class="">
I have a slightly lengthy reply because we have been doing this for a while,<br class="">
with certain pro's and cons. We have stored a number of single HCS images in<br class="">
HDF5 containers, to make them better suited for the storage and archiving.<br class="">
Our common data sizes are 384WP, 9 sites per well, 3-4 channels. Therefore we<br class="">
have a total of ~12.000 images per plate. Currently we host some 5500 plates.<br class="">
<br class="">
When we went to HDF5, we could observe a significant number of effects. First,<br class="">
depending on the block size of the file system of the storage, we use less<br class="">
storage space! That's because many storage systems use a larger block size,<br class="">
which effectively "wastes" a bit of space on every image. We could gain up to<br class="">
20% space by combining the images into one container.<br class="">
<br class="">
Then we could also significantly improve network transfer rates with the<br class="">
containers. Its mostly the storage read/write rates that go up, and we could<br class="">
improve throughput up to 3-fold! This is also great for archiving and backup.<br class="">
<br class="">
However there are also downsides. First its more cumbersome to access the<br class="">
images, because you add a layer of complexity. For us, the HDF5 support is<br class="">
built into the storage data base, so when we browse images on the web and<br class="">
download, the HDF5 is transparent. But every once in a while we need to<br class="">
access the images on the disk. In order to do that, we now use ImageJ, or<br class="">
an archive extractor, or a Matlab HDF5 reader that we implemented ourselves.<br class="">
It works, but its just good to know that the user experience is not the same.<br class="">
<br class="">
In my humble opinion the benefits outweigh the downsides. And I'd recommend<br class="">
an HDF5 based format, because its broadly supported, and very open! Many large<br class="">
players build on it, like Nasa, so I hope it will be supported for long. Also<br class="">
the HDF5 support in Fiji/ImageJ is quite good. However, this still leaves you<br class="">
with a problem how you'd like to "format" your HDF5 file. This is a separate<br class="">
problem! Think of it like a zip-archive: its up to you what folder layout you<br class="">
have inside the zip container, and this can still very much impact the proper-<br class="">
ties of working with the data.<br class="">
<br class="">
We chose the "H5AR" formatting (and library) from SIS, because its easy to use,<br class="">
and because they are the providers of Java HDF5 for Fiji/ImageJ. This is also<br class="">
the format used by openBIS. Its actively developed and quite mature, we did<br class="">
not encounter any problems. CellH5 would be different alternative. And there<br class="">
is the BigDataViewer HDF5 format.<br class="">
I found H5AR very easy to use, its really not much more than a container.<br class="">
The ease of use finally made the race for us. If you find this format worth-<br class="">
while, I can also provide you with Matlab and Python readers for the HDF5<br class="">
that transparently handle the container. So with the readers, you can work<br class="">
with the container as if it where a directory:<br class="">
% Example: Matlab dir() and imread() wrappers:<br class="">
vFileList = dirh5('/path/to/data.h5ar');<br class="">
vImage = imreadh5(['/path/to/data.h5ar/' vFileList(1).name]);<br class="">
<br class="">
<br class="">
Cheers,<br class="">
<br class="">
Mario<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
On 12.10.2016 16:40, Kai Schleicher wrote:<br class="">
<blockquote type="cite" class="">Hi,<br class="">
<br class="">
We are looking for a container-file format to store HTS/HCS data. This format<br class="">
needs to be read and written by Bio-formats.<br class="">
<br class="">
The reason is that our storage file system deals much better with a few large<br class="">
files than with many small files. Hence we'd expect to increase usability and<br class="">
performance drastically when handling the data.<br class="">
<br class="">
The output files of our microscope is for example *.HTD (metadata / structure)<br class="">
next to *.TIF (images and thumbnails). Here each position, channel, time-point<br class="">
and plane are saved as individual *.TIFs. This creates a lot of images for each<br class="">
multi-well plate screen.<br class="">
<br class="">
So far we have been looking into CellH5 which looks promising but development<br class="">
appears discontinued (correct me if I am wrong). Are there alternatives?<br class="">
<br class="">
Thanks for your help and cheers,<br class="">
Kai<br class="">
</blockquote>
</blockquote>
</blockquote>
<br class="">
<br class="">
<br class="">
Viele Gruesse,<br class="">
<br class="">
Mario Emmenlauer<br class="">
<br class="">
<br class="">
--<br class="">
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203<br class="">
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de<br class="">
D-81669 München http://www.biodataanalysis.de/<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
<br class="">
</blockquote>
<br class="">
<br class="">
------------------------------------------------------------------------------------------------<br class="">
------------------------------------------------------------------------------------------------<br class="">
Forschungszentrum Juelich GmbH<br class="">
52425 Juelich<br class="">
Sitz der Gesellschaft: Juelich<br class="">
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498<br class="">
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher<br class="">
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),<br class="">
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,<br class="">
Prof. Dr. Sebastian M. Schmidt<br class="">
------------------------------------------------------------------------------------------------<br class="">
------------------------------------------------------------------------------------------------<br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>