<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Dear Mario,</p>
<p>thanks for this elaborate and very useful answer! <br>
</p>
<p>One limitation for us is that whatever container format we chose
must be <b>read</b><b> by the Bio-formats</b>.</p>
<p>Writing the same format by the Bio-formats would be great too,
but we could probably work around that.</p>
<p>This limitation is only met by CellH5 if I am not mistaken. Are
there alternatives for the Bio-formates?<br>
</p>
<p>Thanks for your input and cheers,<br>
Kai<br>
</p>
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 10/13/2016 11:30 AM, Mario
Emmenlauer wrote:<br>
</div>
<blockquote
cite="mid:%3C1c480495-012b-e11d-a799-831fe638ea18@emmenlauer.de%3E"
type="cite">
<pre wrap="">
Dear Kai,
I have a slightly lengthy reply because we have been doing this for a while,
with certain pro's and cons. We have stored a number of single HCS images in
HDF5 containers, to make them better suited for the storage and archiving.
Our common data sizes are 384WP, 9 sites per well, 3-4 channels. Therefore we
have a total of ~12.000 images per plate. Currently we host some 5500 plates.
When we went to HDF5, we could observe a significant number of effects. First,
depending on the block size of the file system of the storage, we use less
storage space! That's because many storage systems use a larger block size,
which effectively "wastes" a bit of space on every image. We could gain up to
20% space by combining the images into one container.
Then we could also significantly improve network transfer rates with the
containers. Its mostly the storage read/write rates that go up, and we could
improve throughput up to 3-fold! This is also great for archiving and backup.
However there are also downsides. First its more cumbersome to access the
images, because you add a layer of complexity. For us, the HDF5 support is
built into the storage data base, so when we browse images on the web and
download, the HDF5 is transparent. But every once in a while we need to
access the images on the disk. In order to do that, we now use ImageJ, or
an archive extractor, or a Matlab HDF5 reader that we implemented ourselves.
It works, but its just good to know that the user experience is not the same.
In my humble opinion the benefits outweigh the downsides. And I'd recommend
an HDF5 based format, because its broadly supported, and very open! Many large
players build on it, like Nasa, so I hope it will be supported for long. Also
the HDF5 support in Fiji/ImageJ is quite good. However, this still leaves you
with a problem how you'd like to "format" your HDF5 file. This is a separate
problem! Think of it like a zip-archive: its up to you what folder layout you
have inside the zip container, and this can still very much impact the proper-
ties of working with the data.
We chose the "H5AR" formatting (and library) from SIS, because its easy to use,
and because they are the providers of Java HDF5 for Fiji/ImageJ. This is also
the format used by openBIS. Its actively developed and quite mature, we did
not encounter any problems. CellH5 would be different alternative. And there
is the BigDataViewer HDF5 format.
I found H5AR very easy to use, its really not much more than a container.
The ease of use finally made the race for us. If you find this format worth-
while, I can also provide you with Matlab and Python readers for the HDF5
that transparently handle the container. So with the readers, you can work
with the container as if it where a directory:
% Example: Matlab dir() and imread() wrappers:
vFileList = dirh5('/path/to/data.h5ar');
vImage = imreadh5(['/path/to/data.h5ar/' vFileList(1).name]);
Cheers,
Mario
On 12.10.2016 16:40, Kai Schleicher wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
We are looking for a container-file format to store HTS/HCS data. This format
needs to be read and written by Bio-formats.
The reason is that our storage file system deals much better with a few large
files than with many small files. Hence we'd expect to increase usability and
performance drastically when handling the data.
The output files of our microscope is for example *.HTD (metadata / structure)
next to *.TIF (images and thumbnails). Here each position, channel, time-point
and plane are saved as individual *.TIFs. This creates a lot of images for each
multi-well plate screen.
So far we have been looking into CellH5 which looks promising but development
appears discontinued (correct me if I am wrong). Are there alternatives?
Thanks for your help and cheers,
Kai
</pre>
</blockquote>
<pre wrap="">
--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München <a class="moz-txt-link-freetext" href="http://www.biodataanalysis.de/">http://www.biodataanalysis.de/</a>
</pre>
</blockquote>
<br>
</body>
</html>