[ome-users] HCS File Format / HDF5

[Manuel Stritt]

Dear Mario,

thanks a lot for the detailed explanation with concrete measurements, very
valuable!

So did you / someone try CellH5 or did you use your own HDF5 structure ?

Any plans and/or recommendations from OME side for bioformats/Omero
integration?

Regards,
Manuel


> Dear Manuel,
>
> On 10.10.2018 08:46, Manuel Stritt wrote:
> > Dear all,
> >
> > we're currently rethinking our high content screening workflow and thus
> I'm thinking about a good way to store all data.
> > My idea is to create one e.g. HDF5 file per plate which contains all the
> images + meta data.
> > Do  have any recommendations regarding that (I think Mario once
> triggered a discussion around that topic) ?
> >
> > If some kind of HDF5 is considered as solution - then still a structure
> specification would be needed.
> > Kai and Nico mentioned the CellH5 format, a flavor of HDF5.
> > As far as I can see this is supported by bioformats / Omero. However,
> it's still unclear how
> > to pack the output of a e.g. Opera machine into a CellH5 format in a
> convenient way.
> >
> > In addition there was a discussion about an official OME-HDF5 format,
> right?
> > What's the current status for that?
>
> In our testing, the containers provide a big benefit for the file
> system and storage back end, especially when a large NAS storage is
> used. For a typical desktop application with a local spinning disk,
> there was virtually no difference in speed or file system overhead.
>
> On a NAS, we got a 30% eduction in storage space due to reduced chunk
> size overhead. This is a tunable parameter, so your mileage may vary!
> It may be anything between 0% and up to 50% (or more) reduction,
> depending on your image file size and the file system chunk size.
>
> Furthermore we got significantly faster data transfer rates because
> the data is more consecutive. An rsync transfer of a full plate from/to
> the NAS was about twice as fast when transferring the container than
> the individual files.
>
> Last not least the container can apply transparent compression without
> modifying the original file, so you can (transparently) apply bzip2 or
> other compression schemes on the raw file, while keeping the full
> (proprietary) file intact and unchanged.
>
>
> All this comes also at a price. There where situations where we would
> have liked to access images and it was not as easy as we hoped. Its
> good if the database supports download of individual files, to make
> the image access transparent for end users. But when the database is
> down, image access becomes quite hard for end users. Furthermore, there
> is a unlucky number of ~100-1000 files where access is always a hazzle,
> because manual download becomes too cumbersome and container extraction
> is typically not super comfortable.
>
> Long story short: containers are great, but I would really love to
> see them in combination with a simple, graphical, cross-platform file
> management utility that allows adding and extracting files. There are
> some such utilities like HDF5View [1] but they are not yet comparable
> to something like WinZip/WinRAR/7Zip/...
>
> [1] https://support.hdfgroup.org/HDF5/Tutor/hdfview.html
>
> Viele Gruesse,
>
>     Mario Emmenlauer
>
>
> --
> BioDataAnalysis GmbH, Mario Emmenlauer      Tel. Buero: +49-89-74677203
> Balanstr. 43                   mailto: memmenlauer * biodataanalysis.de
> D-81669 München                          http://www.biodataanalysis.de/
>

-- 


The information of this email and in any file transmitted with it is 
strictly confidential and may be legally privileged.
It is intended solely 
for the addressee. If you are not the intended recipient, any copying, 
distribution or any other use of this email is prohibited and may be 
unlawful. In such case, you should please notify the sender immediately and 
destroy this email.
The content of this email is not legally binding unless 
confirmed by letter.
Any views expressed in this message are those of the 
individual sender, except where the message states otherwise and the sender 
is authorized to state them to be the views of the sender's company.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181011/5c1c153a/attachment.html>