<div dir="ltr">Dear
Sebastien & OME Team,<div><br></div><div>thanks for the great summary of all this info and pointers.</div><div>Based on that we'll have some internal discussions and see how we will handle that - maybe </div><div>we can find a way to contribute <a href="http://s.th">s.th</a>. an push things forward.</div><div><br></div><div>The main question seems to be how to get the output of (all!) HCS readers into a standardized</div><div>container format.</div><div><br></div><div>Mario mentioned speed and disk-space. In addition to that I have in mind that we might</div><div>do some computation in the cloud and therefore transferring some data on-demand into the cloud.</div><div>Having all data of one plate in one file would be very handy for that and eliminate latency problems.</div><div><br></div><div>I'll keep you updated...</div><div><br></div><div>Regards,</div><div>Manuel</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 17, 2018 at 10:22 AM Sebastien Besson (Staff) <<a href="mailto:s.besson@dundee.ac.uk">s.besson@dundee.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Manuel and Mario,<br>
<br>
>From the OME side, here are a couple of pointers on open HCS/HDF5-based formats and their compatibility.<br>
<br>
The open CellH5 format developed by EMBL [1] is currently supported by Bio-Formats [2]. Data stored in this<br>
form can be read by our APIs and managed in an OMERO server. Bio-Formats also has support for writing<br>
CellH5 files as a writer which was contributed by the developers of the format alongside the reader.<br>
<br>
Re OME and HDF5, since the beginning of the year we have been modernizing our open file formats in order<br>
to offer support for modalities like multi-resolution images and add support for new performant binary data<br>
containers. This is part of a larger strategy that OME has embarked on where we explicitly work to support a<br>
range of binary containers, in order to address the issues created by the growing number of imaging modalities<br>
that generate large, complex multi-dimensional data. For example, we are currently extending our OME-TIFF<br>
specification to support pyramidal levels and will soon release open readers and writers for these formats. This<br>
work targets the rapidly emerging whole slide imaging and digital pathology community.<br>
<br>
We have also explored adding support for binary vessels like HDF5, KLB and others. Part of this work was<br>
presented during the 2018 OME Users meeting [3]. This work results from an intersection of our desire to expand<br>
support for binary vessels and the requirement to support datasets that will soon be published in IDR [4]. HDF5<br>
has several advantages (Mario has discussed these), but a full implementation requires the development of yet<br>
another file format (YAFF), something that is anathema to OME’s philosophy. There are already useful<br>
implementations of HDF5, e.g., BigDataViewer [5] that we aim to support in Bio-Formats. Regardless, writing<br>
an HDF5-based format requires substantial work. A full technical proposal [5] details the series of changes that<br>
need to happen at the model and API level to make this happen. Unlike the OME-TIFF pyramidal work, full<br>
support for an OME-HDF5 or any other new binary vessel will necessitate breaking changes across our entire<br>
software including a new OME Data Model. We will keep reporting on the progress via blog posts as well as<br>
during our next Users Meeting.<br>
<br>
Best,<br>
Sebastien, Jason & The OME Team<br>
<br>
<br>
[1] <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3673213/" rel="noreferrer" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3673213/</a><br>
[2] <a href="https://docs.openmicroscopy.org/bio-formats/5.9.2/formats/cellh5.html" rel="noreferrer" target="_blank">https://docs.openmicroscopy.org/bio-formats/5.9.2/formats/cellh5.html</a><br>
[3] <a href="https://downloads.openmicroscopy.org/presentations/2018/Users-Meeting/Workshops/NewFileFormats/BinaryVessels/" rel="noreferrer" target="_blank">https://downloads.openmicroscopy.org/presentations/2018/Users-Meeting/Workshops/NewFileFormats/BinaryVessels/</a><br>
[4] <a href="https://www.cell.com/cell/pdf/S0092-8674(18)31243-1.pdf" rel="noreferrer" target="_blank">https://www.cell.com/cell/pdf/S0092-8674(18)31243-1.pdf</a><br>
[5] <a href="https://github.com/bigdataviewer" rel="noreferrer" target="_blank">https://github.com/bigdataviewer</a><br>
[6] <a href="https://docs.google.com/document/d/1vCtIrxtlbA-SWPJVzwHOSnbfkbGKJAbijhvhvlSLMEw/edit#heading=h.r20td2bzgkgw" rel="noreferrer" target="_blank">https://docs.google.com/document/d/1vCtIrxtlbA-SWPJVzwHOSnbfkbGKJAbijhvhvlSLMEw/edit#heading=h.r20td2bzgkgw</a><br>
<br>
> On 11 Oct 2018, at 15:45, Manuel Stritt <<a href="mailto:manuel.stritt@idorsia.com" target="_blank">manuel.stritt@idorsia.com</a>> wrote:<br>
><br>
><br>
> Dear Mario,<br>
><br>
> thanks a lot for the detailed explanation with concrete measurements, very valuable!<br>
><br>
> So did you / someone try CellH5 or did you use your own HDF5 structure ?<br>
><br>
> Any plans and/or recommendations from OME side for bioformats/Omero integration?<br>
><br>
> Regards,<br>
> Manuel<br>
><br>
><br>
> Dear Manuel,<br>
><br>
> On 10.10.2018 08:46, Manuel Stritt wrote:<br>
> > Dear all,<br>
> ><br>
> > we're currently rethinking our high content screening workflow and thus I'm thinking about a good way to store all data.<br>
> > My idea is to create one e.g. HDF5 file per plate which contains all the images + meta data.<br>
> > Do have any recommendations regarding that (I think Mario once triggered a discussion around that topic) ?<br>
> ><br>
> > If some kind of HDF5 is considered as solution - then still a structure specification would be needed.<br>
> > Kai and Nico mentioned the CellH5 format, a flavor of HDF5.<br>
> > As far as I can see this is supported by bioformats / Omero. However, it's still unclear how<br>
> > to pack the output of a e.g. Opera machine into a CellH5 format in a convenient way.<br>
> ><br>
> > In addition there was a discussion about an official OME-HDF5 format, right?<br>
> > What's the current status for that?<br>
><br>
> In our testing, the containers provide a big benefit for the file<br>
> system and storage back end, especially when a large NAS storage is<br>
> used. For a typical desktop application with a local spinning disk,<br>
> there was virtually no difference in speed or file system overhead.<br>
><br>
> On a NAS, we got a 30% eduction in storage space due to reduced chunk<br>
> size overhead. This is a tunable parameter, so your mileage may vary!<br>
> It may be anything between 0% and up to 50% (or more) reduction,<br>
> depending on your image file size and the file system chunk size.<br>
><br>
> Furthermore we got significantly faster data transfer rates because<br>
> the data is more consecutive. An rsync transfer of a full plate from/to<br>
> the NAS was about twice as fast when transferring the container than<br>
> the individual files.<br>
><br>
> Last not least the container can apply transparent compression without<br>
> modifying the original file, so you can (transparently) apply bzip2 or<br>
> other compression schemes on the raw file, while keeping the full<br>
> (proprietary) file intact and unchanged.<br>
><br>
><br>
> All this comes also at a price. There where situations where we would<br>
> have liked to access images and it was not as easy as we hoped. Its<br>
> good if the database supports download of individual files, to make<br>
> the image access transparent for end users. But when the database is<br>
> down, image access becomes quite hard for end users. Furthermore, there<br>
> is a unlucky number of ~100-1000 files where access is always a hazzle,<br>
> because manual download becomes too cumbersome and container extraction<br>
> is typically not super comfortable.<br>
><br>
> Long story short: containers are great, but I would really love to<br>
> see them in combination with a simple, graphical, cross-platform file<br>
> management utility that allows adding and extracting files. There are<br>
> some such utilities like HDF5View [1] but they are not yet comparable<br>
> to something like WinZip/WinRAR/7Zip/...<br>
><br>
> [1] <a href="https://support.hdfgroup.org/HDF5/Tutor/hdfview.html" rel="noreferrer" target="_blank">https://support.hdfgroup.org/HDF5/Tutor/hdfview.html</a><br>
><br>
> Viele Gruesse,<br>
><br>
> Mario Emmenlauer<br>
><br>
><br>
> --<br>
> BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203<br>
> Balanstr. 43 mailto: memmenlauer * <a href="http://biodataanalysis.de" rel="noreferrer" target="_blank">biodataanalysis.de</a><br>
> D-81669 München <a href="http://www.biodataanalysis.de/" rel="noreferrer" target="_blank">http://www.biodataanalysis.de/</a><br>
><br>
><br>
><br>
><br>
><br>
> The information of this email and in any file transmitted with it is strictly confidential and may be legally privileged.<br>
> It is intended solely for the addressee. If you are not the intended recipient, any copying, distribution or any other use of this email is prohibited and may be unlawful. In such case, you should please notify the sender immediately and destroy this email.<br>
> The content of this email is not legally binding unless confirmed by letter.<br>
> Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of the sender's company.<br>
> _______________________________________________<br>
> ome-users mailing list<br>
> <a href="mailto:ome-users@lists.openmicroscopy.org.uk" target="_blank">ome-users@lists.openmicroscopy.org.uk</a><br>
> <a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" rel="noreferrer" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br>
<br>
<br>
The University of Dundee is a registered Scottish Charity, No: SC015096<br>
_______________________________________________<br>
ome-users mailing list<br>
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" target="_blank">ome-users@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" rel="noreferrer" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br>
</blockquote></div><br clear="all"><div><br></div><br></div>
<br>
<div><br></div><div><div><font size="2">The information of this email and in any file transmitted with it is strictly confidential and may be legally privileged.</font></div><div><font size="2">It is intended solely for the addressee. If you are not the intended recipient, any copying, distribution or any other use of this email is prohibited and may be unlawful. In such case, you should please notify the sender immediately and destroy this email.</font></div><div><font size="2">The content of this email is not legally binding unless confirmed by letter.</font></div><div><font size="2">Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of the sender's company.</font></div></div>