[ome-devel] Some OME-HDF questions

[josh.moore at gmx.de]

Hi Edwin, Daniel, & other HDF-interested,

As has been pointed out, there is a lot of good intention toward HDF
on our side, but without the resources to pursue it, we've stuck
mostly to proposals and discussions, so please don't take any of my
comments below as more than that. It's a good time for all of us to
discuss just what should be in an OME-HDF standard.

 > We are planning to create a HDF5 based file format as a replacement
 > for ICS/IDS, which is the current default format in our Huygens
 > software. We need to start storing parameters and data related to
 > spherical aberration together with the images, and we do not think
 > it is a good idea to extend ICS/IDS or the current OME for this
 > purpose.
 > 
 > As we would like our HDF format to be compatible with (the future) 
 > OME-HDF, I've some questions to the OME developers:
 > 
 >  - How are we going to store multi-dimensional data;
 >    * as a native HDF multi-dimensional array (in a single dataset, like 
 > OME-XML), or;
 >    * as lower dimensional datasets within the HDF archive (like the 2D 
 > image planes in OME-TIFF)?

To make full use of HDF's capabilities, I think a single
multi-dimensional array should be supported. There is, however, also
utility in providing a hierarchical structure where individual leaves
could be compressed, signed, etc. One question is do we attempt to
support both or is one just a performance optimization of the other.

 >  - How are we going to store the meta-data;
 >    * as an HDF-clone of the OME-XML standard (using native HDF elements 
 > and attributes; no namespaces), or;
 >    * as a complete OME-XML meta-data block describing the dataset (like 
 > OME-TIFF)?

My personal preference is to make accessing the metadata as HDF5
"native" as possible. Again, this doesn't proclude also storing the
XML, but using the full HDF type system certainly has its
benefits. Nevertheless, there should probably be some agreement on
top-level paths in the file, which are more or less just name spaces.

 > To gain some experience with HDF, we'll do a 'general HDF5 reader' with 
 > limited functionality in the coming month.

It would interesting to here what what requirements you find from that.

 > Regarding the OME files that we export; these have following custom 
 > attributes in ome:Image:
 >     CA:CustomAttributes
 >           svi:RefrIndexMed
 >           svi:PinholeSpacing
 >           svi:ChannelData
 >                 LogicalChannel
 >                 MicroscopeSpec
 >                 ExPhotonCnt

Perhaps you could tell us more about these custom types. Seems like
some of them could fit into the existing model, and if not, where you
had problems might help us improve the model.

 > 
 > Best regards,
 > Edwin Bennink


 > Thank you for your reply. The core functionality of our software (
 > http://www.olympus.co.uk/microscopy/22_scan_R.htm)  is to detect, analyse 
 > and classify particles and generate statistics about found populations. 
 > The classification is done through a cytometry like interface with several 
 > histograms, displaying the particles in arbitrary feature spaces. 
 > Currently we are working on extending the particle capacity to 50m+ 
 > particles, resulting in about 25gb of uncompressed particle feature data. 
 > The current data storrage formate is a proprietary, binary one. To 
 > facilitate exchangeablity and data access we would like to move to a 
 > standart formate that suports a structure like this:
 >   Project/Experiment      
 >                         /Sample/Particledata
 >                         /Settings
 >                         /Result
 > 
 > Although I did not yet familiarise myself in depth with the OME XML data 
 > model, it seems to me it could support such a structure. If so we would be 
 > very interested in a hdf implementation of the ome xml data model.

Daniel, the organizational structure and I assume the instrument
settings would fit into the standard OME model. What do your results
look like? You mention that the particle data are collections of
histograms, correct?

 > best regards,
 > Daniel

Cheers,
~Josh.