[ome-devel] Fwd: File format for large data sets stored at multiple resolutions

[Nico Stuurman]

Dear all,

I have been running into more and more individual efforts to create new
file formats to deal with large datasets that need to be stored at
multiple resolutions to enable fast feedback to the user. Examples are
the hdf5 format used by the BigDataViewer plugin by Tobias Pietzsch and
Stephan Preibisch, the hdf5 format used by Chimera (UCSF-based package
primarily for crystallography and EM that also has amazing capabilities
for 3D visualization of light microscopy data), the Micro-Manager
SlideExplorer plugin for which Arthur Edelstein developed his own
storage system, and the Micro-Manager plugin "Magelan" that Henry
Pinkard is developing right now, and who also stores multiple resolution
versions of the data on disk. Doubtlessly, there are many more examples.

Even when conversion between these formats is possible (as long as they
are reasonably documented), conversion becomes time consuming and takes
up large amounts of disk space, simply because the data sets have become
gigantic.  The reasons why everyone designs their own formats are also
clear, there simply is no standard (at least that I am aware of, if
there is please do let me know! ) that let's one store gigantic datasets
that give fast access to the data in multiple resolutions.

Since you guys have created the standard in light microscopy with
ome.tif, I assume that you have thoughts what a new standard (hdf5
based?) should look like.  In any case, I am very much looking forward
to hearing your thoughts and I will be happy to help avoid a wild growth
of different formats that we will have to live with for years to come if
we do not take action soon.

Best,

Nico