[ome-devel] OMERO.features: Development of a new API for storing image features

[Vebjorn Ljosa]

On Mon, Jul 22, 2013 at 9:23 AM, Lee Kamentsky <leek at broadinstitute.org> wrote:
>> The hierarchical nature of feature data is what made Simon choose HDF5
>> files stored on a per-image basis as the back-end in the first generation of
>> OMERO-WND-CHARM. But for datasets that consist of thousands of images/ROIs,
>> this solution might not scale well, which is why Simon was interested in a
>> NoSQL database for feature storage where the schema about what can be stored
>> is not strict.

The last sentence makes me think about Bisque's key–value store. I
think Kris Kvilekval said that the key–value store is starting to hurt
now that they want to store per-cell features, and that they are
moving in the direction of HDF5 for storing feature vectors.

To give a sense of the scale, let me describe a dataset I'm working
on. It has 70 millon cells in 916,000 images. For each cell, there is
a feature vector with 833 numbers (mostly floats, but some integers).
It's important to be able to store such datasets and to be able to
retrieve horizontal or vertical slices fairly efficiently.

Vebjorn