[ome-users] Multiple backends for OMERO

[Jake Carroll]

Hi.

I have a requirement to change some aspects of the OMERO default datastore layout.


omero.data.dir=/omero.datastore

Where that might be a set of blocks mounted into a local filesystem, an NFS mount or something like it.

OMERO provides a single path in the config file to put data “into” and within that, or underneath it puts its own directory hierarchy. This consists of the usual stuff:


drwxr-xr-x   2 omero omero    6 Jul  1  2014 DropBox

drwxrwxr-x   3 omero omero   31 Oct  2  2015 .omero

drwxr-xr-x   2 omero omero 4.0K Mar  2  2016 FullText

drwxr-xr-x   5 omero omero   85 Jul 17  2016 BioFormatsCache

drwxr-xr-x  10 omero omero 4.0K Aug  6  2016 .

drwx------ 127 omero omero 8.0K May 24 10:07 Pixels

drwx------ 174 omero omero  20K May 29 11:40 Thumbnails

drwxr-xr-x 210 omero omero 8.0K Jun 22 15:10 ManagedRepository

drwxr-xr-x 279 omero omero 8.0K Jun 24 00:00 Files

drwxr-xr-x  25 root  root  4.0K Jul 10 17:51 ..

The problem I face or want to “make better” is as follows.

I’m at a point where I need to be able to completely segregate users into different NFS mounts.

Picture a situation where each user has their own collection and their own NFS mount that belongs to them, for various data categorisation needs. This is part of a broader effort to make a data collection approach for each person.

Such that each user has their own backend store – i.e. when I login my OMERO backing store is some NFS store that is specific to me. So if I login, it might be /omero.datastore/Q0XYZ but my colleague might login and his might be /omero.datastore/Q0ABC

There is a lot of good rationale behind this which I won’t go into in this email as it is a long discussion in itself in and around the way we are managing collections, allocations and data distribution in high performance data fabrics at the many (many) petabyte-scale which we operate – but we absolutely need to change omero from living in its own little world and keeping its own hierarchy to making a more distributed approach to compartmentalise the data structures for individuals.


I.e: As it stands.

OMERO --> storage structure --> All users in here.

OMERO --> n * storage structures for n(users)

I appreciate that this might need some thought in order to make it possible. I even considered fiddling around with some symlinks in the default OMERO Datastore in order to do this – but then rationalised that there is a complex chicken-and-egg game to be played. I.e – when a user logs in for the first time, it’ll create a skeleton for the user the “wrong” way, then I’d need a series of steps and scripts to change it, symlink it and somehow shift it into one of the NFS mounts associated with that user.

Some guidance and thoughts/help would be appreciated.

Cheers.

-jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180711/9dd8edcf/attachment.html>