[ome-users] OMERO with iRODS?

[Josh Moore]

Hi Harri,

this is of course the beginning of a much larger conversation that hopefully others will also participate in, but as a start I've outlined some of the technical considerations below. The one caveat is that I'm not familiar with iRODS and so it's important to hear what the community expects to get out of the combination.

On Oct 20, 2010, at 1:02 PM, Harri Jäälinoja wrote:
> Dear All,
> 
> I want to share with you some ideas. Maybe you can help me figure out if they are worth pursuing or not, or even better, help me refine them into something useful.
> 
> I am evaluating OMERO for use in our microscopy unit, at the moment specifically for a Leica HCS confocal microscope. While I have been reading about various existing systems, I have somehow developed also an interest in the iRODS data management system (http://www.irods.org). What I am wondering now is, would it be possible to use OMERO and iRODS together and what would be the best way to do it?
> 
> iRODS seems to offer proven tools for managing various storage resources, with the flexibility to formulate policies about how the data is stored and shared, sophisticated authentication systems, data integrity checking, even some workflow functionality. This is something that a data storage administrator might want to have. For the microscopist or biologist that is not very useful, they need the rich user interface provided by OMERO. Both aspects are important, and I think this is also what DAX is addressing with the OMERO archiving solution (http://www.daxarchiving.com/en/solutions/omero-archive/challenge.html).
> 
> iRODS is already being used in the OpenCCDB system (http://openccdb.org/index.shtm), and the idea of using it with OMERO has been suggested before (http://www.u.arizona.edu/~dcromey/OMX/), so I think there might be something to it. The question is, how to do it? I can think of three approaches:
> 
> 1. use an iRODS archive as an external storage to OMERO
> - when necessary, move images from the location visible via OMERO.fs to the iRODS system, with no concern about OMERO system integrity
> - I read in the notes of some OME meeting that somebody said for him it would be sufficient if OMERO simply said "Contact your sysadmin..." in case an image file is not found.
> - sounds trivial
> (- a national computing centre will probably soon offer us this service)

Does iRODS have facilities for monitoring file access times and taking actions based on them? If so, that sounds like a valuable addition, since assumably iRODS would also provide a UI to put the files back where they were. If the iRODS rule system allowed it, optimal would be to say that the file location known by OMERO is one of several replicates which gets removed when older than X weeks.

> 2. use OMERO.fs to access data in iRODS
> - iRODS stores files in some directory (default <iRODS_install/Vault>). - this directory could be made read-only visible to OMERO.fs for input into OMERO
> - enter data first in iRODS, then automatically in OMERO via OMERO.fs drop-box
> - re-organizing data in OMERO would not be possible as this would break the iRODS system
> - sounds fairly easy

From the OMERO point-of-view, this sounds very straight-forward. The only issue I can think of off-hand is that OMERO.fs would have to handle the read-only status. I've added a story for this: http://trac.openmicroscopy.org.uk/omero/ticket/3162

> 3. create an iRODS extension to OMERO
> - iRODS has a Java API (Jargon)
> - add features in OMERO for iRODS I/O, work directory synchronization etc.
> - when data is entered in OMERO, automatically also in iRODS
> - sounds nice but requires coding
> - could possibly save OMERO developers the trouble of implementing functionality related to data sharing, permissions etc.?

This is certainly a much larger undertaking, and could possibly involve making Bio-Formats iRODS-aware (just glancing at Jargon -- it does provide a RandomAccessFile interface, so the necessary modifications may be limited). Perhaps there's a way using callbacks to do the reverse of dropbox? i.e. OMERO writes out to a location, and then iRODS takes control of that location on receiving notification?

> I will soon have an environment where I can install and test OMERO and iRODS and other products, so I hope I will have some practical experience of approaches 1. and 2. If you think the general idea seems worthwhile, maybe the OMERO dev-team could check what number 3. would mean in practice and how feasible it is?

That's of course difficult for everyone involved to say. When you've got both systems running, perhaps we could start with a very simple test, like opening a file stored in iRODS via Bio-Formats and converting it to OME-TIFF in iRODS. That would let us know if there are any barriers to moving forward.

> Hoping to hear your comments!
> 
> Best regards,
> Harri

Cheers,
~Josh.