[ome-devel] Integrating OMERO with DiGS

[josh.moore at gmx.de]

James Perry writes:
 > Hi all,

Hi James.

 > Apologies if this is not the appropriate list for asking about this.

Completely appropriate.

 > I am based at Edinburgh Parallel Computing Centre, working on the  
 > "Data Grid For Cell Biology" project. The project aims to adapt the  
 > Distributed Grid Storage (DiGS) software, which was originally  
 > developed for particle physics data, and use it for storing cell  
 > biology data. As part of this work we would like our software to  
 > integrate as well as possible with OMERO.

Sounds great. Hopefully we can be of help.

 > DiGS manages a "virtual file system" which distributes files  
 > transparently across storage at several sites, and replicates them so  
 > that there are multiple copies. It also manages a database of XML  
 > metadata describing the data files, and allows users to search this  
 > database.

 > At a minimum, we would like to use the same data formats as OMERO  
 > (OME-TIFF for image data, OME-XML for metadata), but ideally we would  
 > like a much closer integration. For example, data uploaded to OMERO by  
 > a user would automatically be replicated by DiGS and become available  
 > at another site (either through a DiGS client tool, yet to be written,  
 > or possibly even through another OMERO server). I have obtained the  
 > OMERO server and clients and have been studying them to find out how  
 > difficult this would be.

This sort of federation shouldn't be too difficult and is something
that has certainly been discussed, but building a general-purpose
federation client has not yet been scheduled.

 >  From what I have seen, I think it would be better for DiGS to go  
 > through the OMERO APIs to access image files rather than attempting to  
 > manipulate the OMERO data store directly. As I understand it, the data  
 > store is in an OMERO internal format which may change in different  
 > versions and may not be readable by other software. I was wondering if  
 > there are APIs to do things like the following:

Definitely. What programming language are you looking to do this in?
Currently, C++, Python, and Java are supported.

 >   - read images from an OMERO server (in their original format, or at  
 > least in a standard image format)

If during import, the original image was stored then you can use the
RawFileStore to retrieve the entire file:

  http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/ome/api/RawFileStore.html

However, most clients use the RawPixelsStore which returns sections of
binary data with methods for calculating size, offsets, etc:

  http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/ome/api/RawPixelsStore.html

 >   - write images to an OMERO server

The same interfaces are used, for example by the importer. See the
importImage() method at:

  https://trac.openmicroscopy.org.uk/omero/browser/trunk/components/importer/src/ome/formats/importer/ImportLibrary.java#L341

 >   - get image properties (for example, file size, upload time,
 >   - owner)

Definitely. The most low-level api is IQuery:

  http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/ome/api/IQuery.html

which provides access to the entire database. However, you may want to
get started with IPojos:

  http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/ome/api/IPojos.html

which is a higher-level interface that loads all of the necessary and
much of the otherwise useful metadata.

 >   - enumerate images

Do you mean obtaining a list of images? If so, then yes:

  iQuery.findAll(Image.class, null)

But there are various other ways to get a list of images (or any other
data type).

Note: the list returned will be filtered by the security system.

 >   - access other files (for example, documents attached to images)

RawFileStore can be used to download the attached images. See
StructuredAnnotations:

  https://trac.openmicroscopy.org.uk/omero/wiki/StructuredAnnotations

for information on how these files get attached.

 >   - access the OME-XML metadata (read/write)

I'm not exactly sure what you mean, but in general all metadata that
is imported is available through the API. Can you give a concrete example?

 > I have been looking at the OmeroApi pages on the Wiki but so far I  
 > can't tell for certain whether all these things are supported.

Everything described on OmeroApi is supported, but you may want to look at the
Javadocs for more in-depth coverage. (Even for the C++ and Python
language mappings, we use javadoc as the central API definition
currently.)
 
 > Any help or information would be greatly appreciated.
 > 
 > Thanks,
 > James Perry

Best wishes,
~Josh.