[ome-devel] Synchronize Omero with an external DB

[Josh Moore]

On Sep 26, 2012, at 11:22 AM, Luca Lianas wrote:

> Hi everyone,

Ciao Luca,

> I'm working on a new feature for our OMERO.biobank platform:
> a graph-oriented database developed using Bulbs (http://bulbflow.com) with
> a Neo4j (http://neo4j.org) backend.
> This tool addresses our need to quickly retrieve objects connected by
> chains of actions --- every object in OMERO.biobank can be generated by
> performing an action on another object, and since connections are dynamic
> and unpredictable we can't use a static workflow to represent them.
> 
> Our previous solution --- an in-memory graph built by querying all
> connected objects at runtime --- rapidly reached its scalability limits as
> we started handling something like 500K objects. Preliminary tests of the
> new implementation show a speedup of 10, with greatly reduced memory
> consumption.
> 
> The main issue at this point is keeping the Neo4j DB and
> OMERO synchronized.  Our libraries propagate object creations, deletions
> and updates (only when connections between objects change) to the Neo4j
> server, but this is not enough. Is there a way to obtain something that can
> be used as synchronization token from the OMERO server?
> 
> I'd like to be able to do something like the following:
> 
> conn = connect_to_omero(host, user, passwd)
> token = conn.get_token()
> neo4j = connect_to_neo4j(host)
> if neo4j.check_token(token):
>  go_on()
> else:
>  raise GraphSyncError('server out of sync')

This is how both the PixelData [1] and the Indexer [2] process work. They check if they've already handled a particular EventLog and then go about their business.

Cheers,
~Josh.

[1] https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/server/src/ome/services/pixeldata/PersistentEventLogLoader.java
[2] https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/server/src/ome/services/fulltext/PersistentEventLogLoader.java


> Thanks,
> 
> Luca
> 
> -- 
> Luca Lianas
> Data Fusion - Healthcare Flows
> CRS4 - Centro di Ricerche, Sviluppo e Studi Superiori della Sardegna
> POLARIS, Edificio 1, 09010 PULA (CA - Italy)
> 
> 
> Phone: 070/9250266
> Email: luca.lianas at crs4.it