[ome-devel] Questions about initial development.

[Ilya Goldberg]

Hi Glenn
There are several mechanisms for programatically communicating with  
the OME "back end" from C.
The easiest "quick and dirty" way is to use the text table interface  
using simple GET requests.  This interface is used throughout the web  
UI when displaying results in tables.  Adding "&Format=txt" to the  
URL (the way the "Download as text" links are set up for each table)  
will return a tab-delimited table.  The search criteria are specified  
using the OME::Factory interface as described here:
http://openmicroscopy.org/APIdocs/OME/Factory.html#search_criteria
This is the mechanism used by OME-Excel (src/Excel/OME-Excel.xls) to  
do live queries to an OME server and directly populate Excel  
worksheets.  The "OME-Query" worksheet lets you specify the criteria  
just like in the documentation for OME::Factory above, and constructs  
a URL for a GET request, which it displays for you to see.  This is  
all implemented in Visual Basic (!), which is included as a module in  
the OME-Excel.xls workbook.  Since this interface is used by at least  
two heavily used clients (browsers and excel), it is expected that  
backwards-compatibility will be maintained.  You can use a similar  
simplified interface for writing to OME:
http://www.openmicroscopy.org/custom-annotations/ 
spreadsheet_importer.html

Alternatively, for tighter integration, you can use the Factory  
interface via XMLRPC (its the same call).  There's a handy  
interactive XMLRPC tester in src/perl2/OME/Tests/XMLRPC.pl (though  
its in perl).

The most common calls to OMEIS (the image server which stores the  
original files and pixels) are implemented as a C library which uses  
libcurl for http.  The code is in src/C/omeis-http/, and documented  
here:
http://www.openmicroscopy.org/api/omeis/methods.html
(look for the [C] marker next to the method for those implemented in  
the C library)
It would be trivial to add generic communication capability for OME  
(not just OMEIS) to this library using the simplified Factory/Table  
interface used in Excel/WebUI.  There is already code in this library  
for parsing tab-delimited text into a convenient C format,  
constructing GET requests and receiving the results directly into  
RAM, etc.  A contribution like this would be most welcome.

We don't have an ome-specific XMLRPC interface for C, though we have  
one for Java (checkout OME-JAVA from CVS rather than OME.  Also take  
a look at the Shoola module on CVS).  In many cases, its easiest to  
"hard-code" the specific data types, search criteria, etc that you  
use in your applications because the client's data model is likely to  
be more "fixed" than the OME data model if the client is geared to a  
specific task (your corresponding structures/classes/etc are usually  
hard-coded as well, after all).  It usually doesn't pay to implement  
the degree of flexibility available in the OME data model in the  
client.  Unless of course, that's what you want your client to do.


On Oct 5, 2006, at 9:48 AM, Glenn Pierce wrote:

> Also I saw that Omero is replacing the OME data server.
> How long before this replaces the existing OME server. Would you
> recommend development on that platform ?
> I am concerned about developing a lot of work that will be rendered
> useless soon.

I'm not sure where you saw this, but this is unfortunately likely to  
be a source of confusion for others that we really should clear up.
OMERO doesn't implement certain elements in its design that are  
critical for the proper functioning of the OME back-end.  There is no  
plan in the OMERO design to provide this functionality, and so there  
is no plan to have OMERO replace OME.  At its inception, OMERO could  
interact with the same database as OME, which provided de-facto  
interoperability and a clear transition mechanism.  Unfortunately,  
this is no longer the case because OMERO uses a somewhat similar  
though incompatible database schema.  I imagine that these two  
systems will coexist for some time to come.  Initially communication  
between them will probably be through OME XML files, but there may be  
more stream-lined options.  The source of the incompatibility is that  
OMERO's security model (row-level access control) is incompatible  
with OME's provenance model (result sets linked to a module  
execution, with security/access control at the module-execution  
level).  There is presently no plan to resolve this low-level  
incompatibility, so the fall-back for communicating between the two  
independent systems is through OME-XML files (with or without  
provenance/security/access control).  OMERO can potentially work as a  
server on other OSes (i.e. Windows), and can use RDBMS other than  
PostgreSQL.  OME can potentially do this as well, though support for  
other DBs has not been implemented, and operation on Windows would  
probably require cygwin and has never been tried (to my knowledge).   
OMERO is also is more driven towards interactive image manipulation  
and annotation.  OME on the other hand is driven more towards batch  
processing of thousands of images at a time, and uses a web-based  
interface for user interaction.  Despite this focus, it can also be  
used in more interactive one-at-a-time analysis as demonstrated in a  
recent Biotechniques paper:
http://www.openmicroscopy.org/publications/Biotechniques-FindSpots.pdf

OME has a simpler access-control/security model (similar to unix),  
but a more detailed model of data provenance and a sophisticated  
analysis subsystem (the Analysis Engine, and Distributed Analysis  
Engine - AE and DAE).  The analysis system in OME is being heavily  
used in my lab and increasingly at UCSB.  We have implemented  
analysis work-flows involving over a dozen interconnected and  
multiplexed algorithms for computing > 1200 image feature descriptors  
used for pattern recognition.  We routinely execute week-long  
computations on 10s of thousands of images using this system.

So largely, which system you use depends on what type of work you  
want to do.  Interactive one-at-a-time sort of image analysis/ 
visualization/annotation, or more batch-mode computing on large image  
sets.  Obviously there is quite a bit of overlap between the two  
systems, so you are not likely to be left out in the cold regardless  
of which one you chose to work with.

Its clear to us that the existence of the two projects presents some  
difficulty and/or confusion for potential users which they probably  
don't need, and we are working on a plan to resolve this.  For now  
though, it is unfortunately an either-or thing.

My favorite way to resolve this so far (other than just using the  
same DB) is using a "data mart/data warehouse" model, which is a well  
used concept in data-intensive industries.  That would be a natural  
fit for what the two projects' respective strengths and weaknesses  
are, not to mention the all important buzz-word compliance.  Data- 
mart/data-warehouse generally involves a translational layer to  
federate the two incompatible DB schemas b/w the mart and the  
warehouse.  In these arrangements, the database schemas are not  
directly compatible because they serve different purposes.  The mart  
DB is client-centric and user-facing, while the warehouse is  
generally designed without regard to a specific client in order to  
support many types of different clients (i.e. its data-centric).   
This is not to say that this is how things will shake out, just one  
option that's on the table.

>
> Finally we deal with a image type called ICS. When importing these
> images I wish to pass the metadata out of the images and store it. Can
> someone point me to the place in the code that performed the  
> passing of
> metadata from an image.

I think that BioFormats supports ICS.  We are working together with  
LOCI (the developers of BioFormats) for tighter integration with OME  
and OMERO, though I do think that its possible right now to convert  
ICS to OME-TIFF in BioFormats, and then import the OME-TIFF into OME  
(and presumably OMERO).  LOCI also developed OME-TIFF, BTW.  This is  
OME-XML turned inside out so that on the outside it is an innocent  
looking regular TIFF file, but on the inside (in the Description  
tag), is the full OME XML metadata goodness.  The pixels are of  
course stored in the regular TIFF way rather than being encoded in  
Base64 and stored in the XML.  The OME back-end has full support for  
reading OME-TIFF.


-Ilya


>
>
> Thanks for any help.
>
> Glenn Pierce
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>