[ome-devel] Client Architecture: A Proposal

[Ilya Goldberg]

On Aug 26, 2004, at 3:53 AM, Joshua Moore wrote:

> Check. I think this is what I needed to hear or "Get bent, Josh." I 
> don't think I can match Doug's rate, but I'll get to work. If anyone 
> is interested, get in touch.

Well, I'll have another go at you then.
The sign over your door says "Gridcomputing for the Lifesciences", so 
why all this talk about clients?
Here's something meatier to think about:
The OME analysis engine is built to execute analysis chains - a DAG of 
analysis modules.  The nodes in the DAG are the analysis modules.  The 
edges connect outputs to inputs.  Each analysis module is effectively 
an XML wrapper around some code.  The code can be a command-line 
executable, a matlab module, a perl class, etc.  Each kind of code has 
its own "AnalysisHandler", so there's one for Matlab, one for 
command-line executables, etc.  Between the XML wrapper and the 
implementation-specific handler, the analysis engine can execute an 
analysis implemented any which way you want - passing information into 
the module from the DB and collecting it from the module to put back 
into the DB, and pass it on down the edge to the next node.

An almost magical aspect of the Analysis Engine is that it knows all of 
the data dependency.  It can generate lists of tasks that can execute 
concurrently because it knows they are independent.  It then knows 
enough to wait for the results, then generate a new list of tasks, etc. 
  Until its done executing the chain.  So the key thing here is that 
right now it just executes them all on the same box.  In fact a very 
silly implementation of this knowledge of dependency can be turned on 
and off with a flag, which will cause the AE to fork as many times as 
necessary to execute all independent tasks concurrently.  What would be 
really cool is break this linkage somewhere between the analysis engine 
and the various handlers so that all it has to do is issue commands 
into some computational black box and wait for the black box to return 
the results.  None of us know enough about grid computing to think 
intelligently about where it would be appropriate to make this break, 
what kind of information we send across it, and how we send it.  It 
could be that the black box should talk to the DB directly (using our 
Perl APIs to do so - or not), or it could be more appropriate to send 
the inputs and outputs back and forth and have the analysis engine talk 
to the DB.  It could be that the entire analysis engine should live in 
the black box.  Or all this might sound completely cookoo because my 
knowledge of grid computing can be summarized on the back of a postage 
stamp with a felt-tip marker.  However its done, it would be very nice 
to preserve this agnosticism about the implementation of the module - 
so that we could still write modules in our favorite way.  Of course, 
since determining data dependency is probably the most difficult task 
in figuring out the granularity of the parallelism, the fact that this 
is already done should obviously be taken advantage of.

What do you think of them apples?
-Ilya


>
> -Josh.
>
> Ilya Goldberg wrote:
>
>> So here's a counter proposal.  You seem to know about Eclipse, so why 
>> not try building a plug-in for it to make an initial connection to 
>> OME and pull some of its primary objects out (Projects, Datasets, 
>> Images, etc).  If you can do it easily, and it provides missing 
>> functionality, then maybe everyone will join the band-wagon.  I think 
>> its easier to think about these things and evaluate them once there's 
>> some kind of proof of concept to look at - your vision made concrete. 
>>  Its not very difficult to build this initial connection - Doug did 
>> it once using the OS X development tools in an afternoon.  We decided 
>> against pursuing it because of the limited audience, but it 
>> illustrates that with the right tools something surprisingly 
>> functional can be built very quickly.
>>
>> -Ilya
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>