[ome-devel] Shoola-back-end consistency issues

[Josh Moore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ilya Goldberg wrote:
> On Jul 6, 2005, at 4:23 PM, Josh Moore wrote:
>>>>> Ilya Goldberg wrote:

>> I thought you were all for representing things in the DB as much as
>> possible.  I certainly am.

As we've discussed, I'm _not_ for the DB being the API; I'm for the DB
being the model. The entire definition of what our objects are should
exist in a single language. Whether OWL or SQL Schema or if it must be
XML Schema I don't care, I'll generate the others with an isomorphism.
But the _logic_ for working with that model, the API, I want somewhere
else. Not DB triggers, and not spread over multiple languages.


>> I know its poison to your ears, but interoperability is achieved in the
>> Enterprise using "real" programming languages like VBA, and tools like
>> Excel, and SpotFire, and such things.

Sorry, a misunderstanding. I have nothing against Excel or Spitfire as
clients, nor VBA as a language. I meant I don't want to be doing this
stuff with procedural languages in the DB. The Enterprise is about
making all kinds of legacy things work together. I'm for that; I just
want lots of tools to do it for me!

By the way, probably the easiest/simplest thing to do for excel is have
a service spit out csv. One sheet of your spreadsheet gets updated and
the rest just "refreshes". I know its not an ODBC/JDBC connector or
anything, but, well, we know what I think about that.


>> I am not advocating being everything to everyone.  I am simply
>> advocating that we don't close off interfaces we already have.  Anything
>> else is wide open.  Go nuts, man!

Sure. A decision to be made though, assuming we want to centralize
logic, is what technology will get us this. I.e. we've added interfacing
to our list of requirements for a possible server architecture, along
with performance, maintainability, (what else?). The question then is
what to  use for those requirements.


>> Right, I was talking about Harry's 1,2,3, but you were talking about the
>> Java stuff.  Sorry.

Check.



[Now to the nuts and bolts stuff]

> What does the "in effect" modifier in the above mean?
> 
>> Its versioning without a serial version number.  If you're satisfied
>> with a MEX ID as a version number, then its versioning.

Think my question was more regarding the connections. Is _all_ data
currently tied to a MEX? And not just some MEX through the graph, but
truly a MEX which represents the creation (i.e. INSERT) or update of a
row in the database? Let's look at the example of the image_dataset_map.

> What are the side-effects you're talking about?
> 
>> There are just some edge-cases to work through.  Previously the
>> assumption was that a given dataset has a single 'version' of the images
>> it contains.  Once somebody had something to say about the dataset (i.e.
>> it acquired an 'attribute', and hence a MEX with that dataset as a
>> target), the dataset was considered locked, and its image content could
>> not be changed.
> 
>> If images are mapped to datasets using an object tied to a MEX, it means
>> that the collection of images in a dataset is 'versioned'.  This means
>> that we don't ever have to lock datasets, but now we must identify the
>> 'version' when we apply an analysis chain to it (or look at it in a
>> viewer).

I think was a lot of the issues that Josiah tried to inform us about in
Baltimore. What's the status with all that, J? Having more information
in our maps seems beneficial and cleaner at a first glance, but does it
get you all the requirements you have for the feds?

Where all would such new MEXES need to be added to make this work? We
listed the maps. Are there any use cases where one MEX on a row would
not be sufficient, i.e. two "MODULES" (we'll get to this next) could be
seen to change a row within one action/execution/transsaction? This
would be nasty. Would we need a "timestamp/version_module_execution_map"?


>> You don't have to think of a module as an algorithm either.  If you
>> think of it as a construction that has inputs and outputs, one that can
>> be "executed" to produce its outputs, then anything you do can be
>> considered a module execution.  Please, lets not call it an
>> instantiation or something.

Alright. What are the constraints on a "module execution" to keep the DB
in a consistent statement for the AE? If I can say (within a single
transaction):

 d.addImage(i);
 m = server.createModuleExecution();
 server.update(d,m);

Super. I'm pretty sure all that needs to be hidden from the clients.
They say:

 d.addImage(i).save();

I still think defining "modules" like this can alienate users. Perhaps
that's not an issue. Makes sense when you're building an analysis chain,
but when a user starts reading about his/her modules from hitting a save
button, gets kind of esoteric.

>> Uh, no.  They do get mexes, just not from the AE.  The MEX column in
>> each attribute table now has a NOT NULL constraint, remember?

For the things with MEXES. We still have to make everything a ST before
this holds.


>> This is done with a bit of API that's targeted at import or annotation,
>> etc.  This API registers that a certain module ("Import", "Annotation",
>> etc.) executed and produced the recorded result.  Parts of the AE are
>> indeed used to accomplish this, but it has to do with instantiating STs
>> rather than executing an analysis chain, so its not done via the AE. 
>> The AE is used to do only one thing:  Execute an analysis chain against
>> a dataset.
>> In principle, the AE can be used for things like this, but it was easier
>> to do them through a specialized API rather than shoehorning this into
>> the AE interface.

This is fine. This is exactly the kind of thing I was talking about way
up top. An API sitting in a server doing the stuff its supposed to do. I
think very much that the AE, whether its code, its process, or simply
its "memes" (in terms of user documentation and other cultural goods)
need to be refactored to let the server do instantiation, etc. and have
all the logic sit in one place. (Don't mean AE logic, I mean the logic
to deal with our model.)


Otherwise,
  have a nice day y'all.
    Josh.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCzNvrIwpkR5bKmAsRAuvRAJ47tkAEUFvzzVoOhBAa5LCBBaWvswCfZUpF
pYULF+i1WrfYiA8NDiCbU8A=
=6Qgy
-----END PGP SIGNATURE-----