[ome-devel] Shoola-back-end consistency issues

[Ilya Goldberg]

On Jul 6, 2005, at 2:04 PM, Josh Moore wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Before I respond to the previous points let me add at least one more
> suggestion to the following list.
>
> - - Messaging. In the Java world, there's JMS (also with cross-platform
> capabilities); in the ICE world, Chris can give you a shopping list of
> opportunities.
>
> - - Distributed objects: Again, Java world has Jini/JavaSpaces, RMI and
> Corba, again Chris will tell you what all ICE can do.
>
> These are both technologies that make _everything_ a bit more
> complicated, but once we've committed, we can take full advantage of
> various patterns that we haven't add available to us yet. Basically
> close-to-real time systems. Just a thought.

Great, and when client X modifies the DB using VBA/.NET, which magical 
messaging and distributed objects pattern will ensure that client Y 
using ICE will update itself to be consistent?
Presumably these modifications would end up being processed by some 
back-end aggregator that will then go and issue messages to any 
connected clients to update themselves.  This aggregator will have to 
perform translation services for any and all interfaces used by 
clients.  Will it go and update the Web UI the user's got open also?

What about clients that connect via JDBC/ODBC?

I don't think there are any magic bullets here.  If we want a 
monolithic system where we control all of the ins and outs explicitly, 
this all becomes fairly simple using whatever we pull off a shelf.  If 
we want a system that's maximally open with many ways in and out (the 
"O" in OME), then this becomes a more difficult (not to say 
interesting) problem.  When deciding on these interfaces, we should be 
thinking "Who and what can we afford to exclude from access into this 
system?".  Its instructive to look at a long list of real-world systems 
that we need to talk to - not hypothetically sometime in the future, 
but like yesterday.  None of these things use ICE or Java or anything 
even remotely that intelligent.  Some can do web-services or at least 
pretend to.  Its really these systems that we need to concentrate on 
talking to.  As far as clients that we develop, we have pretty much 
free reign as long as we don't close anything off in the process.


>
> Ok. On the other points:
>
>> & >>>: Harry Hochheiser wrote:
>>> Ilya Goldberg wrote:
>
> We can generalize the possible solutions: Notification or polling.
> That's it. Have a lot of different names, but either you ask the server
> (whether the "agent" or the "user" doesn't matter) or the server tells 
> you.
>
> The solutions above are notification.

They're actually polling, aren't they?  The server doesn't notify the 
client that its time to refresh.  The client just does a refresh at 
some point, and either it gets new data or it doesn't (i.e. polling).

> Everything else mentioned is
> polling. Questions we should ask: what's most implementable and
> manageable? What's most performant? What's most safe in terms of
> consistency?

When we've asked these questions before, we've pretty consistently 
decided in favor of polling.  If you look around the web, that's pretty 
much what everyone else decided as well.  Various forms of "push" have 
sprouted up before and quickly died.  RSS readers are all universally 
polling.  Some of the news web-sites I visit do timed refresh, but 
that's just polling again.  Real notification over a stateless protocol 
(like HTTP) would require the client to implement a server so that the 
"real" server could issue requests on it, saying "Hey, time to 
update!".  There are a great many issues with a server initiating 
requests to a client machine.

>
> We can actually define levels of performance and consistency that we
> want to achieve and see if we can implement the balance of the two in a
> reasonable time period. But we will have to do one of these, and I 
> don't
> think there's an agent that can be "stateless" enough to not need these
> functionalities.

I agree with that.  Pixels on the screen that are left on are a form of 
state, so there's no such thing as a stateless client - only those that 
deny their statefulness.

>
>>>> 6) We might add version/timestamp info to each row in tables like
>>>> datasets, projects, etc. This is too painful a thought for me to
>>>> contemplate at the moment.
>
> As mentioned in http://bugs.openmicroscopy.org.uk/show_bug.cgi?id=510
> this is the "offline locking" pattern, as Andrea would surely have
> informed us. It's built into the functionality of Hibernate so I would
> HIGHLY recommend some field on each table that is "version"-like. It
> seems MEX would be a good candidate, but it will need some changes to
> the definition of MEX, I think.

Versioning introduces a whole host of new side-effects.  In effect, all 
data in OME is already versioned because its attached to a MEX.  The 
self-consistent solution here is to enforce that all dataset changes 
(like adding or removing images) result in a MEX.  I don't think there 
are any major side-effect issues with doing that.

>
> For example:
>
>> Could we change code for modifying datasets, projects, etc. to
>> generate a MEX for each action? Would this do the trick?
>> more discussion of these alternatives may be necessary.
>
> which I would prefer over database triggers. I know there's a place for
> them, Ilya, but I have reservations and would prefer to keep the logic
> centralized.

I don't like ad-hoq solutions either unless its the only way to avoid 
major side-effects.  I would prefer doing this with MEXes.  It would be 
slightly harder to implement because it would likely introduce some 
minor side-effects, but I don't think it would be too bad.  The 
pay-back for doing this is that it closes the penultimate loophole for 
escaping the wrath of the MEX.  The last loophole is creation of the 
container objects themselves, which is also not done with a MEX.  Once 
those two are closed, everything can become a semantic type.


>
> Ilya mentions:
>>> specific problem:  A way to check the container objects for things
>>> about them that can be modified without a MEX.  Adding a new 
>>> attribute
>>> to a dataset, for example, would mark it as "modified" just like
>>> adding a new image to it would.  Or, we could add a separate last MEX
>>> timestamp (though that would be redundant).
>
> Can we stretch our idea of MEX to make this clean? This returns to my
> comment about the mex_execution table being an "AuditLog", Ilya. If
> things not strictly done by the AE could also be included in the
> AuditLog, we're in business.

There's plenty of that going on.  Annotations aren't done with the AE.  
Import is done partially with the AE and partially not.  The only 
requirement is that the state of the system after this is done is 
compatible with the AE (otherwise the AE won't be able to use this 
info).

I don't think its escaped anyone's attention that the MEX table is an 
audit log.  Its been called many other names too.  Some of which are 
not appropriate to mention here.


>
>>>> 7) Another painful idea would involve building some data managers on
>>>> the client and back-end that would manage state. Perhaps by passing 
>>>> a
>>>> token around for different requests, these managers could provide a
>>>> generally useful and yet reasonably performant consistency 
>>>> mechanism.
>>>> Not having thought through the design, I'm not sure how this would
>>>> work, but I'd bet it could be done reasonably cleanly.
>
> This is getting close to distributed objects. We _don't_ want to do 
> this
> ourselves. For that there's middleware.

Yeah, we definitely don't want to do anything like that ourselves.

-Ilya


>
> More to come (I'm sure)
>  Josh.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.5 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFCzB0VIwpkR5bKmAsRAqHuAKC+JOTXeteOgbwh267JmXhkmvDrvgCgstoi
> 0a345IP+o3L/VqfN64OvLTs=
> =kBsc
> -----END PGP SIGNATURE-----
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>