[ome-devel] Shoola-back-end consistency issues

[Josh Moore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Before I respond to the previous points let me add at least one more
suggestion to the following list.

- - Messaging. In the Java world, there's JMS (also with cross-platform
capabilities); in the ICE world, Chris can give you a shopping list of
opportunities.

- - Distributed objects: Again, Java world has Jini/JavaSpaces, RMI and
Corba, again Chris will tell you what all ICE can do.

These are both technologies that make _everything_ a bit more
complicated, but once we've committed, we can take full advantage of
various patterns that we haven't add available to us yet. Basically
close-to-real time systems. Just a thought.

Ok. On the other points:

> & >>>: Harry Hochheiser wrote:
>> Ilya Goldberg wrote:

>>> 2) Delay retrieval. This is currently done on the data manager. The
>>> list of images in a dataset is not pre-loaded: instead, it is
>>> retrieved when the icon for the dataset is expanded.  This works
>>> somewhat, but is incomplete: what if images are added while the icon
>>> is expanded?

Does nothing for consistency. Just delays the eventual problem.

>>> 3) Refresh: The data manager handles this by providing  "refresh"
>>> buttons.  These buttons work somewhat, but they are less than ideal.
>>> A  naive refresh operation that simply repeats a request will be
>>> painfully inefficient when one image is added to a dataset of 10K
>>> images, but anything more sophisticated will be tricky (more on this
>>> in a bit).

We can generalize the possible solutions: Notification or polling.
That's it. Have a lot of different names, but either you ask the server
(whether the "agent" or the "user" doesn't matter) or the server tells you.

The solutions above are notification. Everything else mentioned is
polling. Questions we should ask: what's most implementable and
manageable? What's most performant? What's most safe in terms of
consistency?

We can actually define levels of performance and consistency that we
want to achieve and see if we can implement the balance of the two in a
reasonable time period. But we will have to do one of these, and I don't
think there's an agent that can be "stateless" enough to not need these
functionalities.

>>> 6) We might add version/timestamp info to each row in tables like
>>> datasets, projects, etc. This is too painful a thought for me to
>>> contemplate at the moment.

As mentioned in http://bugs.openmicroscopy.org.uk/show_bug.cgi?id=510
this is the "offline locking" pattern, as Andrea would surely have
informed us. It's built into the functionality of Hibernate so I would
HIGHLY recommend some field on each table that is "version"-like. It
seems MEX would be a good candidate, but it will need some changes to
the definition of MEX, I think.

For example:

> Could we change code for modifying datasets, projects, etc. to
> generate a MEX for each action? Would this do the trick?
> more discussion of these alternatives may be necessary.

which I would prefer over database triggers. I know there's a place for
them, Ilya, but I have reservations and would prefer to keep the logic
centralized.

Ilya mentions:
>> specific problem:  A way to check the container objects for things
>> about them that can be modified without a MEX.  Adding a new attribute
>> to a dataset, for example, would mark it as "modified" just like
>> adding a new image to it would.  Or, we could add a separate last MEX
>> timestamp (though that would be redundant).

Can we stretch our idea of MEX to make this clean? This returns to my
comment about the mex_execution table being an "AuditLog", Ilya. If
things not strictly done by the AE could also be included in the
AuditLog, we're in business.

>>> 7) Another painful idea would involve building some data managers on
>>> the client and back-end that would manage state. Perhaps by passing a
>>> token around for different requests, these managers could provide a
>>> generally useful and yet reasonably performant consistency mechanism.
>>> Not having thought through the design, I'm not sure how this would
>>> work, but I'd bet it could be done reasonably cleanly.

This is getting close to distributed objects. We _don't_ want to do this
ourselves. For that there's middleware.

More to come (I'm sure)
 Josh.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCzB0VIwpkR5bKmAsRAqHuAKC+JOTXeteOgbwh267JmXhkmvDrvgCgstoi
0a345IP+o3L/VqfN64OvLTs=
=kBsc
-----END PGP SIGNATURE-----