[ome-devel] Shoola-back-end consistency issues

[Harry Hochheiser]

Josh, et al

On Jul 6, 2005, at 2:04 PM, Josh Moore wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Before I respond to the previous points let me add at least one more
> suggestion to the following list.
>
> - - Messaging. In the Java world, there's JMS (also with cross-platform
> capabilities); in the ICE world, Chris can give you a shopping list of
> opportunities.
>
> - - Distributed objects: Again, Java world has Jini/JavaSpaces, RMI and
> Corba, again Chris will tell you what all ICE can do.
>
> These are both technologies that make _everything_ a bit more
> complicated, but once we've committed, we can take full advantage of
> various patterns that we haven't add available to us yet. Basically
> close-to-real time systems. Just a thought.
>

Is this one option or two? And ICE and Java play nicely together, right?


> Ok. On the other points:
>
>> & >>>: Harry Hochheiser wrote:
>>> Ilya Goldberg wrote:
>
>>>> 2) Delay retrieval. This is currently done on the data manager. The
>>>> list of images in a dataset is not pre-loaded: instead, it is
>>>> retrieved when the icon for the dataset is expanded.  This works
>>>> somewhat, but is incomplete: what if images are added while the icon
>>>> is expanded?
>
> Does nothing for consistency. Just delays the eventual problem.
>

yup.

>>>> 3) Refresh: The data manager handles this by providing  "refresh"
>>>> buttons.  These buttons work somewhat, but they are less than ideal.
>>>> A  naive refresh operation that simply repeats a request will be
>>>> painfully inefficient when one image is added to a dataset of 10K
>>>> images, but anything more sophisticated will be tricky (more on this
>>>> in a bit).
>
> We can generalize the possible solutions: Notification or polling.
> That's it. Have a lot of different names, but either you ask the server
> (whether the "agent" or the "user" doesn't matter) or the server tells 
> you.
>
sure.

> The solutions above are notification. Everything else mentioned is
> polling. Questions we should ask: what's most implementable and
> manageable? What's most performant? What's most safe in terms of
> consistency?
>
right
> We can actually define levels of performance and consistency that we
> want to achieve and see if we can implement the balance of the two in a
> reasonable time period. But we will have to do one of these, and I 
> don't
> think there's an agent that can be "stateless" enough to not need these
> functionalities.
>

yes, I think this is right.

>>>> 6) We might add version/timestamp info to each row in tables like
>>>> datasets, projects, etc. This is too painful a thought for me to
>>>> contemplate at the moment.
>
> As mentioned in http://bugs.openmicroscopy.org.uk/show_bug.cgi?id=510
> this is the "offline locking" pattern, as Andrea would surely have
> informed us. It's built into the functionality of Hibernate so I would
> HIGHLY recommend some field on each table that is "version"-like. It
> seems MEX would be a good candidate, but it will need some changes to
> the definition of MEX, I think.
>

OK

> For example:
>
>> Could we change code for modifying datasets, projects, etc. to
>> generate a MEX for each action? Would this do the trick?
>> more discussion of these alternatives may be necessary.
>
> which I would prefer over database triggers. I know there's a place for
> them, Ilya, but I have reservations and would prefer to keep the logic
> centralized.
>

Agreed.

> Ilya mentions:
>>> specific problem:  A way to check the container objects for things
>>> about them that can be modified without a MEX.  Adding a new 
>>> attribute
>>> to a dataset, for example, would mark it as "modified" just like
>>> adding a new image to it would.  Or, we could add a separate last MEX
>>> timestamp (though that would be redundant).
>
> Can we stretch our idea of MEX to make this clean? This returns to my
> comment about the mex_execution table being an "AuditLog", Ilya. If
> things not strictly done by the AE could also be included in the
> AuditLog, we're in business.
>

I'm not sure what you mean by this. In what way is MEX not clean?


>>>> 7) Another painful idea would involve building some data managers on
>>>> the client and back-end that would manage state. Perhaps by passing 
>>>> a
>>>> token around for different requests, these managers could provide a
>>>> generally useful and yet reasonably performant consistency 
>>>> mechanism.
>>>> Not having thought through the design, I'm not sure how this would
>>>> work, but I'd bet it could be done reasonably cleanly.
>
> This is getting close to distributed objects. We _don't_ want to do 
> this
> ourselves. For that there's middleware.
>
oh, yeah, agreed.  Unfortunately, we've already gone half-way on 
distributed objects. Seems to me that this is the big question:  are we 
ready to go further down that road?  Personally, I'd prefer not to.

-harry