[ome-devel] Bio-Formats Ice bindings

[Curtis Rueden]

Hi Josh,

Was your and Mario's primary goal to get away from the in-process issues of
> the JVM? Would it be an option to fork, create the JVM, and then use Ice
> rather than raw sockets for the communication? In that case would you need
> all objects? Or does the API that Mario is already using come close to
> sufficing? What are the other features/requirements that need to be added?
>

I believe the goal was to have an inter-process solution to decouple the JVM
a bit from the C++ program. However, I admit that I am fuzzy on the
specifics of why Mario wants an inter-process solution. In his talk, he
mentioned one problem with allocating memory to the JVM a priori, and how to
deal with OutOfMemoryError, but I am not clear on why an inter-process
solution would improve on that situation. Mario, any comments?

Curtis, I assume your plan is to turn everything into an Ice Prx? e.g. would
> you have an IRandomAccessPrx? If so, I'm skeptical that this would work as
> well as you want.
>

Yep, everything would be exposed as proxies. So sure, there would be an
IRandomAccessPrx, but it's not like most consumers of the API would actually
make use of it (since I agree, it would be awful performance-wise). Most
people would interact with the Java library at a less fine-grained
level—e.g., calling IFormatReader.openBytes to grab one plane at a time.
There is still overhead at that level, but much less.

I can't remember the specific problem we were having, Curtis. To you have
> that email still and/or link on ome-devel? I found only
> http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2010-October/001733.html
>

I couldn't find it on the mailing list either; perhaps we discussed it over
chat.

To understand the problem again after so long, I mocked up a Slice interface
for a small subset of the Bio-Formats API. In so doing, I found the solution
to the previous issue on this page of the Ice manual:

  http://doc.zeroc.com/display/Ice/Object+Incarnation+in+Java

To describe briefly, say we have the following Slice definition:

  interface CoreMetadata;
  interface IMetadata;
  interface IFormatReader {
    CoreMetadata* getCoreMetadata();
    IMetadata* getMetadataStore();
  };

When implementing servants on the server side, you must implement methods
that return proxies of other servants; e.g.:

    public CoreMetadataPrx getCoreMetadata(Current current);
    public IMetadataPrx getMetadataStore(Current current);

I was stumped on how to translate a server-side Java object into a proxy of
a servant that wraps that object. The solution (from the Ice page linked
above) is:

  CoreMetadataPrx proxy = CoreMetadataPrxHelper.uncheckedCast(
    current.adapter.addWithUUID(new CoreMetadataI(coreMetadata)));

(Assuming that the CoreMetadataI servant implementation has a constructor
that takes a CoreMetadata object, of course.)

Personally, I would describe that invocation as "horribly unintuitive" and
perhaps "needlessly complex" (why not "current.adapter.createProxy(new
CoreMetadataI(coreMetadata))"), but at least there is a way to do it!

I have posted my full solution at:

  http://dev.loci.wisc.edu/curtis/bf-ice/

There are shell scripts for launching the server and the client, and a small
test to verify that information is being communicated properly. All works,
so I think my proposal to autogenerate a Slice file from a Java JAR file is
doable. The only question is: when will I find the time?

Regards,
Curtis

On Tue, Jun 21, 2011 at 7:12 AM, Josh Moore <josh at glencoesoftware.com>wrote:

>
> On Jun 17, 2011, at 11:17 AM, Curtis Rueden wrote:
>
> > Hi everyone,
>
> Hi,
>
> > Yesterday Mario Emmenlauer and I discussed the Bio-Formats Ice bindings,
> and
> > how it would be nice to resurrect them so that Bio-Formats has an
> > inter-process, cross-language integration solution.
> >
> > I started looking at how best to resurrect them. The way they were being
> > done before was very ad hoc. We generated them from a Velocity template
> that
> > was quite hard-coded in many ways compared to the way the way the C++
> > bindings work. See:
> >
> >
> >
> http://git.openmicroscopy.org/?p=bioformats.git;a=blob;f=components/autogen/src/ice/bio-formats.vm;hb=HEAD
> >
> > In contrast, the C++ bindings wrap every method of every Java class in
> > bio-formats.jar, loci-common.jar and ome-xml.jar.
> >
> > We could do something similar for the Bio-Formats Ice bindings—generate a
> > Slice file mirroring the Bio-Formats API as fully as possible—but it
> would
> > require a bit of work. And of course it results in a very granular
> remoting
> > API compared to how people often use Ice. Nonetheless, it would make for
> an
> > intuitive API, as long as people exercise some care not to make too many
> > remote calls as Mario mentioned in his XuvTools talk yesterday.
> >
> > Do others agree that such a thing would be useful? If so, I am thinking
> > about how best to implement it. My initial inclination is to write some
> Java
> > code that takes a JAR file as input, and generates a Slice file with
> > interfaces mirroring each class in the JAR, as well as the Java server
> > implementation code that simply delegates to the wrapped JAR.
>
> Was your and Mario's primary goal to get away from the in-process issues of
> the JVM? Would it be an option to fork, create the JVM, and then use Ice
> rather than raw sockets for the communication? In that case would you need
> all objects? Or does the API that Mario is already using come close to
> sufficing? What are the other features/requirements that need to be added?
>
> > My questions, mostly to those who know more about Ice, are:
> >
> > 1) Is there already a tool out there for doing this? Or a better way in
> > general?
>
> None that I know of.
>
> > 2) Are there any roadblocks you can foresee with this approach, aside
> from
> > bad performance in some cases?
>
> Curtis, I assume your plan is to turn everything into an Ice Prx? e.g.
> would you have an IRandomAccessPrx? If so, I'm skeptical that this would
> work as well as you want.
>
> > One potential roadblock: years ago I had trouble with methods of one
> > interface returning objects of a different interface. At the time, Josh
> > Moore and I concluded that it was sort of infeasible to have these sorts
> of
> > methods. E.g., IFormatReader.getCoreMetadata() returns a CoreMetadata
> > object, but this is difficult to implement properly in the server-side
> > implementation due to how Ice proxies work. But my intuition is that
> there
> > may be a clever workaround...
>
> I can't remember the specific problem we were having, Curtis. To you have
> that email still and/or link on ome-devel? I found only
> http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2010-October/001733.html
>
> > Thoughts welcome!
> > -Curtis
>
> Cheers,
> ~Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20110630/eb1636c4/attachment.html>