[ome-devel] [ome-nitpick] OME/ImageJ interoperability plan

Curtis Rueden ctrueden at wisc.edu
Thu Mar 21 15:30:43 GMT 2013 

Hi Josh & everyone,

> Thoughts on moving to ome-devel?

Rerouting as suggested!

For those newcomers: we are discussing how best to support execution of
ImageJ commands and scripts from an OMERO server. The idea is that you
could upload desired ImageJ (probably only ImageJ2) scripts to your OMERO
installation, where they could run server-side (or potentially delegated to
a cluster, etc.) and return their results straight back into OMERO, very
similar to how you can run Python scripts from OMERO now.

I have also forwarded my initial and revised plans originally discussed
amongst the OME team internally (see bottom). Comments, suggestions,
thoughts, etc., from the OME developer community are very welcome!

> There's a description framework:

Thanks for all the pointers, Josh. I will dig in today.

> Don't know if you've seen the script guide:

Nope, but I will read it now! Thanks!

Regards,
Curtis


On Thu, Mar 21, 2013 at 5:01 AM, Josh Moore <josh at glencoesoftware.com>wrote:

> Thoughts on moving to ome-devel?
>
> On Mar 20, 2013, at 9:49 PM, Curtis Rueden wrote:
>
> > Hi Josh, J-M, et. al,
> >
> >> If these files are uploaded to lib/scripts (like all other scripts),
> >> then they should show up in the drop-down menus, assuming there is the
> >> proper Ice params wrapper from above.
> >
> > Are you suggesting there would be one (or some constant C) Ice params
> > wrapper which would enable ImageJ support? Or that each ImageJ script
> added
> > to the server would have its own (autogenerated?) Ice params wrapper?
>
> There's a description framework:
>
>
> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/blitz/resources/omero/Scripts.ice
>
> So each script would define it's own INs and OUTs, just like the Python
> scripts would do. In fact, the clients wouldn't really even know which was
> which, other than perhaps the file endings.
>
> What's necessary to make that work is some logic which when called with a
> special property ("omero.script.parse=true") it just tells the server what
> its params are and doesn't execute:
>
>
> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/tools/OmeroPy/src/omero/scripts.py#L391
>
>
> >> NEXT STEPS: - Jean-Marie is going to email me with some code links
> >> to the Python scripting infrastructure of OMERO.
> > ...
> >> Thinking back to our conversations in Paris (2 years ago? 3?), the
> >> alternative would be for there to be JVM code (jython?) which produces
> >> the Ice params objects. There are 2-3 server code locations which
> >> would need to be taught how not to only check for "*.py"-based
> >> scripts, but also "*.m" (matlab) and whatever else IJ2 would
> >> want/need.
> >
> > Could either Josh, J-M or both send me some pointers into the OMERO
> > codebase for getting started? I would like to grok how the current Python
> > scripting stuff works before discussing the development directions much
> > further, since right now I am pretty clueless.
>
> Ah, and I lied slightly: It's not just the file-endings, also the
> mimetype, which makes sense, I guess:
>
>
> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/server/src/ome/services/scripts/ScriptRepoHelper.java#L305
>
> Here are the rough steps that happen:
>  * Admin places a file under OMERO_DIST/lib/scripts
>  * Client asks for a list of scripts and chooses the newly appeared file
>  * Client requests the params for the files
>    - if the server has the params cached, these are returned
>    - if not, the file is executed with omero.scripts.parse=true
>  * Client generates a dialog based on the params.
>  * Script is executed with params in a separate session.
>  * ...
>
> Don't know if you've seen the script guide:
>
>
> https://www.openmicroscopy.org/site/support/omero4/developers/Modules/Scripts.html
>
> https://www.openmicroscopy.org/site/support/omero4/developers/Modules/Scripts/Guide.html
>
> But that's outlined in more detail (at least from the client perspective)
> there.
>
> Cheers,
> ~Josh.
>
>
> NB: To do this with no code changes, here's what a wrapper Python script
> would look like:
>
>
> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/examples/ScriptingService/NativeWrapperExample.py
>
>
> > Thanks!
> > -Curtis
> >
> >
> > On Wed, Mar 20, 2013 at 11:37 AM, Josh Moore <josh at glencoesoftware.com
> >wrote:
> >
> >>
> >>> SHORT TERM GOAL: Expose ImageJ2 commands as OMERO scripts. We believe
> >> this
> >>> can be done in a fairly general way, perhaps by generating Python
> >> wrappers
> >>> on demand.
> >>>
> >>> NEXT STEPS:
> >>> - Jean-Marie is going to email me with some code links to the Python
> >>> scripting infrastructure of OMERO.
> >>> - I will start investigating how to enhance it in a way that makes
> ImageJ
> >>> commands accessible to OMERO in a seamless way.
> >>> - We will get images working first, with other data types (e.g., ROIs)
> to
> >>> come later.
> >>
> >> Thinking back to our conversations in Paris (2 years ago? 3?), the
> >> alternative would be for there to be JVM code (jython?) which produces
> the
> >> Ice params objects. There are 2-3 server code locations which would
> need to
> >> be taught how not to only check for "*.py"-based scripts, but also "*.m"
> >> (matlab) and whatever else IJ2 would want/need.
> >>
> >>> During our discussion, Jean-Marie & I noted that the ImageJ GUI is very
> >>> good at "rapid prototyping" of a workflow or script, using the Macro
> >>> Recorder and related tools. Once you have the ImageJ script the way you
> >>> want, it would be very useful to be able to upload it to OMERO, for
> >>> execution across many images on the server side.
> >>
> >> If these files are uploaded to lib/scripts (like all other scripts),
> then
> >> they should show up in the
> >> drop-down menus, assuming there is the proper Ice params wrapper from
> >> above.
> >>
> >> ~J.
>


---------- Original proposal from November ----------

Date: Wed, Nov 28, 2012 at 1:47 PM
Subject: OME/ImageJ interoperability plan

Hi all,

Most of you know me, but for those new to the OME team: my name is Curtis
Rueden and I am the lead software developer at the Laboratory for Optical
and Computational Instrumentation (LOCI) at the University of
Wisconsin-Madison. LOCI has been one of the core OME teams since 2004,
spearheading the OME-TIFF and Bio-Formats projects, and we now are looking
forward to development of better ImageJ and OME interoperability.

The purpose of this mail is to outline my plan for OMERO/ImageJ
interoperability. It is something we have been discussing for years, and
have put various scattered efforts toward, but it did not have dedicated
funding until the recent Wellcome Trust award.

I see three major short-term areas of ImageJ/OMERO interoperability:

1. With OMERO.server
2. With OMERO.insight
3. OMERO-backed ImgLib2 images


== OMERO.server ==

At the Dresden hackathon last December Josh & I & others discussed the
ability for OMERO to be able to execute ImageJ2 plugins on the server side.
Since ImageJ2 plugins are much more flexible than ImageJ1 (no AWT or other
UI dependencies, in general), this is very doable. We made some progress
toward this goal at the hackathon, but since then little has been done.

Once it is possible to execute ImageJ plugins and scripts server-side, it
becomes simply another scripting option usable from all OMERO-enabled tools
such as Insight and the web client. Presumably we could reuse whatever
infrastructure already exists to upload scripts to the server, etc., but as
JAR files that contain ImageJ2 plugins.


== OMERO.insight ==

Also at the hackathon, Jean-Marie worked on making Insight capable of
executing ImageJ2 plugins, with some small assistance from me, and by the
end he had it working. But again (J-M, correct me if I am wrong), I don't
think much has happened with that since then.

Running ImageJ plugins from Insight is on the one hand really
straightforward, but on the other a bit tricky because you have to decide
what sort of image data to feed to the plugin. Rendered data from the
client side? Raw data pulled on demand from the server? And what happens to
modified pixels? Of the three areas listed above, this is the one I am
least sure about w.r.t. the specific details.


== OMERO + ImgLib2 ==

One of the things ImageJ2 was supposed to improve upon was the agnosticism
(or generality if you will) of image data sources. It must be equally
possible to process image data drawn from disk on demand, from arrays in
memory, or from an OMERO database over the wire—and neither the user nor
the algorithm being used should care about that data source. Fortunately,
ImgLib2 was designed with exactly this sort of thing in mind. The ImgLib2
architecture is extremely general, and already supports image data from
various sources.

However, there does not yet exist an "image container" for OMERO-backed
image pixels. This is what I want to create. Stephan Saalfeld (one of the
primary architects of ImgLib2) already created a CATMAID-backed image
container together with what he refers to as "stupid caching", meaning it
caches blocks in and out on demand. And he demoed this at the last two
ImageJ conferences (Barcelona & Luxembourg). So there is already code
proving this sort of thing is feasible. Someone (me) just needs to do the
same for OMERO.

Once we have this OMERO-backed ImgLib2 image with good caching (probably
using the Ehcache project which Josh suggested), working with your OMERO
data from within ImageJ2 will become much closer to a reality. There are
still some (relatively minor) architectural changes we need to make in
ImageJ2 itself to fully realize this potential, but it is absolutely the
direction we are going anyway.


== Next steps ==

My next step will be to focus on the OMERO-backed image container for
ImgLib2. Our 2.0.0-beta7 release of ImageJ2, slated for late January, is
focused on "robust core support for N-dimensional data, including very
large image data, and data/display decoupling." This is perfectly in line
with the needed OMERO+ImgLib2 development. It would be an awesome showcase
for ImageJ2 if I could get a prototype of OMERO interoperability working in
conjunction with beta7, or soon thereafter.

I am really excited about the other avenues of interoperability as well,
but they can wait a bit longer until we have the core OMERO-backed data in
place.


== Longer term ==

Josh reminded me that we will also need various other OMERO features
available from ImageJ, such as tagging, metadata querying, server-side
rendering, and so on. And we will obviously need the ability to e.g. commit
modified data back to the database (although there is an open question of
how the versioning should work with that). We can think about further
planning for these features once we have made some initial progress on the
above, though.

Lastly, in general, we want to continue to explore areas where
OMERO.insight and ImageJ2 can be brought closer together, reusing
functionality where possible.

All feedback welcome.

Regards,
Curtis


---------- Revised proposal from earlier this month ----------

Date: Tue, Mar 5, 2013 at 10:43 PM
Subject: Re: OME/ImageJ interoperability plan

Hi everyone,

Thanks everyone for last week's meeting. Jean-Marie & I had a great
discussion and ideas session for the OMERO/ImageJ interoperability project.
The following are my notes, along with my response to earlier discussion in
this thread.

As mentioned previously, there are two main ways for ImageJ to operate with
respect to OMERO:
1) executed as a job on the server side;
2) communicating with OMERO as a client.

>From various discussions, suggestions, perspectives, etc., it is clear that
we need both. But we spent much of our meeting time discussing server-side
functionality.

As many of you know, right now, OMERO scripts are all Python based. This
allows both OMERO.web and OMERO.insight to discover and invoke available
Python scripts fairly easily.

SHORT TERM GOAL: Expose ImageJ2 commands as OMERO scripts. We believe this
can be done in a fairly general way, perhaps by generating Python wrappers
on demand.

NEXT STEPS:
- Jean-Marie is going to email me with some code links to the Python
scripting infrastructure of OMERO.
- I will start investigating how to enhance it in a way that makes ImageJ
commands accessible to OMERO in a seamless way.
- We will get images working first, with other data types (e.g., ROIs) to
come later.

LONGER TERM: ImageJ2 scripts should attach structure annotations describing
how the results were acquired (which parameter values, input data, etc.).
On the other hand, this may be a good use case for a more general-purpose
data provenance data schema, though, like Bob Murphy brought up during one
of the presentations.

Kevin asked the Davis Lab which ImageJ plugins they would like to see
supported in OMERO first. Here is the list so far:

- Image/stacks/Z project
- Image/stacks/orthogonal views
- stacks/tools/combine
- stack/tools/slice remover
- image/transform/ "all the different options"
- image/adjust/threshold
- process/binary/make binary
- process/math/ "all the different options"
- process/image calculator
- analyse/tools/ROI manager
- analyse/set measurements
- analyse/measure
- process/macros/run "used with custom macros"

- denoising (bilateral filter)
- deconvolution (deconvolution lab and/or Bob Dougherty's iterative 3D)
- normalization (normalize local contrast, quantile-based?)
- enhance local contrast (CLAHE in 3D)
- 3D spatial filters (mean, median, gaussian, convolve)
- 3D FFT and bandpass
- registration (stackreg, turboreg)
- SPIM registration & multi-view fusion
- FeatureJ (Laplacian, Hessian) and/or surfaceness, tubeness, vesselness
- Weka machine learning / segmentation
- Tracking (TrackMate)
- colocalization: coloc_2, JACoP
- batch macro runner

Some of these are already part of ImageJ2, which are the ones we will
target first. Others may not be practical in their current form: we will
need to be split them into an iterative/interactive portion which calls the
programmatic operation portion, the latter of which is what can be scripted
from OMERO. We will certainly not to try tackle everything on the above
list, but will pick a few "low hanging fruit" as a proof of concept. After
that, we can reexamine which ones to pursue next.

During our discussion, Jean-Marie & I noted that the ImageJ GUI is very
good at "rapid prototyping" of a workflow or script, using the Macro
Recorder and related tools. Once you have the ImageJ script the way you
want, it would be very useful to be able to upload it to OMERO, for
execution across many images on the server side.

We then discussed several other key "upload to OMERO" type operations we
would like to have within ImageJ on the client side:
- upload script to OMERO
 -- obviously though we need server-side script discovery & execution too
- upload image
- upload results table
 -- either: global, or user specifies which image it's associated with
- upload ROIs
 -- from ROI manager, or from active selection
 -- again, must somehow decide to what they are attached

We were thinking we could start by implementing a simple suite of ImageJ
commands that perform the above actions, one for each bullet point.

Another topic we discussed was the newly minted SciJava Common
plugin/service framework [1]. I showed Jean-Marie how it works [2], and he
commented that Insight had invented a "Container" that covered essentially
the same concepts: it acts as an application context that houses all of
Insight's state. So Jean-Marie is going to investigate whether SciJava
Common could meet Insight's needs there instead. One advantage of switching
is that with SciJava Common, Insight would have not only a working service
framework and application context, but an extensible plugin framework,
meaning that it could define its own types of Insight plugins for various
purposes [3].

I think that ImageJ2, SCIFIO and Insight all sharing this common framework
goes a long way toward interoperability between them.

Lastly, I will reply to some of Jean-Marie's earlier comments:

> Following recent discussions with users, some adjustments will be made
> to the "insight plugin" Solution to save data back to OMERO while used
> as a plugin is in progress.

That sounds good. The ability to "save data back to OMERO" can initially be
fulfilled in the various ways we listed above (upload script, upload image,
upload results table, upload ROIs). Writing each as a separate ImageJ
command makes the most sense to me. We can have a GUI to invoke the
commands, but fundamentally writing them as commands provides maximum
flexibility: they would be usable headless from ImageJ macros and scripts,
for example.

> Insight as an Image2 plugin and running ImageJ2 plugin in insight but
> due to the development stage of ImageJ2 too many things had to be
> hard-coded/assumed. Since then ImageJ2 has matured, so we could
> re-activate that work.

If you want to pursue that sort of integration as a feature of Insight when
run standalone, OK. But since Insight can already work as an ImageJ plugin,
it may be unnecessary. Users can invoke Insight from within ImageJ2; it may
not be necessary to also enable invocation of ImageJ2 from within Insight.

> During the meeting held in Dundee in November, I made good progress on
> the OMERO/imglib2 front (cf. Jason's report). I have 2 working
> prototype reading data from OMERO using different approaches (for one
> I follow catmaid example)

Excellent. I saw one of the approaches, using OmeroReader. This is
ultimately how I want to do it, too, except that we also need a smart
ImgLib2 container that can page in and page out blocks from a source, using
some arbitrary mechanism (in our case, a Bio-Formats reader). Eventually I
want to have this container also support writing data into memory too, and
caching out those "dirty blocks" to disk, with the modified data overriding
the original input mechanism. That way, you could read in blocks of data
from disk or from OMERO using Bio-Formats, make changes by running ImageJ
commands, etc., all on a huge dataset with bits of it being cached in and
out of memory using scratch space on disk. Finally you could save the
changed image back to OMERO using the "upload image to OMERO" command (see
above).

Regarding the other approach—the one you modeled after Stephan Saalfeld's
CATMAID integration—where is that code? I would love to check it out.

> Looking closely at the code of ImgLib2, some assumptions are made  so
> they will need to be modified in order to work and we will also need
> to re-discuss the dependency stack of imglib2 (cf. Dresden discussion)

Could you elaborate on that a bit? I designed the ImgLib2 build system, so
can comment on any potential dependency issues there. The typical solution
to such things is to modularize code, both to avoid circular dependencies,
and to split code with differing dependencies into separate sub-projects.

> Following Dresden hackathon, I started to modify insight so it can be
> used in headless mode (still some work to be done there) and this is
> tested/used as prototype by Knime, The reason for such work is that
> people will not have to re-invent the communication layer with OMERO
> and/or the caching (ehcache is already in used by insight!)

I would definitely like to reuse any OMERO communication layer that already
exists. Initially, the OmeroReader class in ome-io is perfect for this,
though later we can explore how to reuse parts of Insight that provide
other sorts of communication beyond images. For this to work, though, it
will be paramount for Insight to be modular, and structured as reusable
libraries/JARs (ideally Maven artifacts) consumable within ImgLib2- and
ImageJ2-based projects.

As for Insight's ehcache-based caching, it may be reusable from within
ImgLib2 as part of the cell container implementation I mentioned above, but
I would need to see the code. Could you send me a pointer to the relevant
code you had in mind?

Regards,
Curtis

[1] Background: It was originally the ImageJ2 plugin framework, but Mark
Hiner & I realized that it would be ideal for SCIFIO to also use the same
framework, rather than reinventing its own (which it was in the process of
doing). So I split it out of ImageJ2, and now both ImageJ2 and SCIFIO are
using it, to great effect. See also:
https://groups.google.com/d/topic/scijava/1l61Nd_f9Tw/discussion,
https://github.com/scijava/scijava-common

[2]
https://github.com/imagej/imagej-tutorials/blob/b7c197a9/intro-to-imagej-api/src/main/java/IntroToImageJAPI.java

[3] One example in ImageJ2 that I showed Jean-Marie is that of the
CalculatorOp, an ImageJ2 plugin type that extends the Image Calculator
command with an additional operation. The Image Calculator command lets you
process two images with a binary operation, but in ImageJ1 the operations
were hardcoded. In ImageJ2 you can define your own. See:
https://github.com/imagej/imagej/tree/518650c8/plugins/commands/src/main/java/imagej/core/commands/calculator

On Thu, Nov 29, 2012 at 12:49 AM, Jean-Marie Burel <j.burel at dundee.ac.uk>
 wrote:

> Hi Curtis
>
> Few points to clarify
> OMERO/ImageJ
>
> I have modified the insight client to be an imageJ plugin (or other plugin
> e.g. It is used as a knime node too).This has been released and is already
> in use in various labs.
> Following recent discussions with users, some adjustments will be made to
> the "insight plugin"
> Solution to save data back to OMERO while used as a plugin is in progress.
>
> While in Dresden, I had integrations prototype with ImageJ2
> Insight as an Image2 plugin and running ImageJ2 plugin in insight but due
> to the development stage of ImageJ2 too many things had to be
> hard-coded/assumed. Since then ImageJ2 has matured, so we could re-activate
> that work.
>
>
> OMERO/imgLib2
> During the meeting held in Dundee in November, I made good progress on the
> OMERO/imglib2 front (cf. Jason's report).
> I have 2 working prototype reading data from OMERO using different
> approaches (for one I follow catmaid example)
> Looking closely at the code of ImgLib2, some assumptions are made  so they
> will need to be modified in order to work and we will also need to
> re-discuss the dependency stack of imglib2 (cf. Dresden discussion)
>
> Following Dresden hackathon, I started to modify insight so it can be used
> in headless mode (still some work to be done there) and this is tested/used
> as prototype by Knime,
>  The reason for such work is that people will not have to re-invent the
> communication layer with OMERO and/or the caching (ehcache is already in
> used by insight!)
>
>
>  As you can see, a lot of work has already been done and/or released.
> Happy to discuss so we do not waste time and effort.
>
> Ciao
>
> Jmarie
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20130321/02068281/attachment-0001.html>

More information about the ome-devel mailing list