[ome-devel] [ome-nitpick] OME/ImageJ interoperability plan

Mon Apr 1 17:32:23 BST 2013

Hi all,

> we are discussing how best to support execution of ImageJ commands and
> scripts from an OMERO server.

For those interested in following this work, there is now a ticket tree in
the OME Trac:
    http://trac.openmicroscopy.org.uk/ome/ticket/10608

It covers the various avenues of interoperability discussed in this thread.

Regards,
Curtis

On Thu, Mar 21, 2013 at 10:30 AM, Curtis Rueden <ctrueden at wisc.edu> wrote:

> Hi Josh & everyone,
>
> > Thoughts on moving to ome-devel?
>
> Rerouting as suggested!
>
> For those newcomers: we are discussing how best to support execution of
> ImageJ commands and scripts from an OMERO server. The idea is that you
> could upload desired ImageJ (probably only ImageJ2) scripts to your OMERO
> installation, where they could run server-side (or potentially delegated to
> a cluster, etc.) and return their results straight back into OMERO, very
> similar to how you can run Python scripts from OMERO now.
>
> I have also forwarded my initial and revised plans originally discussed
> amongst the OME team internally (see bottom). Comments, suggestions,
> thoughts, etc., from the OME developer community are very welcome!
>
> > There's a description framework:
>
> Thanks for all the pointers, Josh. I will dig in today.
>
> > Don't know if you've seen the script guide:
>
> Nope, but I will read it now! Thanks!
>
> Regards,
> Curtis
>
>
> On Thu, Mar 21, 2013 at 5:01 AM, Josh Moore <josh at glencoesoftware.com>wrote:
>
>> Thoughts on moving to ome-devel?
>>
>> On Mar 20, 2013, at 9:49 PM, Curtis Rueden wrote:
>>
>> > Hi Josh, J-M, et. al,
>> >
>> >> If these files are uploaded to lib/scripts (like all other scripts),
>> >> then they should show up in the drop-down menus, assuming there is the
>> >> proper Ice params wrapper from above.
>> >
>> > Are you suggesting there would be one (or some constant C) Ice params
>> > wrapper which would enable ImageJ support? Or that each ImageJ script
>> added
>> > to the server would have its own (autogenerated?) Ice params wrapper?
>>
>> There's a description framework:
>>
>>
>> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/blitz/resources/omero/Scripts.ice
>>
>> So each script would define it's own INs and OUTs, just like the Python
>> scripts would do. In fact, the clients wouldn't really even know which was
>> which, other than perhaps the file endings.
>>
>> What's necessary to make that work is some logic which when called with a
>> special property ("omero.script.parse=true") it just tells the server what
>> its params are and doesn't execute:
>>
>>
>> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/tools/OmeroPy/src/omero/scripts.py#L391
>>
>>
>> >> NEXT STEPS: - Jean-Marie is going to email me with some code links
>> >> to the Python scripting infrastructure of OMERO.
>> > ...
>> >> Thinking back to our conversations in Paris (2 years ago? 3?), the
>> >> alternative would be for there to be JVM code (jython?) which produces
>> >> the Ice params objects. There are 2-3 server code locations which
>> >> would need to be taught how not to only check for "*.py"-based
>> >> scripts, but also "*.m" (matlab) and whatever else IJ2 would
>> >> want/need.
>> >
>> > Could either Josh, J-M or both send me some pointers into the OMERO
>> > codebase for getting started? I would like to grok how the current
>> Python
>> > scripting stuff works before discussing the development directions much
>> > further, since right now I am pretty clueless.
>>
>> Ah, and I lied slightly: It's not just the file-endings, also the
>> mimetype, which makes sense, I guess:
>>
>>
>> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/components/server/src/ome/services/scripts/ScriptRepoHelper.java#L305
>>
>> Here are the rough steps that happen:
>>  * Admin places a file under OMERO_DIST/lib/scripts
>>  * Client asks for a list of scripts and chooses the newly appeared file
>>  * Client requests the params for the files
>>    - if the server has the params cached, these are returned
>>    - if not, the file is executed with omero.scripts.parse=true
>>  * Client generates a dialog based on the params.
>>  * Script is executed with params in a separate session.
>>  * ...
>>
>> Don't know if you've seen the script guide:
>>
>>
>> https://www.openmicroscopy.org/site/support/omero4/developers/Modules/Scripts.html
>>
>> https://www.openmicroscopy.org/site/support/omero4/developers/Modules/Scripts/Guide.html
>>
>> But that's outlined in more detail (at least from the client perspective)
>> there.
>>
>> Cheers,
>> ~Josh.
>>
>>
>> NB: To do this with no code changes, here's what a wrapper Python script
>> would look like:
>>
>>
>> https://github.com/openmicroscopy/openmicroscopy/blob/dev_4_4/examples/ScriptingService/NativeWrapperExample.py
>>
>>
>> > Thanks!
>> > -Curtis
>> >
>> >
>> > On Wed, Mar 20, 2013 at 11:37 AM, Josh Moore <josh at glencoesoftware.com
>> >wrote:
>> >
>> >>
>> >>> SHORT TERM GOAL: Expose ImageJ2 commands as OMERO scripts. We believe
>> >> this
>> >>> can be done in a fairly general way, perhaps by generating Python
>> >> wrappers
>> >>> on demand.
>> >>>
>> >>> NEXT STEPS:
>> >>> - Jean-Marie is going to email me with some code links to the Python
>> >>> scripting infrastructure of OMERO.
>> >>> - I will start investigating how to enhance it in a way that makes
>> ImageJ
>> >>> commands accessible to OMERO in a seamless way.
>> >>> - We will get images working first, with other data types (e.g.,
>> ROIs) to
>> >>> come later.
>> >>
>> >> Thinking back to our conversations in Paris (2 years ago? 3?), the
>> >> alternative would be for there to be JVM code (jython?) which produces
>> the
>> >> Ice params objects. There are 2-3 server code locations which would
>> need to
>> >> be taught how not to only check for "*.py"-based scripts, but also
>> "*.m"
>> >> (matlab) and whatever else IJ2 would want/need.
>> >>
>> >>> During our discussion, Jean-Marie & I noted that the ImageJ GUI is
>> very
>> >>> good at "rapid prototyping" of a workflow or script, using the Macro
>> >>> Recorder and related tools. Once you have the ImageJ script the way
>> you
>> >>> want, it would be very useful to be able to upload it to OMERO, for
>> >>> execution across many images on the server side.
>> >>
>> >> If these files are uploaded to lib/scripts (like all other scripts),
>> then
>> >> they should show up in the
>> >> drop-down menus, assuming there is the proper Ice params wrapper from
>> >> above.
>> >>
>> >> ~J.
>>
>
>
> ---------- Original proposal from November ----------
>
> Date: Wed, Nov 28, 2012 at 1:47 PM
> Subject: OME/ImageJ interoperability plan
>
>
> Hi all,
>
> Most of you know me, but for those new to the OME team: my name is Curtis
> Rueden and I am the lead software developer at the Laboratory for Optical
> and Computational Instrumentation (LOCI) at the University of
> Wisconsin-Madison. LOCI has been one of the core OME teams since 2004,
> spearheading the OME-TIFF and Bio-Formats projects, and we now are looking
> forward to development of better ImageJ and OME interoperability.
>
> The purpose of this mail is to outline my plan for OMERO/ImageJ
> interoperability. It is something we have been discussing for years, and
> have put various scattered efforts toward, but it did not have dedicated
> funding until the recent Wellcome Trust award.
>
> I see three major short-term areas of ImageJ/OMERO interoperability:
>
> 1. With OMERO.server
> 2. With OMERO.insight
> 3. OMERO-backed ImgLib2 images
>
>
> == OMERO.server ==
>
> At the Dresden hackathon last December Josh & I & others discussed the
> ability for OMERO to be able to execute ImageJ2 plugins on the server side.
> Since ImageJ2 plugins are much more flexible than ImageJ1 (no AWT or other
> UI dependencies, in general), this is very doable. We made some progress
> toward this goal at the hackathon, but since then little has been done.
>
> Once it is possible to execute ImageJ plugins and scripts server-side, it
> becomes simply another scripting option usable from all OMERO-enabled tools
> such as Insight and the web client. Presumably we could reuse whatever
> infrastructure already exists to upload scripts to the server, etc., but as
> JAR files that contain ImageJ2 plugins.
>
>
> == OMERO.insight ==
>
> Also at the hackathon, Jean-Marie worked on making Insight capable of
> executing ImageJ2 plugins, with some small assistance from me, and by the
> end he had it working. But again (J-M, correct me if I am wrong), I don't
> think much has happened with that since then.
>
> Running ImageJ plugins from Insight is on the one hand really
> straightforward, but on the other a bit tricky because you have to decide
> what sort of image data to feed to the plugin. Rendered data from the
> client side? Raw data pulled on demand from the server? And what happens to
> modified pixels? Of the three areas listed above, this is the one I am
> least sure about w.r.t. the specific details.
>
>
> == OMERO + ImgLib2 ==
>
> One of the things ImageJ2 was supposed to improve upon was the agnosticism
> (or generality if you will) of image data sources. It must be equally
> possible to process image data drawn from disk on demand, from arrays in
> memory, or from an OMERO database over the wire—and neither the user nor
> the algorithm being used should care about that data source. Fortunately,
> ImgLib2 was designed with exactly this sort of thing in mind. The ImgLib2
> architecture is extremely general, and already supports image data from
> various sources.
>
> However, there does not yet exist an "image container" for OMERO-backed
> image pixels. This is what I want to create. Stephan Saalfeld (one of the
> primary architects of ImgLib2) already created a CATMAID-backed image
> container together with what he refers to as "stupid caching", meaning it
> caches blocks in and out on demand. And he demoed this at the last two
> ImageJ conferences (Barcelona & Luxembourg). So there is already code
> proving this sort of thing is feasible. Someone (me) just needs to do the
> same for OMERO.
>
> Once we have this OMERO-backed ImgLib2 image with good caching (probably
> using the Ehcache project which Josh suggested), working with your OMERO
> data from within ImageJ2 will become much closer to a reality. There are
> still some (relatively minor) architectural changes we need to make in
> ImageJ2 itself to fully realize this potential, but it is absolutely the
> direction we are going anyway.
>
>
> == Next steps ==
>
> My next step will be to focus on the OMERO-backed image container for
> ImgLib2. Our 2.0.0-beta7 release of ImageJ2, slated for late January, is
> focused on "robust core support for N-dimensional data, including very
> large image data, and data/display decoupling." This is perfectly in line
> with the needed OMERO+ImgLib2 development. It would be an awesome showcase
> for ImageJ2 if I could get a prototype of OMERO interoperability working in
> conjunction with beta7, or soon thereafter.
>
> I am really excited about the other avenues of interoperability as well,
> but they can wait a bit longer until we have the core OMERO-backed data in
> place.
>
>
> == Longer term ==
>
> Josh reminded me that we will also need various other OMERO features
> available from ImageJ, such as tagging, metadata querying, server-side
> rendering, and so on. And we will obviously need the ability to e.g. commit
> modified data back to the database (although there is an open question of
> how the versioning should work with that). We can think about further
> planning for these features once we have made some initial progress on the
> above, though.
>
> Lastly, in general, we want to continue to explore areas where
> OMERO.insight and ImageJ2 can be brought closer together, reusing
> functionality where possible.
>
> All feedback welcome.
>
> Regards,
> Curtis
>
>
> ---------- Revised proposal from earlier this month ----------
>
> Date: Tue, Mar 5, 2013 at 10:43 PM
> Subject: Re: OME/ImageJ interoperability plan
>
> Hi everyone,
>
> Thanks everyone for last week's meeting. Jean-Marie & I had a great
> discussion and ideas session for the OMERO/ImageJ interoperability project.
> The following are my notes, along with my response to earlier discussion in
> this thread.
>
> As mentioned previously, there are two main ways for ImageJ to operate
> with respect to OMERO:
> 1) executed as a job on the server side;
> 2) communicating with OMERO as a client.
>
> From various discussions, suggestions, perspectives, etc., it is clear
> that we need both. But we spent much of our meeting time discussing
> server-side functionality.
>
> As many of you know, right now, OMERO scripts are all Python based. This
> allows both OMERO.web and OMERO.insight to discover and invoke available
> Python scripts fairly easily.
>
> SHORT TERM GOAL: Expose ImageJ2 commands as OMERO scripts. We believe this
> can be done in a fairly general way, perhaps by generating Python wrappers
> on demand.
>
> NEXT STEPS:
> - Jean-Marie is going to email me with some code links to the Python
> scripting infrastructure of OMERO.
> - I will start investigating how to enhance it in a way that makes ImageJ
> commands accessible to OMERO in a seamless way.
> - We will get images working first, with other data types (e.g., ROIs) to
> come later.
>
> LONGER TERM: ImageJ2 scripts should attach structure annotations
> describing how the results were acquired (which parameter values, input
> data, etc.). On the other hand, this may be a good use case for a more
> general-purpose data provenance data schema, though, like Bob Murphy
> brought up during one of the presentations.
>
> Kevin asked the Davis Lab which ImageJ plugins they would like to see
> supported in OMERO first. Here is the list so far:
>
> - Image/stacks/Z project
> - Image/stacks/orthogonal views
> - stacks/tools/combine
> - stack/tools/slice remover
> - image/transform/ "all the different options"
> - image/adjust/threshold
> - process/binary/make binary
> - process/math/ "all the different options"
> - process/image calculator
> - analyse/tools/ROI manager
> - analyse/set measurements
> - analyse/measure
> - process/macros/run "used with custom macros"
>
> - denoising (bilateral filter)
> - deconvolution (deconvolution lab and/or Bob Dougherty's iterative 3D)
> - normalization (normalize local contrast, quantile-based?)
> - enhance local contrast (CLAHE in 3D)
> - 3D spatial filters (mean, median, gaussian, convolve)
> - 3D FFT and bandpass
> - registration (stackreg, turboreg)
> - SPIM registration & multi-view fusion
> - FeatureJ (Laplacian, Hessian) and/or surfaceness, tubeness, vesselness
> - Weka machine learning / segmentation
> - Tracking (TrackMate)
> - colocalization: coloc_2, JACoP
> - batch macro runner
>
> Some of these are already part of ImageJ2, which are the ones we will
> target first. Others may not be practical in their current form: we will
> need to be split them into an iterative/interactive portion which calls the
> programmatic operation portion, the latter of which is what can be scripted
> from OMERO. We will certainly not to try tackle everything on the above
> list, but will pick a few "low hanging fruit" as a proof of concept. After
> that, we can reexamine which ones to pursue next.
>
> During our discussion, Jean-Marie & I noted that the ImageJ GUI is very
> good at "rapid prototyping" of a workflow or script, using the Macro
> Recorder and related tools. Once you have the ImageJ script the way you
> want, it would be very useful to be able to upload it to OMERO, for
> execution across many images on the server side.
>
> We then discussed several other key "upload to OMERO" type operations we
> would like to have within ImageJ on the client side:
> - upload script to OMERO
>  -- obviously though we need server-side script discovery & execution too
> - upload image
> - upload results table
>  -- either: global, or user specifies which image it's associated with
> - upload ROIs
>  -- from ROI manager, or from active selection
>  -- again, must somehow decide to what they are attached
>
> We were thinking we could start by implementing a simple suite of ImageJ
> commands that perform the above actions, one for each bullet point.
>
> Another topic we discussed was the newly minted SciJava Common
> plugin/service framework [1]. I showed Jean-Marie how it works [2], and he
> commented that Insight had invented a "Container" that covered essentially
> the same concepts: it acts as an application context that houses all of
> Insight's state. So Jean-Marie is going to investigate whether SciJava
> Common could meet Insight's needs there instead. One advantage of switching
> is that with SciJava Common, Insight would have not only a working service
> framework and application context, but an extensible plugin framework,
> meaning that it could define its own types of Insight plugins for various
> purposes [3].
>
> I think that ImageJ2, SCIFIO and Insight all sharing this common framework
> goes a long way toward interoperability between them.
>
> Lastly, I will reply to some of Jean-Marie's earlier comments:
>
> > Following recent discussions with users, some adjustments will be made
> > to the "insight plugin" Solution to save data back to OMERO while used
> > as a plugin is in progress.
>
> That sounds good. The ability to "save data back to OMERO" can initially
> be fulfilled in the various ways we listed above (upload script, upload
> image, upload results table, upload ROIs). Writing each as a separate
> ImageJ command makes the most sense to me. We can have a GUI to invoke the
> commands, but fundamentally writing them as commands provides maximum
> flexibility: they would be usable headless from ImageJ macros and scripts,
> for example.
>
> > Insight as an Image2 plugin and running ImageJ2 plugin in insight but
> > due to the development stage of ImageJ2 too many things had to be
> > hard-coded/assumed. Since then ImageJ2 has matured, so we could
> > re-activate that work.
>
> If you want to pursue that sort of integration as a feature of Insight
> when run standalone, OK. But since Insight can already work as an ImageJ
> plugin, it may be unnecessary. Users can invoke Insight from within
> ImageJ2; it may not be necessary to also enable invocation of ImageJ2 from
> within Insight.
>
> > During the meeting held in Dundee in November, I made good progress on
> > the OMERO/imglib2 front (cf. Jason's report). I have 2 working
> > prototype reading data from OMERO using different approaches (for one
> > I follow catmaid example)
>
> Excellent. I saw one of the approaches, using OmeroReader. This is
> ultimately how I want to do it, too, except that we also need a smart
> ImgLib2 container that can page in and page out blocks from a source, using
> some arbitrary mechanism (in our case, a Bio-Formats reader). Eventually I
> want to have this container also support writing data into memory too, and
> caching out those "dirty blocks" to disk, with the modified data overriding
> the original input mechanism. That way, you could read in blocks of data
> from disk or from OMERO using Bio-Formats, make changes by running ImageJ
> commands, etc., all on a huge dataset with bits of it being cached in and
> out of memory using scratch space on disk. Finally you could save the
> changed image back to OMERO using the "upload image to OMERO" command (see
> above).
>
> Regarding the other approach—the one you modeled after Stephan Saalfeld's
> CATMAID integration—where is that code? I would love to check it out.
>
> > Looking closely at the code of ImgLib2, some assumptions are made  so
> > they will need to be modified in order to work and we will also need
> > to re-discuss the dependency stack of imglib2 (cf. Dresden discussion)
>
> Could you elaborate on that a bit? I designed the ImgLib2 build system, so
> can comment on any potential dependency issues there. The typical solution
> to such things is to modularize code, both to avoid circular dependencies,
> and to split code with differing dependencies into separate sub-projects.
>
> > Following Dresden hackathon, I started to modify insight so it can be
> > used in headless mode (still some work to be done there) and this is
> > tested/used as prototype by Knime, The reason for such work is that
> > people will not have to re-invent the communication layer with OMERO
> > and/or the caching (ehcache is already in used by insight!)
>
> I would definitely like to reuse any OMERO communication layer that
> already exists. Initially, the OmeroReader class in ome-io is perfect for
> this, though later we can explore how to reuse parts of Insight that
> provide other sorts of communication beyond images. For this to work,
> though, it will be paramount for Insight to be modular, and structured as
> reusable libraries/JARs (ideally Maven artifacts) consumable within
> ImgLib2- and ImageJ2-based projects.
>
> As for Insight's ehcache-based caching, it may be reusable from within
> ImgLib2 as part of the cell container implementation I mentioned above, but
> I would need to see the code. Could you send me a pointer to the relevant
> code you had in mind?
>
> Regards,
> Curtis
>
> [1] Background: It was originally the ImageJ2 plugin framework, but Mark
> Hiner & I realized that it would be ideal for SCIFIO to also use the same
> framework, rather than reinventing its own (which it was in the process of
> doing). So I split it out of ImageJ2, and now both ImageJ2 and SCIFIO are
> using it, to great effect. See also:
> https://groups.google.com/d/topic/scijava/1l61Nd_f9Tw/discussion,
> https://github.com/scijava/scijava-common
>
> [2]
> https://github.com/imagej/imagej-tutorials/blob/b7c197a9/intro-to-imagej-api/src/main/java/IntroToImageJAPI.java
>
> [3] One example in ImageJ2 that I showed Jean-Marie is that of the
> CalculatorOp, an ImageJ2 plugin type that extends the Image Calculator
> command with an additional operation. The Image Calculator command lets you
> process two images with a binary operation, but in ImageJ1 the operations
> were hardcoded. In ImageJ2 you can define your own. See:
> https://github.com/imagej/imagej/tree/518650c8/plugins/commands/src/main/java/imagej/core/commands/calculator
>
> On Thu, Nov 29, 2012 at 12:49 AM, Jean-Marie Burel <j.burel at dundee.ac.uk>
>  wrote:
>
>> Hi Curtis
>>
>> Few points to clarify
>> OMERO/ImageJ
>>
>> I have modified the insight client to be an imageJ plugin (or other
>> plugin e.g. It is used as a knime node too).This has been released and is
>> already in use in various labs.
>> Following recent discussions with users, some adjustments will be made to
>> the "insight plugin"
>> Solution to save data back to OMERO while used as a plugin is in
>> progress.
>>
>> While in Dresden, I had integrations prototype with ImageJ2
>> Insight as an Image2 plugin and running ImageJ2 plugin in insight but due
>> to the development stage of ImageJ2 too many things had to be
>> hard-coded/assumed. Since then ImageJ2 has matured, so we could re-activate
>> that work.
>>
>>
>> OMERO/imgLib2
>> During the meeting held in Dundee in November, I made good progress on
>> the OMERO/imglib2 front (cf. Jason's report).
>> I have 2 working prototype reading data from OMERO using different
>> approaches (for one I follow catmaid example)
>> Looking closely at the code of ImgLib2, some assumptions are made  so
>> they will need to be modified in order to work and we will also need to
>> re-discuss the dependency stack of imglib2 (cf. Dresden discussion)
>>
>> Following Dresden hackathon, I started to modify insight so it can be
>> used in headless mode (still some work to be done there) and this is
>> tested/used as prototype by Knime,
>>  The reason for such work is that people will not have to re-invent the
>> communication layer with OMERO and/or the caching (ehcache is already in
>> used by insight!)
>>
>>
>>  As you can see, a lot of work has already been done and/or released.
>> Happy to discuss so we do not waste time and effort.
>>
>> Ciao
>>
>> Jmarie
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20130401/89fb55ec/attachment-0001.html>