[ome-devel] Fwd: Notes from Ricardo re last LSR meeting at UCL

Tue Oct 28 14:49:34 GMT 2014

Dear Jason and everyone,

A quick recapture of the round table discussion we went through with the ones present on the 17th (~12 participants).

- The general interest is on having OMERO store super-resolution (SR) data and not prioritise having OMERO actually analyse the data. This is due to the large family of existing analysis methods for SR, specially within the realm of SR Localisation Microscopy.

- For SR Localisation Microscopy (e.g.: PALM/STORM) in particular there are 3 types of data used that need to be stored:
A) the raw-sequence of frames as collected by the microscope (generally 200-150.000)
B) a particle table generated after the raw-sequence is analysed for particle detection and localisation (generally 1^3-10^9 particle elements)
C) a super-resolution reconstruction, a single-frame or few-frames, but generally >= 5120x5120 pixels in size each frame

There are two main tasks that we would like to see integrated into OMERO:
1) The capacity to store the particle table, the difficulty here is that the particle table can follow multiple formats depending on the algorithm used although it tends to have some common elements such as particles X, Y, Z positions and particle signal. We've discussed HDF5 - one the issues raised was that developers creating SR Localisation Microscopy may not be comfortable with the format - we've also discussed XML - but it may be a bit computationally overwhelming to parse xml particle tables with 10^9 elements.
2) We need to have clear ways to link this data together, allowing us to backtrace the relations between raw-data, particle-tables and SR reconstructions.

Overall it was a good discussion as it allows us to focus a bit more. I would propose for us to set up a skype meeting before we engage a development session. Would you agree? If so, I can try to set up a doodle for this.

All the best,
-Ricardo

On Mon, Oct 20, 2014 at 1:34 PM, Jason Swedlow <j.r.swedlow at dundee.ac.uk<mailto:j.r.swedlow at dundee.ac.uk>> wrote:
Hi All

Apologies for missing the meeting on Friday.  Family issues have messed up my schedule.

Regarding David Pinto's critique, a few comments, which I think completely agree with (almost) everything he says, but probably reaches a different conclusion, which we can definitely discuss.

OMERO does not include a workflow system. There are several, built with different targets, concepts and focus on usability (or lack thereof). Taverna (http://www.taverna.org.uk<http://www.taverna.org.uk/>), KNIME (https://www.knime.org<https://www.knime.org/>) and Galaxy (http://galaxyproject.org<http://galaxyproject.org/>) are three well-known workflow systems.  Other groups outside the OME Consortium have built OMERO/KNIME and OMERO/Galaxy integration (in fact we were so excited about CRS4's integration of OMERO and Galaxy, we managed to get funding to continue this work;http://www.openmicroscopy.org/site/products/partner/omero.biobank/).  We prefer the flexibility that "integrate your own, for your own problem" delivers.

In addition, OMERO does not currently provide good ways (where "good" is defined as strong, technically sensible, scaleable) to express relationships between different images (e.g., raw, processed1, processed2, analysed, etc.) and other data.  That's something we are working on now, based on new funding we have just started (and is the focus of what Simon Li is working on)

As I understand it, those points are the basis for much of what David is saying.

However, I disagree with David's statement that these relationships and workflows can't be expressed with OMERO now.  Relationships can be expressed (admittedly in a fairly hacked way) using OMERO's annotation mechanism.  Several groups have used this quite successfully to process much larger and complex datasets than what we are discussing here.  Simple workflows (A-->B-->C) can be run using OMERO's Python scripting system, or for example the extensions that David describes.

Regarding David's suggestion for a standardized file format, there are two parts to this.  First, is quite simple-- the column names we have to support.  I presume this group can define them quite quickly and easily.  The second, and trickier, is how these are implemented-- CSV file, HDF5 file, etc. (Note: slightly tongue-in-cheek, but very true in practice: http://xkcd.com/927/.)  If this group can define those column names, we can look at different implementations-- file-based, API-based in OMERO-- and start to build solutions.

Speaking from an OMERO perspective, based on my understanding of the Localisation use case, OMERO.tables can handle the output data calculated by most localisation routines, and does work at the scale this data is generated and analysed. Again, specifics would help here.

Thus if possible, I'd like to start concretely defining what we need to do, focusing on the localization output.  In the OME project, we'd like to start linking OMERO to localisation results, not just because we can, but because if we start this work, we can have a better understanding of other usage requirements-- image generation and visualisation of localisation analytics are two that come to mind.  This is how we have worked with HCS, Matlab, ROIs, digital pathology, FLIM, LSFM (current work), and so on.  We need to determine what workflow system is most suitable for localisation data, and we need to understand the relationships we need to express and at what scale.  Those requirements are best explored with prototyping work.

Thus, I'd propose we start that work.  We'd be happy to sponsor a small meeting in Dundee the week of Nov 17 to build the first localisation data prototypes.  That will allow us to have something concrete to test, play with, discuss and take forward.  As I said, this is the approach we have used for several other extensions of OMERO, and we'd love to try this with localisation as well.

We'd welcome comments or feedback on these ideas.

Thanks again to all for the efforts to take this concept forward, and especially David for a very clear, detailed write-up.

Cheers,

Jason

--------------------
Centre for Gene Regulation & Expression | Open Microscopy Environment | University of Dundee

Phone:  +44 (0) 1382 385819<tel:%2B44%20%280%29%201382%20385819>
email: j.swedlow at dundee.ac.uk<mailto:j.swedlow at dundee.ac.uk>

Web: http://www.lifesci.dundee.ac.uk/people/jason-swedlow
Open Microscopy Environment: http://openmicroscopy.org<http://openmicroscopy.org/>
-----Original Message-----
From: Ian Dobbie [mailto:ian.dobbie at bioch.ox.ac.uk<mailto:ian.dobbie at bioch.ox.ac.uk>]
Sent: Friday, October 17, 2014 10:02 AM
To: Ricardo Henriques
Cc: Graeme Ball; Simon Li; Munro, Ian; Jason Swedlow; Josh Moore; Dougas Russell; Keller, Debora; Ashley J Cadby; French, Paul (PHOT) M W (Photonics); Ilan Davis; Pedro Almada; Sian Culley; Kwakwa, Kwasi A; Dunsby, Christopher W; Nils Gustafsson; David Pinto; Mortari, Filippo
Subject: Re: Next LSR group meeting - UCL (LMCB) October 17th 16:00

Hi Richardo and everyone else,

I'm very sorry for the late change of plan but unfortunately none of us from Oxford will be able to make it today. I will try to make it by skype if at all possible.

David Pinto has written a very good summary of our current position and what he sees as the current issues with data processing in OMERO, please find it attached. The active session issue has been a huge headache for us, requiring lots of jumping through hoops to allow us to ship off even medium sized data sets for analysis and keep the connection alive so we can re-import the results.

Hope to speak to you this afternoon,

Thanks,

Ian

The University of Dundee is a registered Scottish Charity, No: SC015096

--
Ricardo Henriques, Ph. D.
Senior Lecturer, Academic Lead for Imaging in Biosciences
http://www.ucl.ac.uk/lmcb/research-group/ricardo-henriques-research-group
MRC-Laboratory for Molecular Cell Biology. University College London.
Gower Street, London WC1E 6BT. UK

Interested in advanced microscopy and super-resolution? Join our meetings:
http://www.ucl.ac.uk/lmcb/iq-club
https://mailman.ic.ac.uk/mailman/listinfo/lsr-group

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20141028/07294ed8/attachment.html>