[ome-devel] Super-Resolution standard format

Wed Jul 1 18:49:03 BST 2015

Hi Pedro,

Indeed, I worked a bit on this a couple of years ago, but since we do 
not do much super resolution ourselves I kind of lost steam. The effort 
was also driven by Bo Huang, who also got focused on other thing.  
Instead of trying to read every format out there, I tried to develop 
libraries for the various relevant development platforms (C++, Java, 
Matlab) that could read and write the proposed format.

We settled on a format based on google protocol buffers 
(https://developers.google.com/protocol-buffers/) because it appeared to 
be a mature library for fast transmission of structured data.  Since 
PALM/STORM data sets can become quite large, speed of transmission and 
parsing (as well as data size) become important considerations.  To be 
honest, I never fully bench-marked our tagged spot format 
(https://micro-manager.org/wiki/Tagged_Spot_File_%28tsf%29_format). and 
it would be good to do such comprisons with other formats.

Please do have a look at the tagged spot format, but feel free to to 
propose something better.  in any case, I am convinced that you need to 
start out with libraries for reading and writing for all major computing 
environments, have import and export functions for text-based formats, 
and that you need to enthuse the developers community to adapt your file 
format.  Convincing Daniel Sage to adapt it as the format for the next 
PALM/STORM challenge would go a long way...

Hope this helps!

Best,

Nico

On 6/25/15 8:15 AM, Pedro Almada wrote:
> Hello all,
>
> Seamus Holden, involved with PALM-siever and the ISBI challenge has a 
> few comments to add:
>
> A few thoughts - this is actually something that I think is quite 
> interesting and I would like to see done correctly! Busy of course 
> with the new lab, but happy to help where I can.
>
> Nico Sturman (Micromanager) was trying to get something similar going,
> https://micro-manager.org/wiki/Tagged_Spot_File_%28tsf%29_format
> It seemed promising last time I checked. Link up the two efforts? I'd 
> hate to see two different Europe/ US standards emerge!
>
> Also - import can be a bit of a challenge, as I learned for PALMsiever 
> - the BIG.EPFL approach wont work for every case since some software, 
> eg rapidstorm, dynamically change the number/ order of columns 
> depending on the input settings, and use a change in the header to 
> reflect this. Then you have to get a bit cleverer with parsing the 
> file headers. An import/ export program where the import is extendible 
> via a plugin architecture might be the most flexible solution - we 
> tried to implement this for PALMsiever, but it turns out rather ugly 
> if you try and do it in MATLAB.
>
> We also have example data for a couple of extra formats which could be 
> included:
> - PeakSelector (Harold Hess's IDL software)
> - Leica GSD format
> - DAOSTORM
>
> I'm talking to Daniel Sage later today about the next ISBI challenge, 
> one thing to consider is requiring new algorithms in the challenge to 
> be submitted in the correct format.
>
> Oh - also - don't forget about the coordinate system of the different 
> softwares - does 0,0 define the corner of a pixel or the middle. 
> Inconsistently chosen between different softwares, and crucial when 
> you start comparing algorithms.
>
> Anyway, just a few initial thoughts.
> Best
>
> Seamus
>
> PhD Student
>
> MRC Laboratory for Molecular Cell Biology
> University College London
> Gower Street
> London
> WC1E 6BT
> Telephone: +44 (0)20 7679 7806
>
> On 25 June 2015 at 09:48, Pedro Almada <pedro.almada.13 at ucl.ac.uk 
> <mailto:pedro.almada.13 at ucl.ac.uk>> wrote:
>
>     Dear all,
>
>     Some of the members of the UK Super-Resolution community have
>     discussed the integration of single-molecule localization
>     microscopy data (PALM/STORM) into OMERO and what became completely
>     clear is the need to have a general way to import the output of
>     the localization algorithms. We've made a push towards having a
>     general importer and file-standard by looking at the most common
>     commercial and free software available. The early results are
>     available as a public google doc
>     <https://docs.google.com/spreadsheets/d/1YxC_WBFEvgy5jo__0_DaC6B0kzy9ptOhc2Z284PnRhU/edit?usp=sharing>.
>
>     Our goals are to follow the BioFormats route and:
>      - Create an importer capable of reading most localization data
>      - Establish a standard, flexible format that the importer would
>     write into.
>
>     It's our hope future algorithms would then start using a standard
>     format, since this would finally decouple localization data from
>     the software used to create it. A few advantages to the SR field
>     would come from this:
>      - Analysis software (eg. clustering) would no longer need to be
>     custom designed for a specific format
>      - Rendering could finally be decoupled from the localization
>     software, as the output of any algorithm should be compatible with
>     any rendering method.
>      - Comparison of the output of different algorithms could be
>     efficiently compared analytically and visual biases from different
>     rendering methods would be removed.
>
>     We need to publicly discuss how to implement this, and we believe
>     the OMERO developers community is the correct starting point. For
>     now, it seems the minimum headers common to all datasets are X
>     position, Y position and the frame number of the localized
>     particle. Other features are immediately useful such as signal
>     intensity/photon-count but not all formats include it.
>
>     There are also questions of implementation. The BIG.EPFL people
>     have tackled this problem
>     <http://bigwww.epfl.ch/smlm/methods/index.html?p=metrics>in a
>     fashion, by creating an XML generator. It asks the user to name
>     the headers, column separation, etc and an XML is created which is
>     used by their comparison software to read any text based file such
>     as .csv or .xls. However, some software packages actually generate
>     binary files and not text based files, which means an extra
>     exporting step is necessary. Also, XML has an associated overhead.
>     I would like to see an importer that can handle the binary data,
>     but have no clue as to how hard that would be to implement (I can
>     share example outputs though).
>
>     What do you think?
>
>     All the best,
>     Pedro
>
>     PhD Student
>     Quantitative Imaging and Nanobiophysics Group
>     MRC Laboratory for Molecular Cell Biology
>     University College London
>     Gower Street
>     London
>     WC1E 6BT
>     Telephone: +44 (0)20 7679 7806 <tel:%2B44%20%280%2920%207679%207806>
>
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

-- 
Nico Stuurman
Vale lab, UCSF/HHMI
Genentech Hall, N316, MC2200
600 - 16th Street
San Francisco, CA 94158-2517
415 514 3927
nico.stuurman at ucsf.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20150701/787a3f68/attachment.html>