<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr">
<div>
<div>Hi Alex<br>
<br>
</div>
Thanks for your very valuable thoughts- I think it would be great if our final format incorporated all your ideas. Our thinking was to start with a very basic format, and if it looks workable to build on it. More comments inline.<br>
</div>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 25 September 2015 at 15:42, Alex Herbert <span dir="ltr">
<<a href="mailto:a.herbert@sussex.ac.uk" target="_blank">a.herbert@sussex.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Simon,<br>
<br>
I think that the minimum requirements for a standard format would be<br>
storing the intensity of a localisation at position TXYZ.<br>
</blockquote>
<div><br>
</div>
<div>I think everyone's agreed on this point!<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Intensity and TXYZ would require units. A factor should be provided to<br>
convert the given units to standard (e.g. seconds, nm, photons). This<br>
would allow the format to record using units of choice for the software.<br>
</blockquote>
<div><br>
</div>
<div>That makes sense. However, for our first version how about we stick to one unit for simplicity. nm? pixels? Hopefully the original image contains enough metadata to convert between units.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
It would be good to allow for storing the bounds of the data. This is<br>
relevant to rendering images, density analysis and other data<br>
post-processing.<br>
</blockquote>
<div><br>
</div>
<div>I think most of us assumed the localisation data would be stored alongside the source images which would provide some (most?) of this information.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
To make it possible to link localisations together (i.e. to mark the<br>
same molecule in different frames) would require an ID field. These<br>
requirement make the file need both a header and records. It could look<br>
something like this:<br>
<br>
Header:<br>
<br>
Dimension,Min,Max,Unit,Conversion<br>
ID,1,59,Count,2.34<br>
T,1,1000,Frames,0.25<br>
X,0,64,Pixels,107<br>
Y,0,64,Pixels,107<br>
Z,-500,500,nm,1<br>
Intensity,10.987,1234,ADUs,0.025<br>
<br>
This header corresponds to an image with 59 molecules, with an average<br>
of 2.34 localisations per molecule (117 localisations in total). The<br>
image was taken for 1000 frames at a frame rate of 250 milliseconds,<br>
with 107nm pixel pitch on a 64x64 pixel camera, 500nm depth of field for<br>
3D fitting, with a gain of 40.<br>
</blockquote>
<div><br>
</div>
<div>An particle ID field also makes sense. Could you (and other people on this list) go into a bit more detail about how this additional summary information would be used? Is there a significant advantage to storing it as opposed to calculating it as necessary?<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
If tracing of localisations into molecules has not been performed then<br>
the IDs would be sequential from 1 and the Conversion for the ID set to<br>
1. Note that others may have a different take on the use of an ID<br>
column. For example it could be used to record the unique ID of a<br>
fitting candidate. In this case it would not be sequential as candidates<br>
that were rejected will be omitted from the results.<br>
</blockquote>
<div><br>
</div>
<div>Sounds reasonable.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Also note that results may be written directly by a parallel processing<br>
fitting routine. In such a case they may not necessarily be ordered by<br>
ID or by time T, and the bounds for some dimensions (ID, T, Z,<br>
Intensity) may not be available when the header is written. In this case<br>
the bounds can be optional. The bounds for XY should be available given<br>
the image from the camera is a fixed size, however it is more flexible<br>
if these are optional too.<br>
<br>
Records (simple delimited data):<br>
<br>
ID,T,X,Y,Z,Intensity<br>
<br>
This could be extended by allowing custom columns to be described in the<br>
header. These columns would be appended to the standard ones.<br>
<br>
The previously highlighted issue of pixel offset (e.g. positions 0,0 or<br>
0.5,0.5 correspond to the centre of a pixel) could be either included in<br>
the header, or just set mandatory that the coordinates use a standard<br>
offset defined by the file type.<br>
</blockquote>
<div> </div>
<div>OK. I'm not an SR user, so unless someone else decides between these options I'll just go with (0,0)<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Also note that the data parser written by the EPFL for the Localisation<br>
Microscopy Challenge 2013 had the facility to specify the delimiter for<br>
records. It may also be required that the locale for numbers be of a<br>
mandatory type or specified in the file given that some formats use<br>
different characters for decimal points. This is something that has<br>
caused a problem for my own software for a French user where the locale<br>
uses commas as the decimal separator.<br>
</blockquote>
<div><br>
</div>
<div>You might've noticed we're leaning towards HDF5. This should avoid most of the encoding/locale issues, and having to convert from ascii to binary.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
How the header is composed is open to debate. It could be XML allowing<br>
verbose descriptions of the data columns. Or a simple tabular format<br>
like the example shown above. But I recommend it should be easy to<br>
detect when the header has ended and the rows of data records begin.<br>
</blockquote>
<div><br>
</div>
<div>If we do go with HDF5 then we can use one or more of the description/attribute fields on tables, columns, etc. Either something really easy and simple (e.g. units for each column), or if it's more complicated, a block of OME-XML.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I hope that provides a few ideas.<br>
</blockquote>
<div> <br>
</div>
<div>It does, thanks very much.<br>
<br>
If anyone else has comments on the format we're working towards you should speak up soon!<br>
<br>
</div>
<div>Simon<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Alex<br>
<br>
<br>
<br>
On 19/09/15 12:00, <a href="mailto:ome-devel-request@lists.openmicroscopy.org.uk" target="_blank">
ome-devel-request@lists.openmicroscopy.org.uk</a> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Message: 1<br>
Date: Fri, 18 Sep 2015 15:03:49 +0100<br>
From: Simon Li <<a href="mailto:spli@dundee.ac.uk" target="_blank">spli@dundee.ac.uk</a>><br>
To: OME External Developer List<br>
<<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a>><span class=""><br>
Subject: Re: [ome-devel] Super-Resolution standard format<br>
</span>Message-ID:<br>
<<a href="mailto:CAMvbRBEkvEPyRWH0QvyCTihaYoudRkF_c-xjjFUoWY3p0bfTrQ@mail.gmail.com" target="_blank">CAMvbRBEkvEPyRWH0QvyCTihaYoudRkF_c-xjjFUoWY3p0bfTrQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<span class=""><br>
<br>
Dear Alex<br>
<br>
Thanks for adding your file formats. As you might've seen earlier in this thread I made a start on a Python script for importing some SR data into OMERO.tables, which is essentaily HDF5 behind the scenes with some OMERO-specific conventions (in terms of metadata
inside the HDF5). Our current (tentative) plans are to stick with this, but remove the OMERO-specific requirements.<br>
<br>
<a href="https://github.com/manics/omero-superresolution-tables" rel="noreferrer" target="_blank">https://github.com/manics/omero-superresolution-tables</a><br>
<br>
Ian Munro has done some work on getting PALM-siever and ThunderSTORM to work with this format.<br>
<br>
Where we could really use your help is in getting everyone to agree on what columns should be mandatory, their types (for example, should x/y/z always be in nm), and anything else we might've missed.<br>
<br>
Best wishes<br>
<br>
Simon<br>
</span></blockquote>
<br>
<div class="">
<div class="h5">_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" rel="noreferrer" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
<br>
The University of Dundee is a registered Scottish Charity, No: SC015096<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>