<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Hi Nico, Johan et al-</div>
<div><br>
</div>
<div>Apologies for the delay in responding. Nico’s email started a lot of discussion in the OME team. I saw Nico and Curtis Rueden a couple of days after this was sent at AQLM (<a href="http://www.mbl.edu/education/special-topics-courses/analytical-quantitative-light-microscopy">http://www.mbl.edu/education/special-topics-courses/analytical-quantitative-light-microscopy</a>/),
and we had a long discussion about this issue over chips, salsa, and PSFs. </div>
</div>
<div><br>
</div>
<div>As many of you know, this has been a hot topic of late. Several new and some established technologies (LSFM, DigPath, HCS and others) are now routinely generating heterogeneous (binary pixel data, metadata, and analytics) multi-dimensional, multi-TB datasets.
Within OME, we’ve been discussing how we approach this trend— whether we amend OME-TIFF, define a new format (mindful of a lot of work by others, in particular Cellh5 (<a href="http://www.cellh5.org">http://www.cellh5.org</a>/), BDV (<a href="http://fiji.sc/BigDataViewer">http://fiji.sc/BigDataViewer</a>),
OpenSlide (<a href="http://openslide.org">http://openslide.org</a>/) and others), or just wait for someone else to generate yet another file format (YAFF®) or in all likelihood several new file formats (SNFFs®) and doggedly support them all in Bio-Formats.
Johan’s proposed pointer-based solution is, if I have things correct, already implemented in Micro-Manager’s OME-TIFF, for exactly the reason described (apologies to Nico and the MM team if I have this incorrect— please do provide the accurate description
of what is going on there if I have failed).</div>
<div><br>
</div>
<div>AFAICT, there is a rather old, well-worn debate between the filesystem and database camps on this issue. To my mind, Jim Gray and colleagues captured this tension most clearly and accurately in 2005 (<a href="http://research.microsoft.com/pubs/64537/tr-2005-10.pdf">http://research.microsoft.com/pubs/64537/tr-2005-10.pdf</a>).
It’s almost certainly true that both approaches will evolve side by side, and we (where we = the community) should try to develop both solutions— they each have their place and utility. OME’s version of this is to develop a data spec (e.g., OME-TIFF), an
I/O library (Bio-Formats), and applications that use the format (OMERO). We are committed to this stratgey for any spec we develop. That might explain, but not excuse, our rather slow approach on this issue. We insist on the spec
<u><i>and</i></u> its many implementations.</div>
<div><br>
</div>
<div>Mindful of all this, OME's priority is building tools that are as useful, generic and performant as possible. In this case, that means developing a format that works in many different domains and that includes support for multi-res pixel sets, acquisition
metadata, ROIs, trajectories, etc. In discussing this with Nico and Curtis, we agreed the obvious— anything we build has to be done in steps. The over-riding immediate problem that several people face is support for large, multi-res, multi-D pixel sets.
The existing (partial) solutions are worth considering, but anything we do must also support both Java and native environments— OME is committed to at least bypassing, if not removing, the barriers between the Java and native worlds, where we have the resources
to do so. </div>
<div><br>
</div>
<div>We’ve added this topic to the workshops at our upcoming Users meeting and welcome input there (<a href="https://www.openmicroscopy.org/site/community/minutes/meetings/10th-annual-users-meeting-june-2015">https://www.openmicroscopy.org/site/community/minutes/meetings/10th-annual-users-meeting-june-2015</a>).
It looks like we will have a very strong turnout for the meeting. We’d encourage anyone interested to join us there, but obviously also welcome input on this list, Forums, etc. We’ll report back with our plan for addressing this very important point.</div>
<div><br>
</div>
<div>As always, thanks for your support.</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Jason</div>
<div><br>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="color: black; font-size: 12px;">--------------------</span></p>
<div>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;">Centre for Gene Regulation & Expression | Open Microscopy Environment | University of Dundee </span><span style="color: black;"></span></span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="color: black; font-size: 12px;"> </span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;">Phone: +44 (0) 1382 385819 </span><span style="color: black;"></span></span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;">email: <a href="mailto:j.swedlow@dundee.ac.uk" target="_blank" style="color: rgb(149, 79, 114);"><span style="color: blue;">j.swedlow@dundee.ac.uk</span></a></span><span style="color: black;"></span></span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;"> </span><span style="color: black;"></span></span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;">Web: </span><span style="color: black;"></span><span style="color: rgb(0, 0, 0);"><a href="http://www.lifesci.dundee.ac.uk/people/jason-swedlow">http://www.lifesci.dundee.ac.uk/people/jason-swedlow</a> </span></span></p>
<p class="MsoNormal" style="margin: 0in 0in 0.0001pt; color: rgb(40, 40, 40);"><span style="font-size: 12px;"><span style="color: black;">Open Microscopy Environment: <span style="color: blue;"><a href="http://openmicroscopy.org/" target="_blank" style="color: rgb(149, 79, 114);">http://openmicroscopy.org</a></span></span></span></p>
</div>
</div>
<div><br>
</div>
</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Johan Henriksson <<a href="mailto:mahogny@areta.org">mahogny@areta.org</a>><br>
<span style="font-weight:bold">Reply-To: </span>OME Development <<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, 6 May 2015 22:45<br>
<span style="font-weight:bold">To: </span>OME Development <<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [ome-devel] Fwd: File format for large data sets stored at multiple resolutions<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi Nico!<br>
<br>
</div>
First of all, have you tried the pyramid compression of jpeg2000? (I have not!)<br>
<br>
</div>
Second, last time I tried large datasets in ome-tiff it was a huge issue. I tried to convert our 40gb+ recordings to ome-tiff and got indexing times from hell (up to 10 minutes). I never had time to properly investigate this. Part of the problem might be that
I changed the OME-writer to add in JPEG-compressed data (since we have a lot of jpegs since before, and I did not feel like converting those to PNGs).<br>
<br>
</div>
JPEG2000 pyramids would help you with huge 2D-images but not with huge 5D datasets. I believe the problem (anyone please correct me here) is that the ome-reader first indexes the TIFF-file - but does so in worst case by going through the entire file to find
where each plane is(?). This is in no way fast in the current implementation as I suspect it jumps through the entire dataset as tifs are essentially a linked list of planes. if your output compressed file is 5gb+ then this alone is really slow<br>
<br>
</div>
my solution to this would have been an extension with a special data object containing pointers to most of all planes, in a single place. thus very few reads would be needed to map all planes. but then I moved to another lab and never had time to return to
this. but I think it's a problem/solution worth reconsidering. if a tif-reader does not understand such a special data object it would just ignore it, but specialized readers could gain a lot of speed by reading it<br>
<br>
</div>
cheers,<br>
</div>
Johan<br>
<div>
<div><br>
<br>
<div><br>
<br>
<div>
<div><br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, May 4, 2015 at 2:08 AM, Nico Stuurman <span dir="ltr">
<<a href="mailto:nico.stuurman@ucsf.edu" target="_blank">nico.stuurman@ucsf.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Dear all,<br>
<br>
I have been running into more and more individual efforts to create new<br>
file formats to deal with large datasets that need to be stored at<br>
multiple resolutions to enable fast feedback to the user. Examples are<br>
the hdf5 format used by the BigDataViewer plugin by Tobias Pietzsch and<br>
Stephan Preibisch, the hdf5 format used by Chimera (UCSF-based package<br>
primarily for crystallography and EM that also has amazing capabilities<br>
for 3D visualization of light microscopy data), the Micro-Manager<br>
SlideExplorer plugin for which Arthur Edelstein developed his own<br>
storage system, and the Micro-Manager plugin "Magelan" that Henry<br>
Pinkard is developing right now, and who also stores multiple resolution<br>
versions of the data on disk. Doubtlessly, there are many more examples.<br>
<br>
Even when conversion between these formats is possible (as long as they<br>
are reasonably documented), conversion becomes time consuming and takes<br>
up large amounts of disk space, simply because the data sets have become<br>
gigantic. The reasons why everyone designs their own formats are also<br>
clear, there simply is no standard (at least that I am aware of, if<br>
there is please do let me know! ) that let's one store gigantic datasets<br>
that give fast access to the data in multiple resolutions.<br>
<br>
Since you guys have created the standard in light microscopy with<br>
ome.tif, I assume that you have thoughts what a new standard (hdf5<br>
based?) should look like. In any case, I am very much looking forward<br>
to hearing your thoughts and I will be happy to help avoid a wild growth<br>
of different formats that we will have to live with for years to come if<br>
we do not take action soon.<br>
<br>
Best,<br>
<br>
Nico<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr">-- <br>
-----------------------------------------------------------<br>
Johan Henriksson, PhD<br>
Karolinska Institutet / European Bioinformatics Institute (EMBL-EBI)<br>
Labstory - Integrated laboratory documentation and databases (<a href="http://www.labstory.se" target="_blank">www.labstory.se</a>)<br>
<a href="http://mahogny.areta.org" target="_blank">http://mahogny.areta.org</a> <a href="http://www.endrov.net" target="_blank">
http://www.endrov.net</a><br>
<br>
<a href="http://www.endrov.net" target="_blank"></a></div>
</div>
</div>
</div>
</div>
</span><br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>