<div dir="ltr"><div><div><div><div><div><div>Hi Nico!<br><br></div>First of all, have you tried the pyramid compression of jpeg2000? (I have not!)<br><br></div>Second, last time I tried large datasets in ome-tiff it was a huge issue. I tried to convert our 40gb+ recordings to ome-tiff and got indexing times from hell (up to 10 minutes). I never had time to properly investigate this. Part of the problem might be that I changed the OME-writer to add in JPEG-compressed data (since we have a lot of jpegs since before, and I did not feel like converting those to PNGs).<br><br></div>JPEG2000 pyramids would help you with huge 2D-images but not with huge 5D datasets. I believe the problem (anyone please correct me here) is that the ome-reader first indexes the TIFF-file - but does so in worst case by going through the entire file to find where each plane is(?). This is in no way fast in the current implementation as I suspect it jumps through the entire dataset as tifs are essentially a linked list of planes. if your output compressed file is 5gb+ then this alone is really slow<br><br></div>my solution to this would have been an extension with a special data object containing pointers to most of all planes, in a single place. thus very few reads would be needed to map all planes. but then I moved to another lab and never had time to return to this. but I think it's a problem/solution worth reconsidering. if a tif-reader does not understand such a special data object it would just ignore it, but specialized readers could gain a lot of speed by reading it<br><br></div>cheers,<br></div>Johan<br><div><div><br><br><div><br><br><div><div><br><br></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 4, 2015 at 2:08 AM, Nico Stuurman <span dir="ltr"><<a href="mailto:nico.stuurman@ucsf.edu" target="_blank">nico.stuurman@ucsf.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Dear all,<br>
<br>
I have been running into more and more individual efforts to create new<br>
file formats to deal with large datasets that need to be stored at<br>
multiple resolutions to enable fast feedback to the user. Examples are<br>
the hdf5 format used by the BigDataViewer plugin by Tobias Pietzsch and<br>
Stephan Preibisch, the hdf5 format used by Chimera (UCSF-based package<br>
primarily for crystallography and EM that also has amazing capabilities<br>
for 3D visualization of light microscopy data), the Micro-Manager<br>
SlideExplorer plugin for which Arthur Edelstein developed his own<br>
storage system, and the Micro-Manager plugin "Magelan" that Henry<br>
Pinkard is developing right now, and who also stores multiple resolution<br>
versions of the data on disk. Doubtlessly, there are many more examples.<br>
<br>
Even when conversion between these formats is possible (as long as they<br>
are reasonably documented), conversion becomes time consuming and takes<br>
up large amounts of disk space, simply because the data sets have become<br>
gigantic. The reasons why everyone designs their own formats are also<br>
clear, there simply is no standard (at least that I am aware of, if<br>
there is please do let me know! ) that let's one store gigantic datasets<br>
that give fast access to the data in multiple resolutions.<br>
<br>
Since you guys have created the standard in light microscopy with<br>
ome.tif, I assume that you have thoughts what a new standard (hdf5<br>
based?) should look like. In any case, I am very much looking forward<br>
to hearing your thoughts and I will be happy to help avoid a wild growth<br>
of different formats that we will have to live with for years to come if<br>
we do not take action soon.<br>
<br>
Best,<br>
<br>
Nico<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk" target="_blank">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr">-- <br>-----------------------------------------------------------<br>Johan Henriksson, PhD<br>Karolinska Institutet / European Bioinformatics Institute (EMBL-EBI)<br>
Labstory - Integrated laboratory documentation and databases (<a href="http://www.labstory.se" target="_blank">www.labstory.se</a>)<br><a href="http://mahogny.areta.org" target="_blank">http://mahogny.areta.org</a> <a href="http://www.endrov.net" target="_blank">http://www.endrov.net</a><br><br><a href="http://www.endrov.net" target="_blank"></a></div></div>
</div>