Hi Ghislain,<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I would be most interested to hear how you compress your 3D data! Do
you actually compress 1 slice at a time, or do you somehow use
compression like in JPEG motion where info from multiple slice are
pulled together to improve the overall compression.</blockquote><div><br></div><div>We have explored JPEG2000 compression at LOCI in the past, and can corroborate Frans's observation that 3D and 4D compression of image stacks results in pretty impressive space savings. As you suggest, it works by blocking together data across multiple planes, so decoding individual image planes becomes a little trickier.<br>
<br>It has been a while, but IIRC with our approach you could choose to divide the data into blocks of a specific size. The smaller your block size, the less effective your compression, but the more fine grained your control over decoding; for example, if you are using a block size of 2, you only need to decode the blocks corresponding to two time points and two focal planes (i.e., four image planes total) in order to fully reconstruct a specific image plane.<br>
<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">If so, how could
this be implemented for multi stack tiffs etc. and build into
bioformats?<br>
</blockquote><br>The easiest way to support this sort of thing in Bio-Formats would be to cache the most recently decoded image planes, such that as long as you retrieve image planes in a performant order, you do not have to decode anything more than once. We currently use a scheme like this in the ChannelSeparator class, to avoid reading an RGB image from disk three times when returning the R, G and B channels separately.<br>
</div></div><br>And if this sort of compression becomes more widespread, we could add an explicit API for it in the future, too.<br><br>-Curtis<br><br><div class="gmail_quote">On Sun, Dec 7, 2008 at 4:04 PM, Ghislain Bonamy <span dir="ltr"><<a href="mailto:GBonamy@gnf.org">GBonamy@gnf.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Frans,<br>
<br>
Thanks for the answer. Like you I agree that JPEG2000, offers state of the art compression when it comes to images. This is why I have implemented JPEG2000 compression in bioformats for the TIFF and the OME files (readers and writers). This should hopefully be released soon. I am also trying to implement a solution to allow for variable rate compression. For instance images that are on a 16bit scale but only use the first 8 bit, should be saved on 1byte and not 2, this would also increase compression by a factor of 2. I also would like to generalize this so that images that use 12bit of data be compressed as 12 bit images and not 16 (if this is possible, this would certainly save a lot of space this would be particularly useful in lossy compression). Finally I intend to implement lossy Jpeg2000 compression in this variable bit rate format, which should allow 10 fold compression, without many artifacts, in particular for images that are dim.<br>
<br>
I would be most interested to hear how you compress your 3D data! Do you actually compress 1 slice at a time, or do you somehow use compression like in JPEG motion where info from multiple slice are pulled together to improve the overall compression. If so, how could this be implemented for multi stack tiffs etc. and build into bioformats?<br>
<br>
For the storage solution am not quite sure how Castor operates nor how expensive it is, but I prompted our IT guys to have a look at it. This could ultimately be the solution of choice!<br>
<br>
Best,<br>
<br>
Ghislain Bonamy, PhD<br>
_______________________________<br>
<div class="Ih2E3d">Genomic Institute of the<br>
Novartis Research<br>
Foundation<br>
</div>Functional Genomics, G214<br>
<div class="Ih2E3d">+1 (858) 812-1534 (W & F)<br>
</div>+1 (858) 354-7388 (C)<br>
<a href="http://www.gnf.org" target="_blank">www.gnf.org</a> <<a href="http://www.gnf.org/" target="_blank">http://www.gnf.org/</a>><br>
<br>
Hudson Alfa Instiute for Biotechnology<br>
<a href="http://www.haib.org" target="_blank">www.haib.org</a> <<a href="http://www.haib.org/" target="_blank">http://www.haib.org/</a>><br>
<<a href="http://www.gnf.org/" target="_blank">http://www.gnf.org/</a>><br>
<br>
________________________________<br>
<br>
From: Cornelissen, Frans [PRDBE] [mailto:<a href="mailto:FCORNELI@its.jnj.com">FCORNELI@its.jnj.com</a>]<br>
Sent: Sat 12/6/2008 4:55 AM<br>
To: <a href="mailto:ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>; Ghislain Bonamy<br>
Subject: RE: ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression and duplicate storage (Ghislain Bonamy)<br>
<div><div></div><div class="Wj3C7c"><br>
<br>
<br>
<br>
4. OMERO compression and duplicate storage (Ghislain Bonamy)<br>
<br>
----------------------------------------------------------------------<br>
<br>
Ghislain,<br>
<br>
<br>
About: "store lossy images on disk":<br>
<br>
We are currently using Jpeg 2000 compression for this.<br>
The advantage of using this would be that you do not even need to store<br>
2 versions of your image files; JP2 allow sto read any size or any<br>
quality<br>
image version out of a single (lossless or lossy) compressed file.<br>
<br>
JP2 is currently THE state of the art way to get maximum flexibility AND<br>
Maximum compression ratio; for 2D Biological images lossless, factors of<br>
2.5 to 4 are achievable (depending on the actual image content)<br>
In 3D, quasi lossless compression (PSNR >45) of 20-100 times is possible<br>
<br>
<br>
About:"slow tape system to store files"<br>
<br>
you could have a look at CASTOR from CARINGO (<a href="http://www.caringo.com" target="_blank">www.caringo.com</a>), a<br>
simple,cheap, but very efficient next-gen Software system for<br>
distributed storage on heterogenous hardware, with very good<br>
performance.<br>
Scales very well to > 80 PB.<br>
We have been testing it to store JPEG 2000 images for 18 months, worked<br>
withour a flaw!<br>
Also in use at the The Center of Inherited Disease Research (CIDR) at<br>
Johns Hopkins University<br>
<br>
<br>
<br>
Message: 4<br>
Date: Fri, 5 Dec 2008 10:10:54 -0800<br>
From: "Ghislain Bonamy" <<a href="mailto:GBonamy@gnf.org">GBonamy@gnf.org</a>><br>
Subject: [ome-users] OMERO compression and duplicate storage<br>
To: <<a href="mailto:ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>><br>
Message-ID:<br>
<<a href="mailto:F5A26DAD36F60843830631774C95CAE205807A58@EXCH2.rec.gnf.org">F5A26DAD36F60843830631774C95CAE205807A58@EXCH2.rec.gnf.org</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Dear all,<br>
<br>
I was wondering if OMERO has already implement, is thinking or would<br>
consider implementing a mechanism to store images into two different<br>
format while keeping there metadata linked.<br>
<br>
I am working in a center were we are generating several TB of data a<br>
month and where keeping all of our images on disk becomes impossible. To<br>
remedy to this, we have a >2 PT tape storage solution, which is however<br>
very slow (takes about 2-4 Minutes to retrieve a file the first time<br>
until it is de-cued) . My idea would be to store lossy images on disk<br>
for people to view and modify metadata, while keeping the full blown<br>
image onto our slow storage solution in case they needed to be<br>
reanalyzed for instance. The metadata for this images would however be<br>
kept identical between images.<br>
<br>
Perhaps, a way to do this would be to store in Omero the file and keep<br>
in the metadata a link to the original image. If there are another and<br>
better solution please let me know.<br>
<br>
In addition, does OMERO store the metadata in a compressed way (as well<br>
as the images), or is there a way to have OMERO apply a script (for<br>
instance a gzip compression) when importing images and when an image is<br>
queried?<br>
<br>
Thanks a bunch for all your help and answers,<br>
<br>
Best,<br>
<br>
Ghislain Bonamy, PhD<br>
__________________________________________<br>
<br>
Research Investigator I<br>
<br>
Genomic Institute of the<br>
<br>
Novartis Research<br>
<br>
Foundation<br>
<br>
Department of Molecular & Cell Biology, room G214<br>
<br>
10675 John Jay Hopkins Drive<br>
<br>
San Diego CA 92121<br>
<br>
USA<br>
<br>
<br>
+1 (858) 812-1534 (W & F)<br>
<br>
+1 (757) 941-4194 (H)<br>
<br>
+1 (858) 354-7388 (M)<br>
<br>
<a href="http://www.gnf.org" target="_blank">www.gnf.org</a><br>
<br>
<br>
Hudson-Alpha Institute for Biotechnology<br>
<br>
<a href="http://www.hudsonalpha.org" target="_blank">www.hudsonalpha.org</a><br>
<br>
<br>
_______________________________________________<br>
ome-users mailing list<br>
<a href="mailto:ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br>
<br>
<br>
End of ome-users Digest, Vol 45, Issue 9<br>
****************************************<br>
<br>
<br>
<br>
_______________________________________________<br>
ome-users mailing list<br>
<a href="mailto:ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br>
</div></div></blockquote></div><br>