<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Dear All-<div><br></div><div>Thanks for pushing this-- a very interesting set of ideas. If you check the notes from last year's OME User's meeting in Paris (<a href="http://www.openmicroscopy.org/site/news/2008-user-meeting-report">http://www.openmicroscopy.org/site/news/2008-user-meeting-report</a>), you'll see that we have been discussing these ideas for some time. We are continuing to discuss this issue, which invariably ends up getting into compression, multi-repositories, hierarchical storage, etc.</div><div><br></div><div>Just to clarify a few things.</div><div><br></div><div>First off, OMERO includes a flexible scripting framework and gateways for Java C++ and Python, so there are many alternatives for incorporating your own algorithms-- see <a href="http://trac.openmicroscopy.org.uk/omero/wiki/OmeroScripts">http://trac.openmicroscopy.org.uk/omero/wiki/OmeroScripts</a> and <a href="http://trac.openmicroscopy.org.uk/omero/wiki/OmeroJava">http://trac.openmicroscopy.org.uk/omero/wiki/OmeroJava</a></div><div><br></div><div>In general, the services offered by OMERO are documented at</div><div><br></div><div><a href="http://trac.openmicroscopy.org.uk/omero/wiki/ServerDesign">http://trac.openmicroscopy.org.uk/omero/wiki/ServerDesign</a></div><div><br></div><div>Second, we are working on access to Bio-Formats from C++-- I'll let Curtis weigh in there. </div><div><br></div><div>Third, you are welcome to add in your own libraries or others as well, for your own use. Adding code the OME repository for distribution means abiding by our licensing policies. We can discuss this off-line, as there is nothing really new here-- standard open source stuff. But we adhere to the ideals of the GPL license very strongly, and can not amend that. Contact me off-list if you want to discuss.</div><div><br></div><div>Fourth, delivering tools for HCS is certainly on our target list, but we can't do this halfway-- we can't deliver an HCS viewer without great import that supports many thousands of images, heatmaps, full metadata, etc. You will see these pieces coming together over the next series of our releases. </div><div><br></div><div>By all means, please use OMERO for whatever purpose you like-- that is the point. Understand that it is a developing project-- and we very much appreciate your help, suggestions, and if at all possible, contributions. </div><div><br></div><div>As always, thanks for your support.</div><div><br></div><div>Cheers,</div><div><br></div><div>Jason</div><div><br><div><div>On 12 Dec 2008, at 16:50, Ghislain Bonamy wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Frans, Curtis, Prof. Schelkens, <br><br>I have added this thread to the Ome-Dev List as it now seems more<br>suited. Hopefully I have the right address this time around.<br><br>It would indeed wonderful if the 4D JP2000 (ie. XY,C,Z) could be<br>implemented into Bioformats. Prof. Schelkens, would you agree to provide<br>your source code for the OME open source project, or at least allow the<br>use of the binaries?<br><br>Curtis, how would you fell about using some C code as part of bioformat?<br>I can imagine the problem being having platform specific compilations<br>(unless we package many different binaries and use a JNI wrapper to call<br>the right version). <br><br>Unfortunately, I am not well versed in C programming, and never wrote<br>any JNI wrappers. Perhaps, someone who shares this interest to provide<br>greater image compression could jump and try to implement some hooks to<br>Prof. Schelkens Codec. In addition if the source code is available,<br>perhaps adding also a pure java code could be useful (to mirror the JAI<br>project).<br><br>Best,<br><br>Ghislain Bonamy, PhD<br>__________________________________________<br>Research Investigator I<br><br>Genomic Institute of the<br>Novartis Research<br>Foundation<br>Department of Molecular & Cell Biology, room G214<br>10675 John Jay Hopkins Drive<br>San Diego CA 92121<br>USA<br><br>+1 (858) 812-1534 (W & F)<br>+1 (757) 941-4194 (H)<br>+1 (858) 354-7388 (M)<br><a href="http://www.gnf.org">www.gnf.org</a><br><br>Hudson-Alpha Institute for Biotechnology<br>www.hudsonalpha.org<br><br><br>-----Original Message-----<br>From: Cornelissen, Frans [PRDBE] [mailto:FCORNELI@its.jnj.com] <br>Sent: Friday, December 12, 2008 5:17 AM<br>To: Ghislain Bonamy; Curtis Rueden; Schelkens Peter<br>Cc: <br>Subject: Using 3D JPEG2000 compression in OMERO<br>Importance: High<br><br>Ghislain,<br><br>As far as 3D-JPEG2000 compression (=compression of XYZ with multiple<br>color channels) is ocncerned, there is one reference implementation<br>(written in C) from the Free Univerisity of Brussels.<br><br>You could ask Prof. Peter Schelkens for permission to use the<br>(compiled?source?-)code in OMERO (maybe under a kind of NDA about the<br>source code...)<br><br>It would be fantastic to see this embedded in OMERO/Bioformats!<br><br>Best regards, frans<br><br>-----Original Message-----<br>From: Ghislain Bonamy [mailto:GBonamy@gnf.org] <br>Sent: Friday, 12 December 2008 12:13 AM<br>To: Cornelissen, Frans [PRDBE]; Curtis Rueden<br>Cc: ome-users@lists.openmicroscopy.org.uk<br>Subject: RE: ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression<br>and duplicate storage (Ghislain Bonamy)<br><br>Frans, Curtis<br><br>Thanks for the response. It would be very nice to implement this<br>strategy in bioformats directly to avoid having to go through 3rd party<br>software to decompress images before reading them into OMERO for<br>instance but also imageJ etc.<br><br>Since OME is becoming the reference standard for images in microscopy<br>and perhaps, biological images as a whole. It would make sense to take<br>the time to implement this.<br><br>Does JAI or the version incubated at LOCI provide support for 3D<br>compression? If so how is this achieved? If not what libraries could we<br>use for JP3D? This is a bit over my head, so if you have any old code<br>for me to play with and try to implement in the compressor that would be<br>great.<br><br>Best,<br><br>Ghislain Bonamy, PhD<br>__________________________________________<br>Research Investigator I<br><br>Genomic Institute of the<br>Novartis Research<br>Foundation<br>Department of Molecular & Cell Biology, room G214<br>10675 John Jay Hopkins Drive<br>San Diego CA 92121<br>USA<br><br>+1 (858) 812-1534 (W & F)<br>+1 (757) 941-4194 (H)<br>+1 (858) 354-7388 (M)<br>www.gnf.org<br><br>Hudson-Alpha Institute for Biotechnology<br>www.hudsonalpha.org<br><br><br>-----Original Message-----<br>From: Cornelissen, Frans [PRDBE] [mailto:FCORNELI@its.jnj.com] <br>Sent: Wednesday, December 10, 2008 2:11 AM<br>To: Ghislain Bonamy; ome-users@lists.openmicroscopy.org.uk<br>Subject: RE: ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression<br>and duplicate storage (Ghislain Bonamy)<br><br>Ghislain,<br><br>We are currently using the MCT (multi component trasnsform) version of<br>JPEG 2000 to compress 3D volumes.<br>We use the SDK from AWARE<br>(http://www.aware.com/imaging/digitalarchives.htm) to do this.<br>There also is a refererence implementation of the "real" 3D jpeg 2000<br>compression already (X,Y,Z and C(eg RGB) compressed in one "volume"<br>file). This will be available within a number of months...<br><br>Regards, frans<br><br>-----Original Message-----<br>From: Ghislain Bonamy [mailto:GBonamy@gnf.org] <br>Sent: Sunday, 7 December 2008 11:04 PM<br>To: Cornelissen, Frans [PRDBE]; ome-users@lists.openmicroscopy.org.uk<br>Subject: RE: ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression<br>and duplicate storage (Ghislain Bonamy)<br><br>Frans,<br><br>Thanks for the answer. Like you I agree that JPEG2000, offers state of<br>the art compression when it comes to images. This is why I have<br>implemented JPEG2000 compression in bioformats for the TIFF and the OME<br>files (readers and writers). This should hopefully be released soon. I<br>am also trying to implement a solution to allow for variable rate<br>compression. For instance images that are on a 16bit scale but only use<br>the first 8 bit, should be saved on 1byte and not 2, this would also<br>increase compression by a factor of 2. I also would like to generalize<br>this so that images that use 12bit of data be compressed as 12 bit<br>images and not 16 (if this is possible, this would certainly save a lot<br>of space this would be particularly useful in lossy compression).<br>Finally I intend to implement lossy Jpeg2000 compression in this<br>variable bit rate format, which should allow 10 fold compression,<br>without many artifacts, in particular for images that are dim.<br><br>I would be most interested to hear how you compress your 3D data! Do you<br>actually compress 1 slice at a time, or do you somehow use compression<br>like in JPEG motion where info from multiple slice are pulled together<br>to improve the overall compression. If so, how could this be implemented<br>for multi stack tiffs etc. and build into bioformats?<br><br>For the storage solution am not quite sure how Castor operates nor how<br>expensive it is, but I prompted our IT guys to have a look at it. This<br>could ultimately be the solution of choice!<br><br>Best,<br><br> Ghislain Bonamy, PhD <br>_______________________________<br>Genomic Institute of the<br>Novartis Research<br>Foundation<br>Functional Genomics, G214<br>+1 (858) 812-1534 (W & F)<br>+1 (858) 354-7388 (C)<br>www.gnf.org <http://www.gnf.org/> <br><br>Hudson Alfa Instiute for Biotechnology<br>www.haib.org <http://www.haib.org/> <br><http://www.gnf.org/> <br><br>________________________________<br><br>From: Cornelissen, Frans [PRDBE] [mailto:FCORNELI@its.jnj.com]<br>Sent: Sat 12/6/2008 4:55 AM<br>To: ome-users@lists.openmicroscopy.org.uk; Ghislain Bonamy<br>Subject: RE: ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression<br>and duplicate storage (Ghislain Bonamy)<br><br><br><br><br> 4. OMERO compression and duplicate storage (Ghislain Bonamy)<br><br>----------------------------------------------------------------------<br><br>Ghislain,<br><br><br>About: "store lossy images on disk":<br><br>We are currently using Jpeg 2000 compression for this.<br>The advantage of using this would be that you do not even need to store<br>2 versions of your image files; JP2 allow sto read any size or any<br>quality<br>image version out of a single (lossless or lossy) compressed file.<br><br>JP2 is currently THE state of the art way to get maximum flexibility AND<br>Maximum compression ratio; for 2D Biological images lossless, factors of<br>2.5 to 4 are achievable (depending on the actual image content)<br>In 3D, quasi lossless compression (PSNR >45) of 20-100 times is possible<br><br><br>About:"slow tape system to store files"<br><br>you could have a look at CASTOR from CARINGO (www.caringo.com), a<br>simple,cheap, but very efficient next-gen Software system for<br>distributed storage on heterogenous hardware, with very good<br>performance.<br>Scales very well to > 80 PB.<br>We have been testing it to store JPEG 2000 images for 18 months, worked<br>withour a flaw!<br>Also in use at the The Center of Inherited Disease Research (CIDR) at<br>Johns Hopkins University<br><br><br><br>Message: 4<br>Date: Fri, 5 Dec 2008 10:10:54 -0800<br>From: "Ghislain Bonamy" <GBonamy@gnf.org><br>Subject: [ome-users] OMERO compression and duplicate storage<br>To: <ome-users@lists.openmicroscopy.org.uk><br>Message-ID:<br> <F5A26DAD36F60843830631774C95CAE205807A58@EXCH2.rec.gnf.org><br>Content-Type: text/plain; charset="us-ascii"<br><br>Dear all,<br><br>I was wondering if OMERO has already implement, is thinking or would<br>consider implementing a mechanism to store images into two different<br>format while keeping there metadata linked.<br><br>I am working in a center were we are generating several TB of data a<br>month and where keeping all of our images on disk becomes impossible. To<br>remedy to this, we have a >2 PT tape storage solution, which is however<br>very slow (takes about 2-4 Minutes to retrieve a file the first time<br>until it is de-cued) . My idea would be to store lossy images on disk<br>for people to view and modify metadata, while keeping the full blown<br>image onto our slow storage solution in case they needed to be<br>reanalyzed for instance. The metadata for this images would however be<br>kept identical between images.<br><br>Perhaps, a way to do this would be to store in Omero the file and keep<br>in the metadata a link to the original image. If there are another and<br>better solution please let me know.<br><br>In addition, does OMERO store the metadata in a compressed way (as well<br>as the images), or is there a way to have OMERO apply a script (for<br>instance a gzip compression) when importing images and when an image is<br>queried?<br><br>Thanks a bunch for all your help and answers,<br><br>Best,<br><br>Ghislain Bonamy, PhD<br>__________________________________________<br><br>Research Investigator I<br><br>Genomic Institute of the<br><br>Novartis Research<br><br>Foundation<br><br>Department of Molecular & Cell Biology, room G214<br><br>10675 John Jay Hopkins Drive<br><br>San Diego CA 92121<br><br>USA<br><br><br>+1 (858) 812-1534 (W & F)<br><br>+1 (757) 941-4194 (H)<br><br>+1 (858) 354-7388 (M)<br><br>www.gnf.org<br><br><br>Hudson-Alpha Institute for Biotechnology<br><br>www.hudsonalpha.org<br><br><br>_______________________________________________<br>ome-users mailing list<br>ome-users@lists.openmicroscopy.org.uk<br>http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br><br><br>End of ome-users Digest, Vol 45, Issue 9<br>****************************************<br><br><br><br><br><br>_______________________________________________<br>ome-users mailing list<br>ome-users@lists.openmicroscopy.org.uk<br>http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br>_______________________________________________<br>ome-users mailing list<br>ome-users@lists.openmicroscopy.org.uk<br>http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br></div></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="font-family: Helvetica; "><br class="Apple-interchange-newline"><br class="khtml-block-placeholder"></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">**************************</span></div><div style="font-family: Helvetica; "><font class="Apple-style-span" face="Arial">Wellcome Trust Centre for Gene Regulation & Expression</font></div><div style="font-family: Helvetica; "><font class="Apple-style-span" face="Arial">College of Life Sciences</font></div><div style="font-family: Helvetica; "><font class="Apple-style-span" face="Arial">MSI/WTB/JBC Complex</font></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">University of Dundee</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">Dow Street</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">Dundee DD1 5EH</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">United Kingdom</span></div><div style="font-family: Helvetica; "><br style="font-family: Helvetica; "></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">phone (01382) 385819</span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">Intl phone: 44 1382 385819 </span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">FAX (01382) 388072 </span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">email: <a href="mailto:jason@lifesci.dundee.ac.uk">jason@lifesci.dundee.ac.uk</a></span></div><div style="font-family: Helvetica; "><br style="font-family: Helvetica; "></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">Lab Page: <a href="http://www.dundee.ac.uk/lifesciences/swedlow/">http://www.dundee.ac.uk/lifesciences/swedlow/</a></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">Open Microscopy Environment: <a href="http://openmicroscopy.org">http://openmicroscopy.org</a></span></div><div style="font-family: Helvetica; "><span class="Apple-style-span" style="font-family: Helvetica; ">**************************</span></div><div style="font-family: Helvetica; "><br class="webkit-block-placeholder"></div><div style="font-family: Helvetica; "><div>The University of Dundee is a Scottish Registered Charity, No. SC015096.</div></div><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline"> </div><br></div></body></html>