[ome-users] ome-users Digest, Vol 45, Issue 9 - -4. OMERO compression and duplicate storage (Ghislain Bonamy)

[Jason Swedlow]

Dear All-

Just a heads up.

We have a similar approach here at Dundee, using a hierarchical  
storage system based on Tivoli Storage Manager.  Direct support for  
these kind of systems is not hard in principle, but what is a major  
problem is supporting everyone's own version.  It is unlikely we will  
be able to provide support for specific commercial installations in  
the near future.

However, some quick points:

Have a look at:

http://trac.openmicroscopy.org.uk/omero/wiki/ServerDesign

There is a lot there, but you might want to poke around the Rendering  
and Thumbnailing sections.  We don't pretend that we have everything  
done that one might need, but there is alot done already-- and it's  
fast.

Moreover, the general problem you highlight is one we are actively  
addressing-- OMERO.fs.  This is in it's early days, but we have a  
prototype working and those of you coming to ASCB in San Francisco  
next week will get to see it. The general problem of multiple  
repositories is a critical one that we are working on currently.  The  
challenge is dealing with the diversity of installations, but we hope  
to provide a framework that can support a variety of installations.   
We won't handle hierarchical systems in the first release (due in  
Q1/09), but you can see where this will go over the next year.

Cheers,

Jason

On 6 Dec 2008, at 12:55, Cornelissen, Frans [PRDBE] wrote:

>    4. OMERO compression and duplicate storage (Ghislain Bonamy)
>
> ----------------------------------------------------------------------
>
> Ghislain,
>
>
> About: "store lossy images on disk":
>
> We are currently using Jpeg 2000 compression for this.
> The advantage of using this would be that you do not even need to  
> store
> 2 versions of your image files; JP2 allow sto read any size or any
> quality
> image version out of a single (lossless or lossy) compressed file.
>
> JP2 is currently THE state of the art way to get maximum flexibility  
> AND
> Maximum compression ratio; for 2D Biological images lossless,  
> factors of
> 2.5 to 4 are achievable (depending on the actual image content)
> In 3D, quasi lossless compression (PSNR >45) of 20-100 times is  
> possible
>
>
> About:"slow tape system to store files"
>
> you could have a look at CASTOR from CARINGO (www.caringo.com), a
> simple,cheap, but very efficient next-gen Software system for
> distributed storage on heterogenous hardware, with very good
> performance.
> Scales very well to > 80 PB.
> We have been testing it to store JPEG 2000 images for 18 months,  
> worked
> withour a flaw!
> Also in use at the The Center of Inherited Disease Research (CIDR) at
> Johns Hopkins University
>
>
>
> Message: 4
> Date: Fri, 5 Dec 2008 10:10:54 -0800
> From: "Ghislain Bonamy" <GBonamy at gnf.org>
> Subject: [ome-users] OMERO compression and duplicate storage
> To: <ome-users at lists.openmicroscopy.org.uk>
> Message-ID:
> 	<F5A26DAD36F60843830631774C95CAE205807A58 at EXCH2.rec.gnf.org>
> Content-Type: text/plain; charset="us-ascii"
>
> Dear all,
>
> I was wondering if OMERO has already implement, is thinking or would
> consider implementing a mechanism to store images into two different
> format while keeping there metadata linked.
>
> I am working in a center were we are generating several TB of data a
> month and where keeping all of our images on disk becomes  
> impossible. To
> remedy to this, we have a >2 PT tape storage solution, which is  
> however
> very slow (takes about 2-4 Minutes to retrieve a file the first time
> until it is de-cued) . My idea would be to store lossy images on disk
> for people to view and modify metadata, while keeping the full blown
> image onto our slow storage solution in case they needed to be
> reanalyzed for instance. The metadata for this images would however be
> kept identical between images.
>
> Perhaps, a way to do this would be to store in Omero the file and keep
> in the metadata a link to the original image. If there are another and
> better solution please let me know.
>
> In addition, does OMERO store the metadata in a compressed way (as  
> well
> as the images), or is there a way to have OMERO apply a script (for
> instance a gzip compression) when importing images and when an image  
> is
> queried?
>
> Thanks a bunch for all your help and answers,
>
> Best,
>
> Ghislain Bonamy, PhD
> __________________________________________
>
> Research Investigator I
>
> Genomic Institute of the
>
> Novartis Research
>
> Foundation
>
> Department of Molecular & Cell Biology, room G214
>
> 10675 John Jay Hopkins Drive
>
> San Diego CA 92121
>
> USA
>
>
> +1 (858) 812-1534 (W & F)
>
> +1 (757) 941-4194 (H)
>
> +1 (858) 354-7388 (M)
>
> www.gnf.org
>
>
> Hudson-Alpha Institute for Biotechnology
>
> www.hudsonalpha.org
>
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
> End of ome-users Digest, Vol 45, Issue 9
> ****************************************
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users



**************************
Wellcome Trust Centre for Gene Regulation & Expression
College of Life Sciences
MSI/WTB/JBC Complex
University of Dundee
Dow Street
Dundee  DD1 5EH
United Kingdom

phone (01382) 385819
Intl phone:  44 1382 385819
FAX   (01382) 388072
email: jason at lifesci.dundee.ac.uk

Lab Page: http://www.dundee.ac.uk/lifesciences/swedlow/
Open Microscopy Environment: http://openmicroscopy.org
**************************

The University of Dundee is a Scottish Registered Charity, No. SC015096.