[ome-users] OME and offline archiving

[Tom Macura]

Hello Mario,

> The project for which I'm evaluating OME plans to produce 9 TB  
> (uncompressed) of images during its lifetime.
Nice!

We (Ilya Goldberg's group at the NIH) are, obviously, also using OME  
to manage our images.

We are currently using 300 GB (compressed) for pixels and 600MB for  
the data-base (that's 25,000 images). We are applying computer vision  
algorithms (WND-CHARM: http://users.openmicroscopy.org.uk/~tmacur1/ 
OME-WEBSITE/howto/wnd-charm-ome.html) to analyze our images and  
storing the results  (1025 measurements per image) in OME. Therefore  
your meta-data requirements will be much lower, I'd say 1GB of pixels  
to 1MB of meta-data is standard.

Have you look at this web-page ?: http://users.openmicroscopy.org.uk/ 
~tmacur1/OME-WEBSITE/api/omeis/disk-usage.html

OME has two features that will be relevant for you: (1) multiple  
repositories per data-server, (2) transparent compression/ 
decompression of files.

Initially, we were storing the lab images on a server that only had  
200GB of free-space. When free-space dwindled down to 15GB, we  
compressed all our pixels with bzip2. I vaguely recalled that  
compression cut our disk usage by 33%. Something like 75% of our  
pixels are computational outputs (e.g. Fourier Transforms, Gabor  
Wavelets) so I don't have experience  with compressing raw pixels but  
I think you'll get better compression.

After a while, even with compression, we were using all the free- 
space on our server so we installed OMEIS on another computer (that  
had another 200GB of free-space) and made it an additional pixels  
repository for our original data-server. This is an easy/transparent  
step. (ome admin configure) Later we migrated our original server  
onto new hard-ware that had a lot more free space (2TB). That  
migration step was as easy as doing (ome data backup --  scp tar  
file  -- ome data restore).

Regarding your comments about offline-archiving. We haven't done it  
ourselves but OMEIS already does nearly all of the graceful signaling  
when it auto inflates compressed .bzip/.gz files. So a background  
process will run and convert pixels e.g. 123 into a small file  
123.offline at scheduled intervals using a heuristic (e.g. file last- 
accessed stamp). Then you will need to provide a utility that would  
convert 123.offline back into 123 upon demand using your tape archive  
(this is your problem....). It's very easy to patch OMEIS to support  
the .offline format along with the way it currently supports .bzip  
files. I could even do that for you.

Fire away if you'd like more clarifications. I personally (not  
official opinion of NIH/US Gov), think 9TB is not too much for OME.

Best,

Tom

>
> Obviously I cannot leave all the images on-line. The idea is to  
> have one or two image
> stacks online and put the rest on our tape archive. Obviously all  
> metadata continue to be
> online.
>
> This could work if I put the /OME/OMEIS directory on the tape  
> archive buffer disk. But
> this leave me without control of the migration of the data (not to  
> mention the diplomacy
> needed with the archive system manager...).
>
> Another solution could be to manage the migration myself (using  
> sftp for example) and hope
> OME gracefully signal when some image data is not available inside  
> OMEIS. If this happens
> I could retrieve the image from the archive and restart the failed  
> operation.
>
> Do you have experience to offer about those archival issues?
>
> Better those issues arise before the project starts...
>
> Thanks for your help!
> 				mario
>
> -- 
> Ing. Mario Valle
> Visualization Group                              | http:// 
> www.cscs.ch/~mvalle
> Swiss National Supercomputing Centre (CSCS)      | Tel:  +41 (91)  
> 610.82.60
> v. Cantonale Galleria 2, 6928 Manno, Switzerland | Fax:  +41 (91)  
> 610.82.82
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users