[ome-users] OME and offline archiving

[Mario Valle]

Thanks Ilya, Ian and Tom for your insights!
I agree that disk are cheap and yes, we already have a 2TB scratch space 
online. But it is "use at your own risk" space and this leaves a 
"little" worried the researchers. Maybe it is simpler to design an 
efficient backup strategy for OMEIS and archive those backups.
Anyway, seems a way or another this problem could be solved. Now the 
most important thing is to start (BTW, any news about 2.6.1?).
Thanks again!
                               mario

Ilya Goldberg wrote:
> I would totally agree with Ian.  We recently bought 10 750 GB SATA  
> "ES" drives from Seagate for $265 each (7.5 tb for $2650).  Of course  
> these were made into 2 4x750 raid 5s with two spares, so we "only"  
> got 4 tb of formatted space out of that, but a fair amount of  
> redundancy and performance for less than $3k.  You can put your 9tb  
> live at all times with decent redundancy and performance for under  
> $10k including the CPUs, cabinets, etc.  If you build it as you need  
> it instead of all at once, it will be much cheaper still.
>
> Of course, there are still people who insist on SCSI or fiberchannel  
> disks, and don't mind paying 10x the cost.  Honestly, I think that  
> SATA disks are so cheap, and software raid so efficient, that scsi  
> and fiberchannel no longer justify their costs.  Its much more cost- 
> effective to design a storage solution around a 10% annual disk  
> failure rate than to pay 10x the cost for a marginal improvement in  
> reliability (see the Google study on disk failure), and a doubtful  
> improvement on end-to-end performance and throughput.  Of course,  
> others in the "enterprise class" data storage arena would disagree,  
> but my feeling is that its only a matter of time before even they see  
> the light.  Some already have...
>
> As Tom mentioned, if you really do want to get more complicated with  
> the hardware, the software is "almost there" in terms of letting you  
> do it this way - basically because its already set up to take  
> intermediate steps to prepare a file to properly service a request.
>
> -Ilya
>
>
> On Jul 16, 2007, at 11:24 AM, Ian Dobbie wrote:
>
>   
>> Mario Valle <mvalle at cscs.ch> writes:
>>
>>     
>>> The project for which I'm evaluating OME plans to produce 9 TB
>>> (uncompressed) of images during its lifetime.
>>>
>>> Obviously I cannot leave all the images on-line. The idea is to have
>>> one or two image stacks online and put the rest on our tape
>>> archive. Obviously all metadata continue to be online.
>>>       
>> I don't see how leaving all the images online is going to be a serious
>> problem. I would not be surprised if buying disks for this size of
>> storage is cheaper than tapes. A 750GB disk now costs less than 300
>> euro, so with a bit of redundancy 9TB will be less than 20 disks, so
>> less than 6,000 euro.
>>
>> The other thing to consider is what kind of time scale are your
>> talking over. If this is, say, a 5 year project with the first 3 years
>> being getting equipment, workflow etc working. Then by the time you
>> are collecting serious data a 2TB disk will probably only be 300-500
>> euro. How expensive is the tape going to be? How much hassle will the
>> off-line storage be?
>>
>> If it was my setup I would build an image server, put the pixel data
>> there and stuff it with disks. Keep the tapes for actual backups not
>> for data that might be in use.
>>
>> Ian
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>>     
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>   


-- 
Ing. Mario Valle
Visualization Group                              | http://www.cscs.ch/~mvalle
Swiss National Supercomputing Centre (CSCS)      | Tel:  +41 (91) 610.82.60
v. Cantonale Galleria 2, 6928 Manno, Switzerland | Fax:  +41 (91) 610.82.82