[ome-users] OME and offline archiving

[Ilya Goldberg]

As regards to 2.6.1, the OME_2_6_0 branch on CVS has been getting bug  
fixes and various sprucing up.  Its basically in a releasable state -  
we've been using in in deployment for some time.
It doesn't cover the OpenLab fix though - this is a BioFormats issue  
and up to Curtis.

Do we want to wait on BioFormats before releasing 2.6.1, or should we  
go ahead and release now without it?
-Ilya

On Jul 16, 2007, at 1:34 PM, Mario Valle wrote:

> Thanks Ilya, Ian and Tom for your insights!
> I agree that disk are cheap and yes, we already have a 2TB scratch  
> space
> online. But it is "use at your own risk" space and this leaves a
> "little" worried the researchers. Maybe it is simpler to design an
> efficient backup strategy for OMEIS and archive those backups.
> Anyway, seems a way or another this problem could be solved. Now the
> most important thing is to start (BTW, any news about 2.6.1?).
> Thanks again!
>                                mario
>
> Ilya Goldberg wrote:
>> I would totally agree with Ian.  We recently bought 10 750 GB SATA
>> "ES" drives from Seagate for $265 each (7.5 tb for $2650).  Of course
>> these were made into 2 4x750 raid 5s with two spares, so we "only"
>> got 4 tb of formatted space out of that, but a fair amount of
>> redundancy and performance for less than $3k.  You can put your 9tb
>> live at all times with decent redundancy and performance for under
>> $10k including the CPUs, cabinets, etc.  If you build it as you need
>> it instead of all at once, it will be much cheaper still.
>>
>> Of course, there are still people who insist on SCSI or fiberchannel
>> disks, and don't mind paying 10x the cost.  Honestly, I think that
>> SATA disks are so cheap, and software raid so efficient, that scsi
>> and fiberchannel no longer justify their costs.  Its much more cost-
>> effective to design a storage solution around a 10% annual disk
>> failure rate than to pay 10x the cost for a marginal improvement in
>> reliability (see the Google study on disk failure), and a doubtful
>> improvement on end-to-end performance and throughput.  Of course,
>> others in the "enterprise class" data storage arena would disagree,
>> but my feeling is that its only a matter of time before even they see
>> the light.  Some already have...
>>
>> As Tom mentioned, if you really do want to get more complicated with
>> the hardware, the software is "almost there" in terms of letting you
>> do it this way - basically because its already set up to take
>> intermediate steps to prepare a file to properly service a request.
>>
>> -Ilya
>>
>>
>> On Jul 16, 2007, at 11:24 AM, Ian Dobbie wrote:
>>
>>
>>> Mario Valle <mvalle at cscs.ch> writes:
>>>
>>>
>>>> The project for which I'm evaluating OME plans to produce 9 TB
>>>> (uncompressed) of images during its lifetime.
>>>>
>>>> Obviously I cannot leave all the images on-line. The idea is to  
>>>> have
>>>> one or two image stacks online and put the rest on our tape
>>>> archive. Obviously all metadata continue to be online.
>>>>
>>> I don't see how leaving all the images online is going to be a  
>>> serious
>>> problem. I would not be surprised if buying disks for this size of
>>> storage is cheaper than tapes. A 750GB disk now costs less than 300
>>> euro, so with a bit of redundancy 9TB will be less than 20 disks, so
>>> less than 6,000 euro.
>>>
>>> The other thing to consider is what kind of time scale are your
>>> talking over. If this is, say, a 5 year project with the first 3  
>>> years
>>> being getting equipment, workflow etc working. Then by the time you
>>> are collecting serious data a 2TB disk will probably only be 300-500
>>> euro. How expensive is the tape going to be? How much hassle will  
>>> the
>>> off-line storage be?
>>>
>>> If it was my setup I would build an image server, put the pixel data
>>> there and stuff it with disks. Keep the tapes for actual backups not
>>> for data that might be in use.
>>>
>>> Ian
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>
>
> -- 
> Ing. Mario Valle
> Visualization Group                              | http:// 
> www.cscs.ch/~mvalle
> Swiss National Supercomputing Centre (CSCS)      | Tel:  +41 (91)  
> 610.82.60
> v. Cantonale Galleria 2, 6928 Manno, Switzerland | Fax:  +41 (91)  
> 610.82.82
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>