[ome-devel] Bug 274 : Incremental Import

[Joshua Moore]

Jason Swedlow wrote:

> Can you define use case-- specifically, what is the amount of data at 
> the point the user wants to evaluate progress, and what is the final 
> amount of data. We will need to be convinced that a simple visual 
> client running locally at the evaluation point is insufficient for 
> this to convince us to substantially change OME design and principles. 

Once we are up an runnning a single microscope run will be imaging 2000 
wells every 10 minutes for 30 hours. Each single-channel image is 1Mb 
for a run of roughly 120Gb.

There will be two parallel systems running roughly twice a week for 
roughly .5Tb per week.

The benefit of making this possible is that, at least in LabView, a GUI 
could be produced which keeps track of which wells are good, and keeps a 
current view by calling Composite();


> I understand the importance of saving (and collecting) only useful 
> data. But OME is fundamentally not a filesystem. It is a data 
> warehouse. There are very good reasons not restrict data deletion.. 
> However, we need to be able fix botched imports ("Oooops, I just 
> imported the whole filesystem"). Note that there are many legal 
> requirements that argue against deletion of primary electronic data 
> (http://www.fda.gov/ora/compliance_ref/part11/).
>
> Anyway, give us some details and let us work this out. We have 
> discussed this at length for the last 3 years and have various ideas 
> we are thinking of implementing. Any comments from other sites?
>
> Cheers,
>
> Jason
>