[ome-devel] tif import performance

[Richard Beare]

Hi,
I'm certainly not intending to be critical! I think that the amount of 
work that has gone into the project so far is huge and the results so 
far are very impressive. One of the indications of how much has been 
done is how much time is needed by a new comer to understand what is 
already in place and where contributions can best be made :-).

Anyway, I'll run a benchmark on an import process to give you an idea of 
the sorts of time I'm seeing - I could easily be doing something wrong.

I'll send results of this ASAP....

Jason Swedlow wrote:
> Richard-
> 
> The problem is that for many TIFF based files, ZWT (focus, channel, 
> timepoint) or other info (e.g., plate position, in a screen) is encoded 
> in the file name, and not in the TIFF tags, Note this is not OME, but 
> the file formats we are provided by the commercial vendors. Anyway, at 
> least some of the basic info OME needs for import is accessed during 
> this process.
> 
> As for performance issues (your previous) email, you can certainly 
> criticize the approach we have implemented. There are historical reasons 
> for this, that aren't really worth going into. But OME import will 
> always be much slower than a simple copy, because a large number of 
> calcs are performed, and then written to the database. This does take 
> real time. Our general sense of a use case is that data is acquired, 
> imported, and then analysed. So far we have not worked on a system where 
> the time from beginning an import to actually seeing the data and/or 
> analysing it is critical. In many cases, we expect that an import of 
> many Gbytes of data can take minutes, depending on the type of data. If 
> you let us know your requirements -- how many TIFFs, how big, etc., we 
> can discuss the limitations and see what we can do.
> 
> Finally, you should note that OME is open source but not because our 
> primary goal is to provide a service to the greater community. We are 
> funded by grants to do our own work, and we are developing OME to 
> support those purposes. We are currently making the transition from a 
> proof-of-concept project to a production project, but we aren't there 
> yet. Our latest difficulties in installs are an example of that. You 
> will doubtless find all sorts of problems in our system, where our own 
> requirements do not match or anticipate your own. We welcome your 
> commentary, but as we are still growing and supporting our own work, we 
> don't have resources to fix the things you might want, at least right 
> away (we do endeavor to fix system bugs ASAP). However, we welcome your, 
> and others, help, and will support that as much as possible. Our group 
> of developers keeps growing, largely through people who need the tools 
> we are working on, and have the skills to help.
> 
> The problem we are trying to address is huge. We have made alot of 
> progress, but still have a long way to go. If you can, please do help us 
> out-- we'd love it.
> 

-- 
Richard Beare, CSIRO Mathematical & Information Sciences
Locked Bag 17, North Ryde, NSW 1670, Australia
Phone: +61-2-93253221 (GMT+~10hrs)  Fax: +61-2-93253200

Richard.Beare at csiro.au