[ome-devel] bioformats performance over network shares

[Johannes Schindelin]

Hi,

On Mon, 20 Jun 2011, Melissa Linkert wrote:

> > You probably know that using bioformats over network shares has much 
> > worst performance than over harddrives. Has nothing to do with the 
> > generation of additional metadata but rather with the Random Access 
> > methods used for writing files.
> 
> Yes, of course.  Writing randomly to a slower drive is naturally going 
> to be much slower than writing randomly to a faster drive.
> 
> > Large datasets can take days to convert and we introduced some changes 
> > that speed up our conversions two times. From 15 minutes to 6 minutes. 
> > I just used a temporal file then moved the output to the target.
> > 
> > This probably does not suit everybody or can be improved. Please let 
> > me know if it is useful.
> 
> In your case, these changes probably work, but I do not think it is the 
> best way of solving the problem in general.  Imagine what happens in the 
> fairly common case where someone is converting a file with the output 
> file on a local disk - there is now the extra overhead of copying the 
> file, without any performance gain from writing to a temporary file 
> first.

Maybe the random access could be buffered with a configurable window size? 
That could improve performance and be a more general solution...

Ciao,
Johannes