[ome-devel] bioformats performance over network shares

[Melissa Linkert]

Hi Rubén,

> You probably know that using bioformats over network shares has much worst performance than over harddrives. Has nothing to do with the generation of additional metadata but rather with the Random Access methods used for writing files. 

Yes, of course.  Writing randomly to a slower drive is naturally going
to be much slower than writing randomly to a faster drive.

> Large datasets can take days to convert and we introduced some changes that speed up our conversions two times. From 15 minutes to 6 minutes.
> I just used a temporal file then moved the output to the target. 
> 
> This probably does not suit everybody or can be improved. 
> Please let me know if it is useful. 

In your case, these changes probably work, but I do not think it is the best way
of solving the problem in general.  Imagine what happens in the fairly
common case where someone is converting a file with the output file on a
local disk - there is now the extra overhead of copying the file,
without any performance gain from writing to a temporary file first.

On Linux/Mac OS X at least, this is a simple problem to solve without
even modifying ImageConverter.  I think the following commands would
accomplish the same thing as your modifications:

$ export OUTPUT_FILE=`mktemp -u`.tif
$ bfconvert /path/to/input/file $OUTPUT_FILE
$ mv $OUTPUT_FILE /path/to/file/on/network/share

Regards,
-Melissa

On Fri, Jun 17, 2011 at 11:14:28AM +0200, Rubén Muñoz wrote:
> Hi Melissa, how are you?
> 
> You probably know that using bioformats over network shares has much worst performance than over harddrives. Has nothing to do with the generation of additional metadata but rather with the Random Access methods used for writing files. 
> 
> Large datasets can take days to convert and we introduced some changes that speed up our conversions two times. From 15 minutes to 6 minutes.
> I just used a temporal file then moved the output to the target. 
> 
> This probably does not suit everybody or can be improved. 
> Please let me know if it is useful. 
> 
> Best regards, 
> 
> Rubén
> 
> File tmpfile = File.createTempFile("image",".tif");
> 
> @@ -311,7 +316,10 @@ public final class ImageConverter {
>        total += numImages;
>  
>        for (int i=startPlane; i<endPlane; i++) {
> -        writer.setId(FormatTools.getFilename(q, i, reader, out));
> +       
> +        //writer.setId(FormatTools.getFilename(q, i, reader, out));
> +        File tmpfile = File.createTempFile("image",".tif");
> +        writer.setId(tmpfile.getAbsolutePath());
>          if (compression != null) writer.setCompression(compression);
>  
>          long s = System.currentTimeMillis();
> @@ -324,6 +332,10 @@ public final class ImageConverter {
>          }
>          long m = System.currentTimeMillis();
>          writer.saveBytes(i - startPlane, buf);
> +        //LOGGER.info("rename: " + tmpfile + " " + (new File(FormatTools.getFilename(q, i, reader, out))).getAbsolutePath());
> +        (new File(FormatTools.getFilename(q, i, reader, out))).getParentFile().mkdirs();
> +        copyFile(tmpfile,new File(FormatTools.getFilename(q, i, reader, out)));
> +        tmpfile.delete();
>          long e = System.currentTimeMillis();
>          read += m - s;
>          write += e - m;
> 
> 
> +  public static void copyFile(File source, File dest) throws IOException {
> +       
> +               if(!dest.exists()) {
> +                       dest.createNewFile();
> +               }
> +        InputStream in = null;
> +        OutputStream out = null;
> +        try {
> +               in = new FileInputStream(source);
> +               out = new FileOutputStream(dest);
> +    
> +               // Transfer bytes from in to out
> +               byte[] buf = new byte[1024];
> +               int len;
> +               while ((len = in.read(buf)) > 0) {
> +                   out.write(buf, 0, len);
> +               }
> +        }
> +        finally {
> +               in.close();
> +            out.close();
> +        }
> +        
> +  }
> 
> 
> 
> 
>