[ome-devel] bioformats performance over network shares

[Rubén Muñoz]

Hi Melissa, 

On Jun 20, 2011, at 4:07 PM, Melissa Linkert wrote:

> Hi Rubén,
> 
>> You probably know that using bioformats over network shares has much worst performance than over harddrives. Has nothing to do with the generation of additional metadata but rather with the Random Access methods used for writing files. 
> 
> Yes, of course.  Writing randomly to a slower drive is naturally going
> to be much slower than writing randomly to a faster drive.

Right. Yes, that's natural. Is there an advantage of using random access methods to write TIF files, instead of the FileOutputStream?

> 
>> Large datasets can take days to convert and we introduced some changes that speed up our conversions two times. From 15 minutes to 6 minutes.
>> I just used a temporal file then moved the output to the target. 
>> 
>> This probably does not suit everybody or can be improved. 
>> Please let me know if it is useful. 
> 
> In your case, these changes probably work, but I do not think it is the best way
> of solving the problem in general.  Imagine what happens in the fairly
> common case where someone is converting a file with the output file on a
> local disk - there is now the extra overhead of copying the file,
> without any performance gain from writing to a temporary file first.
> 
> On Linux/Mac OS X at least, this is a simple problem to solve without
> even modifying ImageConverter.  I think the following commands would
> accomplish the same thing as your modifications:
> 
> $ export OUTPUT_FILE=`mktemp -u`.tif
> $ bfconvert /path/to/input/file $OUTPUT_FILE
> $ mv $OUTPUT_FILE /path/to/file/on/network/share

I have omitted that we convert to multi-file OME.TIF datasets.  These datasets are bigger than the harddisk capacity and therefore we needed to modify the code. 

Best,

Rubén

> 
> Regards,
> -Melissa
> 
> On Fri, Jun 17, 2011 at 11:14:28AM +0200, Rubén Muñoz wrote:
>> Hi Melissa, how are you?
>> 
>> You probably know that using bioformats over network shares has much worst performance than over harddrives. Has nothing to do with the generation of additional metadata but rather with the Random Access methods used for writing files. 
>> 
>> Large datasets can take days to convert and we introduced some changes that speed up our conversions two times. From 15 minutes to 6 minutes.
>> I just used a temporal file then moved the output to the target. 
>> 
>> This probably does not suit everybody or can be improved. 
>> Please let me know if it is useful. 
>> 
>> Best regards, 
>> 
>> Rubén
>> 
>> File tmpfile = File.createTempFile("image",".tif");
>> 
>> @@ -311,7 +316,10 @@ public final class ImageConverter {
>>       total += numImages;
>> 
>>       for (int i=startPlane; i<endPlane; i++) {
>> -        writer.setId(FormatTools.getFilename(q, i, reader, out));
>> +       
>> +        //writer.setId(FormatTools.getFilename(q, i, reader, out));
>> +        File tmpfile = File.createTempFile("image",".tif");
>> +        writer.setId(tmpfile.getAbsolutePath());
>>         if (compression != null) writer.setCompression(compression);
>> 
>>         long s = System.currentTimeMillis();
>> @@ -324,6 +332,10 @@ public final class ImageConverter {
>>         }
>>         long m = System.currentTimeMillis();
>>         writer.saveBytes(i - startPlane, buf);
>> +        //LOGGER.info("rename: " + tmpfile + " " + (new File(FormatTools.getFilename(q, i, reader, out))).getAbsolutePath());
>> +        (new File(FormatTools.getFilename(q, i, reader, out))).getParentFile().mkdirs();
>> +        copyFile(tmpfile,new File(FormatTools.getFilename(q, i, reader, out)));
>> +        tmpfile.delete();
>>         long e = System.currentTimeMillis();
>>         read += m - s;
>>         write += e - m;
>> 
>> 
>> +  public static void copyFile(File source, File dest) throws IOException {
>> +       
>> +               if(!dest.exists()) {
>> +                       dest.createNewFile();
>> +               }
>> +        InputStream in = null;
>> +        OutputStream out = null;
>> +        try {
>> +               in = new FileInputStream(source);
>> +               out = new FileOutputStream(dest);
>> +    
>> +               // Transfer bytes from in to out
>> +               byte[] buf = new byte[1024];
>> +               int len;
>> +               while ((len = in.read(buf)) > 0) {
>> +                   out.write(buf, 0, len);
>> +               }
>> +        }
>> +        finally {
>> +               in.close();
>> +            out.close();
>> +        }
>> +        
>> +  }
>> 
>> 
>> 
>> 
>>