[ome-devel] bioformats performance over network shares

[Melissa Linkert]

Hi,

> Melissa, any thoughts on how difficult it would be to make StreamHandle into
> a concrete class, with a constructor that accepts a DataInputStream as
> argument, and one that accepts a DataOutputStream?

It's relatively easy to make an implementation of StreamHandle that can
accept a DataInputStream and DataOutputStream in its constructor.
However, I am not positive that this will fully solve the problem, as there
is the potential for substantial overhead associated with closing and
re-opening the streams as needed which may outweigh the benefits of
using a BufferedOutputStream.

I have filed a ticket on our issue tracking system to investigate this
problem more thoroughly, and come up with a good general purpose solution:

http://trac.openmicroscopy.org.uk/ome/ticket/5965

I have CC'd Rubén on this ticket already, but if anyone else would like
to be CC'd please let me know.

Please note that this is not scheduled for the next release (4.3.1), but
for the following one (4.3.2).  Comments or ideas on how to solve
this are welcome at any time, but I will not have time to work on this until
after the 4.3.1 release is announced.

Regards,
-Melissa

On Wed, Jun 22, 2011 at 01:11:02PM -0500, Curtis Rueden wrote:
> Hi everyone,
> 
> Bio-Formats has classes known as RandomAccessInputStream and
> RandomAccessOutputStream, which are intended to allow random access behavior
> in a variety of contexts, including streams (with the StreamHandle class).
> All of this logic lives in the loci.common package of loci-common.jar.
> Unfortunately, StreamHandle is an abstract class and there is not a
> general-purpose concrete implementation for streams. If there were, we could
> try using a StreamHandle with a BufferedOutputStream of sufficient buffer
> size, and see if it made any difference for Rubén.
> 
> Melissa, any thoughts on how difficult it would be to make StreamHandle into
> a concrete class, with a constructor that accepts a DataInputStream as
> argument, and one that accepts a DataOutputStream?
> 
> -Curtis
> 
> On Mon, Jun 20, 2011 at 2:37 PM, Gregory Jefferis <
> jefferis at mrc-lmb.cam.ac.uk> wrote:
> 
> > **
> > On 2011-06-20 18:02, Johannes Schindelin wrote:
> >
> > Maybe the random access could be buffered with a configurable window size?
> > That could improve performance and be a more general solution..
> >
> >  This seems like a good idea. For comparison I remember a case for a Fiji
> > plugin I wrote (NrrdReader) that when reading compressed files from network
> > drives (SMB) adding a fairly small buffer (50k) could result in a several
> > fold increase in read speed.
> >
> > Also was curious about the ratio of bulk write speed to OME tiff write
> > speed in OP (Ruben's) case.
> >
> > Best,
> >
> > Greg.
> >
> >
> > --
> > Gregory Jefferis, PhD                      jefferis at mrc-lmb.cam.ac.uk
> > Division of Neurobiology                   LMB Lab:   +44 (0)1223 252943
> > MRC Laboratory of Molecular Biology,       LMB Office:+44 (0)1223 252944
> > Hills Road,                                LMB Fax:   +44 (0)1223 402310
> > Cambridge, CB2 0QH, UK.
> >
> > http://www2.mrc-lmb.cam.ac.uk/group-leaders/h-to-m/g-jefferis
> > http://www.neuroscience.cam.ac.uk/directory/profile.php?gsxej2
> > http://flybrain.stanford.edu
> >
> >
> > _______________________________________________
> > ome-devel mailing list
> > ome-devel at lists.openmicroscopy.org.uk
> > http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
> >
> >