[ome-users] ImageProcessorReader.setId performance

[Mohammad Samarah]

Hi Balaji,

Thanks for your insights. We examined the code more closely and through the
debugger. We are passing a single full file path to the
ImageProcesserReader.setId method. The case that fails specifies
/images/001234/R1.RVG while the images folder has more than 4000
sub-folders. We are able to make a workaround by means of copying the same
file to /temp/images/001234/R1.RVG. Temp/images folder contains a small
number of sub-folders. However this is not ideal as the copy takes time to
complete given the size of the images being processed.

You mentioned that the default folder-depth is 4. How do we change it to 1?
Is there an API method to disable this behavior?

Many thanks for your help,

--
M. Samarah
msamarah at alumni.fit.edu


From: Balaji Ramalingam <b.ramalingam at dundee.ac.uk>
Date: Mon, 24 Mar 2014 10:42:02 +0000
To: "msamarah at alumni.fit.edu" <msamarah at alumni.fit.edu>,
"ome-users at lists.openmicroscopy.org.uk"
<ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] ImageProcessorReader.setId performance

Hi,

Your problem seems to be related to the search for Images by
bio-formats(loci-tools), within your target directory (imagePath).
This is how the algorithm works in bio-formats,
1. The target directory(imagePath) is searched for all Œbioformats-readable¹
images. 
2. -This is the very first step and this is where you are facing problems, I
am assuming that you¹re ŒimagePath¹, is a top directory which has images at
various depth¹s. The default folder-depth at which bio-formats searches is 4
(I.e, it might still be searching for 4 sub-level folder¹s within the top
directory, unless specified).
    - The easy alternative is to pre list all the RVG format files within
the ŒimagePath' (using file-system search), and read them one at a time
(iteratively). I.e imagePath == ³Path to an individual image².
       This basically removes the bio-formats intelligence of grouping
similar files (files part of a stack,etc). Am assuming that should not be  a
problem in your case .

2.  The other alternative is to use the ImagePlusReader instead of the
ImageProcessor as suggested in the code, but this needs testing     and
playing around. 

Quoting from the ³ImageProcessorReader.java" code ,
* A low-level reader for {@link ij.process.ImageProcessor} objects.
*  * For a higher-level reader that returns {@link ij.ImagePlus} objects,
*  * see {@link loci.plugins.in.ImagePlusReader} instead.

Please do let us know, how it goes.

Best,
Balaji


From: Mohammad Samarah <isamarah at mac.com>
Reply-To: "msamarah at alumni.fit.edu" <msamarah at alumni.fit.edu>
Date: Saturday, 22 March 2014 03:36
To: "ome-users at lists.openmicroscopy.org.uk"
<ome-users at lists.openmicroscopy.org.uk>
Subject: [ome-users] ImageProcessorReader.setId performance

ImageProcessorReader.setId performance
We are using loci_tools.jar to read radiographs in RVG format and extract a
thumbnail JPEG image. All works as expected, however if the image in
question is in a directory whose parent directory contains several thousands
subfolders, then the ImageProcesserReader takes minutes to complete the
setId() method. Please see attached code. The same image placed in a folder
with its parent directory having much fewer subfolders works with no delay.
This behavior can also be seen when using loci_tools.jar UI to open the same
image. The issue does not seem to be related to the image itself but rather
the fact that the containing folder is a child of a directory with thousands
of subfolders. Could you please point us to how we can resolve this issue.

Many thanks for your help,

--
M. Samarah
msamarah at alumni.fit.edu

 
private ImagePlus readDicomImage(String imagePath)
    {
        ImagePlus imp            = null;
        ImagePlus colorizedImage = null;
        
        ImageProcessorReader r = new ImageProcessorReader(new
ChannelSeparator(LociPrefs.makeImageReader()));
        try
        {
            log_message("INFO: Examining file " + imagePath);

            // the next line takes 10 to 20 minutes to complete when parent
folder contains 4000+ subfolders. It takes no time when parent folder has
100 subfolders
            r.setId(imagePath);

            log_message("INFO: Setting file path " + imagePath);
            int num = r.getImageCount();
            log_message("INFO: Image count is" + num);
            int width = r.getSizeX();
            int height = r.getSizeY();
            log_message("INFO: Image Width %d Height %d", width, height);
            ImageStack stack = new ImageStack(width, height);
            log_message("INFO: After constructing image stack");
            byte[][][] lookupTable = new byte[r.getSizeC()][][];
            log_message("INFO: After Allocating memmory for image");
            for (int i=0; i<num; i++)
            {
                ImageProcessor ip = r.openProcessors(i)[0];
                stack.addSlice("" + (i + 1), ip);
                int channel = r.getZCTCoords(i)[1];
                lookupTable[channel] = r.get8BitLookupTable();
            }
            log_message("INFO: After constructing lookup table");
            log_message("INFO:Constructing image");
            imp = new ImagePlus("Dicom", stack);
               
            colorizedImage = applyLookupTables(r, imp, lookupTable);
            r.close();

        }
        catch(Exception exc)
        {
           log_message("ERROR an error occurred: " + exc.getMessage());
        }
       return colorizedImage;
    }

Example: Takes 10 to 20 minutes on an 8 core machine with 24GB of memory to
complete setId method
> Data <folder>
>> 0001 <folder>
>> 0002 <folder>
>> 0003 <folder>
>> ...
>> 1000 <folder>
>> 1001 <folder>
>>> File R1.RVG
>> ...
>> 4000 <folder>

Example: Works with no delay
> Data <folder>
>> 0001 <folder>
>> 0002 <folder>
>> 0003 <folder>
>> ...
>> 0050 <folder>
>> 0051 <folder>
>>> File R1.RVG
>> ...
>> 00080 <folder>



The University of Dundee is a registered Scottish Charity, No: SC015096

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20140325/4e6d4f06/attachment.html>