[ome-users] ImageProcessorReader.setId performance

[Balaji Ramalingam]

Hi,

I missed a point in my previous message, turns out that the reader searches and ‘groups’ similar images (this is helpful in the case of stacks, etc). In your case turning that option ‘off’ might do the trick.

If you could add this call,
r.setGroupFiles(false)
Before the call,
R.setId(ImagePath)

Then that should do the trick.

Best,
Balaji

From: Mohammad Samarah <isamarah at mac.com<mailto:isamarah at mac.com>>
Reply-To: "msamarah at alumni.fit.edu<mailto:msamarah at alumni.fit.edu>" <msamarah at alumni.fit.edu<mailto:msamarah at alumni.fit.edu>>
Date: Tuesday, 25 March 2014 14:19
To: Balaji Ramalingam <b.ramalingam at dundee.ac.uk<mailto:b.ramalingam at dundee.ac.uk>>, "ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>" <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] ImageProcessorReader.setId performance

Hi Balaji,

Thanks for your insights. We examined the code more closely and through the debugger. We are passing a single full file path to the ImageProcesserReader.setId method. The case that fails specifies /images/001234/R1.RVG while the images folder has more than 4000 sub-folders. We are able to make a workaround by means of copying the same file to /temp/images/001234/R1.RVG. Temp/images folder contains a small number of sub-folders. However this is not ideal as the copy takes time to complete given the size of the images being processed.

You mentioned that the default folder-depth is 4. How do we change it to 1? Is there an API method to disable this behavior?

Many thanks for your help,

--
M. Samarah
msamarah at alumni.fit.edu

________________________________
From: Balaji Ramalingam <b.ramalingam at dundee.ac.uk>
Date: Mon, 24 Mar 2014 10:42:02 +0000
To: "msamarah at alumni.fit.edu" <msamarah at alumni.fit.edu>, "ome-users at lists.openmicroscopy.org.uk" <ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] ImageProcessorReader.setId performance

Hi,

Your problem seems to be related to the search for Images by bio-formats(loci-tools), within your target directory (imagePath).
This is how the algorithm works in bio-formats,

  1.  The target directory(imagePath) is searched for all ‘bioformats-readable’ images.
  2.  -This is the very first step and this is where you are facing problems, I am assuming that you’re ‘imagePath’, is a top directory which has images at various depth’s. The default folder-depth at which bio-formats searches is 4 (I.e, it might still be searching for 4 sub-level folder’s within the top directory, unless specified).

   - The easy alternative is to pre list all the RVG format files within the ‘imagePath' (using file-system search), and read them one at a time     (iteratively). I.e imagePath == “Path to an individual image”.
       This basically removes the bio-formats intelligence of grouping similar files (files part of a stack,etc). Am assuming that should not be  a    problem in your case .

2.  The other alternative is to use the ImagePlusReader instead of the ImageProcessor as suggested in the code, but this needs testing     and playing around.

Quoting from the “ImageProcessorReader.java" code ,

  *   A low-level reader for {@link ij.process.ImageProcessor} objects.
  *   * For a higher-level reader that returns {@link ij.ImagePlus} objects,
  *   * see {@link loci.plugins.in.ImagePlusReader} instead.

Please do let us know, how it goes.

Best,
Balaji


From: Mohammad Samarah <isamarah at mac.com>
Reply-To: "msamarah at alumni.fit.edu" <msamarah at alumni.fit.edu>
Date: Saturday, 22 March 2014 03:36
To: "ome-users at lists.openmicroscopy.org.uk" <ome-users at lists.openmicroscopy.org.uk>
Subject: [ome-users] ImageProcessorReader.setId performance

ImageProcessorReader.setId performance
We are using loci_tools.jar to read radiographs in RVG format and extract a thumbnail JPEG image. All works as expected, however if the image in question is in a directory whose parent directory contains several thousands subfolders, then the ImageProcesserReader takes minutes to complete the setId() method. Please see attached code. The same image placed in a folder with its parent directory having much fewer subfolders works with no delay. This behavior can also be seen when using loci_tools.jar UI to open the same image. The issue does not seem to be related to the image itself but rather the fact that the containing folder is a child of a directory with thousands of subfolders. Could you please point us to how we can resolve this issue.

Many thanks for your help,

--
M. Samarah
msamarah at alumni.fit.edu


private ImagePlus readDicomImage(String imagePath)
    {
        ImagePlus imp            = null;
        ImagePlus colorizedImage = null;

        ImageProcessorReader r = new ImageProcessorReader(new ChannelSeparator(LociPrefs.makeImageReader()));
        try
        {
            log_message("INFO: Examining file " + imagePath);

            // the next line takes 10 to 20 minutes to complete when parent folder contains 4000+ subfolders. It takes no time when parent folder has 100 subfolders
            r.setId(imagePath);

            log_message("INFO: Setting file path " + imagePath);
            int num = r.getImageCount();
            log_message("INFO: Image count is" + num);
            int width = r.getSizeX();
            int height = r.getSizeY();
            log_message("INFO: Image Width %d Height %d", width, height);
            ImageStack stack = new ImageStack(width, height);
            log_message("INFO: After constructing image stack");
            byte[][][] lookupTable = new byte[r.getSizeC()][][];
            log_message("INFO: After Allocating memmory for image");
            for (int i=0; i<num; i++)
            {
                ImageProcessor ip = r.openProcessors(i)[0];
                stack.addSlice("" + (i + 1), ip);
                int channel = r.getZCTCoords(i)[1];
                lookupTable[channel] = r.get8BitLookupTable();
            }
            log_message("INFO: After constructing lookup table");
            log_message("INFO:Constructing image");
            imp = new ImagePlus("Dicom", stack);

            colorizedImage = applyLookupTables(r, imp, lookupTable);
            r.close();

        }
        catch(Exception exc)
        {
           log_message("ERROR an error occurred: " + exc.getMessage());
        }
       return colorizedImage;
    }

Example: Takes 10 to 20 minutes on an 8 core machine with 24GB of memory to complete setId method
Data <folder>
0001 <folder>
0002 <folder>
0003 <folder>
...
1000 <folder>
1001 <folder>
File R1.RVG
...
4000 <folder>

Example: Works with no delay
Data <folder>
0001 <folder>
0002 <folder>
0003 <folder>
...
0050 <folder>
0051 <folder>
File R1.RVG
...
00080 <folder>



The University of Dundee is a registered Scottish Charity, No: SC015096

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20140325/b8863fe3/attachment.html>