<HTML>
<HEAD>
<TITLE>Re: [ome-users] ImageProcessorReader.setId performance</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi Balaji,<BR>
<BR>
Calling setGroupFiles(false) prior to invoking setId() solved the issue. Many thanks for your help.<BR>
<BR>
Mohammad<BR>
<BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"><B>From: </B>Balaji Ramalingam <<a href="b.ramalingam@dundee.ac.uk">b.ramalingam@dundee.ac.uk</a>><BR>
<B>Date: </B>Tue, 25 Mar 2014 15:55:58 +0000<BR>
<B>To: </B>"<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>" <<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>>, "<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>" <<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>><BR>
<B>Subject: </B>Re: [ome-users] ImageProcessorReader.setId performance<BR>
<BR>
Hi,<BR>
<BR>
I missed a point in my previous message, turns out that the reader searches and ‘groups’ similar images (this is helpful in the case of stacks, etc). In your case turning that option ‘off’ might do the trick.<BR>
<BR>
If you could add this call,<BR>
r.setGroupFiles(false)<BR>
Before the call,<BR>
R.setId(ImagePath)<BR>
<BR>
Then that should do the trick.<BR>
<BR>
Best,<BR>
Balaji<BR>
<BR>
<B>From: </B>Mohammad Samarah <<a href="isamarah@mac.com">isamarah@mac.com</a>><BR>
<B>Reply-To: </B>"<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>" <<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>><BR>
<B>Date: </B>Tuesday, 25 March 2014 14:19<BR>
<B>To: </B>Balaji Ramalingam <<a href="b.ramalingam@dundee.ac.uk">b.ramalingam@dundee.ac.uk</a>>, "<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>" <<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>><BR>
<B>Subject: </B>Re: [ome-users] ImageProcessorReader.setId performance<BR>
<BR>
Re: [ome-users] ImageProcessorReader.setId performance <BR>
Hi Balaji,<BR>
<BR>
Thanks for your insights. We examined the code more closely and through the debugger. We are passing a single full file path to the ImageProcesserReader.setId method. The case that fails specifies /images/001234/R1.RVG while the images folder has more than 4000 sub-folders. We are able to make a workaround by means of copying the same file to /temp/images/001234/R1.RVG. Temp/images folder contains a small number of sub-folders. However this is not ideal as the copy takes time to complete given the size of the images being processed.<BR>
<BR>
You mentioned that the default folder-depth is 4. How do we change it to 1? Is there an API method to disable this behavior? <BR>
<BR>
Many thanks for your help,<BR>
<BR>
--<BR>
M. Samarah<BR>
<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a><BR>
<BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"><B>From: </B>Balaji Ramalingam <<a href="b.ramalingam@dundee.ac.uk">b.ramalingam@dundee.ac.uk</a>><BR>
<B>Date: </B>Mon, 24 Mar 2014 10:42:02 +0000<BR>
<B>To: </B>"<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>" <<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>>, "<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>" <<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>><BR>
<B>Subject: </B>Re: [ome-users] ImageProcessorReader.setId performance<BR>
<BR>
Hi,<BR>
<BR>
Your problem seems to be related to the search for Images by bio-formats(loci-tools), within your target directory (imagePath).<BR>
This is how the algorithm works in bio-formats,<BR>
</SPAN></FONT><OL><LI><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>The target directory(imagePath) is searched for all ‘bioformats-readable’ images.
</SPAN></FONT><LI><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>-This is the very first step and this is where you are facing problems, I am assuming that you’re ‘imagePath’, is a top directory which has images at various depth’s. The default folder-depth at which bio-formats searches is 4 (I.e, it might still be searching for 4 sub-level folder’s within the top directory, unless specified).<BR>
</SPAN></FONT></OL><FONT FACE="Calibri, Verdana, Helvetica, Arial"><FONT SIZE="2"><SPAN STYLE='font-size:10pt'> - The easy alternative is to pre list all the <B>RVG</B> format files within the ‘imagePath' (using file-system search), and read them one at a time (iteratively). I.e imagePath == “Path to an individual image”.<BR>
This basically removes the bio-formats intelligence of grouping similar files (files part of a stack,etc). Am assuming that should not be a problem in your case .<BR>
<BR>
2. </SPAN></FONT><SPAN STYLE='font-size:11pt'>The other alternative is to use the ImagePlusReader instead of the ImageProcessor as suggested in the code, but this needs testing and playing around. <BR>
<BR>
</SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>Quoting from the “ImageProcessorReader.java" code ,<BR>
</SPAN></FONT></FONT><UL><LI><FONT FACE="Calibri, Verdana, Helvetica, Arial"><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>A low-level reader for {@link ij.process.ImageProcessor} objects.
</SPAN></FONT></FONT><LI><FONT FACE="Calibri, Verdana, Helvetica, Arial"><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>* For a higher-level reader that returns {@link ij.ImagePlus} objects,
</SPAN></FONT></FONT><LI><FONT FACE="Calibri, Verdana, Helvetica, Arial"><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>* see {@link loci.plugins.in.ImagePlusReader} instead.<BR>
</SPAN></FONT></FONT></UL><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
Please do let us know, how it goes.<BR>
<BR>
Best,<BR>
Balaji<BR>
</SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'><BR>
<BR>
<B>From: </B>Mohammad Samarah <<a href="isamarah@mac.com">isamarah@mac.com</a>><BR>
<B>Reply-To: </B>"<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>" <<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a>><BR>
<B>Date: </B>Saturday, 22 March 2014 03:36<BR>
<B>To: </B>"<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>" <<a href="ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a>><BR>
<B>Subject: </B>[ome-users] ImageProcessorReader.setId performance<BR>
<BR>
ImageProcessorReader.setId performance <BR>
We are using loci_tools.jar to read radiographs in RVG format and extract a thumbnail JPEG image. All works as expected, however if the image in question is in a directory whose parent directory contains several thousands subfolders, then the ImageProcesserReader takes minutes to complete the setId() method. Please see attached code. The same image placed in a folder with its parent directory having much fewer subfolders works with no delay. This behavior can also be seen when using loci_tools.jar UI to open the same image. The issue does not seem to be related to the image itself but rather the fact that the containing folder is a child of a directory with thousands of subfolders. Could you please point us to how we can resolve this issue.<BR>
<BR>
Many thanks for your help,<BR>
<BR>
--<BR>
M. Samarah<BR>
<a href="msamarah@alumni.fit.edu">msamarah@alumni.fit.edu</a><BR>
</SPAN></FONT><SPAN STYLE='font-size:11pt'><BR>
<BR>
private ImagePlus readDicomImage(String imagePath)<BR>
{<BR>
ImagePlus imp = null;<BR>
ImagePlus colorizedImage = null;<BR>
<BR>
ImageProcessorReader r = new ImageProcessorReader(new ChannelSeparator(LociPrefs.makeImageReader()));<BR>
try<BR>
{<BR>
log_message("INFO: Examining file " + imagePath);<BR>
<BR>
// the next line takes 10 to 20 minutes to complete when parent folder contains 4000+ subfolders. It takes no time when parent folder has 100 subfolders<BR>
<FONT COLOR="#FF0000">r.setId(imagePath);<BR>
</FONT><BR>
log_message("INFO: Setting file path " + imagePath);<BR>
int num = r.getImageCount();<BR>
log_message("INFO: Image count is" + num);<BR>
int width = r.getSizeX();<BR>
int height = r.getSizeY();<BR>
log_message("INFO: Image Width %d Height %d", width, height);<BR>
ImageStack stack = new ImageStack(width, height);<BR>
log_message("INFO: After constructing image stack");<BR>
byte[][][] lookupTable = new byte[r.getSizeC()][][];<BR>
log_message("INFO: After Allocating memmory for image");<BR>
for (int i=0; i<num; i++) <BR>
{<BR>
ImageProcessor ip = r.openProcessors(i)[0];<BR>
stack.addSlice("" + (i + 1), ip);<BR>
int channel = r.getZCTCoords(i)[1];<BR>
lookupTable[channel] = r.get8BitLookupTable();<BR>
}<BR>
log_message("INFO: After constructing lookup table");<BR>
log_message("INFO:Constructing image");<BR>
imp = new ImagePlus("Dicom", stack);<BR>
<BR>
colorizedImage = applyLookupTables(r, imp, lookupTable);<BR>
r.close();<BR>
<BR>
}<BR>
catch(Exception exc)<BR>
{<BR>
log_message("ERROR an error occurred: " + exc.getMessage());<BR>
}<BR>
return colorizedImage; <BR>
}<BR>
<BR>
<FONT COLOR="#FF0000">Example: Takes 10 to 20 minutes on an 8 core machine with 24GB of memory to complete setId method<BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#FF0000">Data <folder><BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#FF0000">0001 <folder><BR>
0002 <folder><BR>
0003 <folder><BR>
...<BR>
1000 <folder><BR>
1001 <folder><BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#FF0000">File R1.RVG<BR>
</FONT></SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#FF0000">...<BR>
4000 <folder><BR>
</FONT></SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<FONT COLOR="#008000">Example: Works with no delay<BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#008000">Data <folder><BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#008000">0001 <folder><BR>
0002 <folder><BR>
0003 <folder><BR>
...<BR>
0050 <folder><BR>
0051 <folder><BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#008000">File R1.RVG<BR>
</FONT></SPAN></FONT></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><FONT COLOR="#008000">...<BR>
00080 <folder><BR>
</FONT></SPAN></FONT></BLOCKQUOTE></BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
<BR>
<BR>
</SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>The University of Dundee is a registered Scottish Charity, No: SC015096<BR>
</SPAN></FONT><SPAN STYLE='font-size:10.5pt'><BR>
</SPAN><FONT SIZE="2"><SPAN STYLE='font-size:10pt'>The University of Dundee is a registered Scottish Charity, No: SC015096</SPAN></FONT><SPAN STYLE='font-size:10.5pt'><BR>
</SPAN></FONT>
</BODY>
</HTML>