<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi,
<div class=""><br class="">
</div>
<div class="">Thank you for providing such a detailed breakdown of your troubleshooting, this really is very helpful for us. Improving the performance of Bio-Formats is currently one of our priorities and as you mentioned with the linked Trello card (<a href="https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing" class="">https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing</a>)
we are currently carrying out some investigation and profiling. We also have some initial PR’s open for testing at the moment which should help improve the performance.</div>
<div class=""><br class="">
</div>
<div class="">With Thanks,</div>
<div class="">David Gault</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 8 Oct 2018, at 22:28, ANI VARJABEDIAN <<a href="mailto:avarjabedian@wisc.edu" class="">avarjabedian@wisc.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><span class="" style="font-size:14px">Hello,</span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">I posted on the ImageJ forum about an issue regarding bio-formats slowdowns on a Windows computers and was instructed to send a bug report to your mailing list. The link to the forum discussion is here: </span><font color="#0069d9" class=""><span class="" style="font-size:14px"><u class=""><a href="https://forum.image.sc/t/changing-memory-allocation-has-no-effect/5477" class="">https://forum.image.sc/t/changing-memory-allocation-has-no-effect/5477</a></u></span></font></div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-size:14px"><b class="">ImageJ information:</b></span></div>
<div class="">
<ul class="x_MailOutline">
<li class=""><span class="" style="font-size:14px">I am using bio-formats 5.9.2 </span></li><li class=""><span class="" style="font-size:14px">Operating system = Windows 10 (comparing with an iMac running Sierra or a laptop running High Sierra)</span></li><li class=""><span class="" style="font-size:14px">Java version = 1.8.0_172 [64-bit]</span></li></ul>
<div class=""><br class="">
</div>
</div>
<div class=""><b class="" style="font-size:14px">The problem: </b></div>
<div class=""><span class="" style="font-size:14px">I have noticed that when using ImageJ/Fiji on a Windows computer, bio-formats importer is incredibly slow to load my hyperstacks compared to when I run the same dataset on any Apple Mac computer. This problem
appears </span><u class="" style="font-size:14px">only when I initialize the import using an .xml file</u><span class="" style="font-size:14px">. When I use a single TIFF file from the dataset as a seed, my Windows computer runs at comparable speeds to my
Mac and utilizes all of the memory I give to it. When I use the XML file as a seed, the Windows computer rarely uses above 1-5% of the RAM I allocate and takes forever to load all of the frames (multiple minutes, compared to seconds on a Mac). On a side note,
writing/saving files is also pretty slow on Windows compared to Mac, but I’m not sure if this is all related or if that issue comes from somewhere else.</span></div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-size:14px"><b class="">Data Type:</b></span></div>
<div class=""><span class="" style="font-size:14px">The data I am working with is multiple ome.tiff files from a Bruker Scanning Confocal running Prairie View software. Usually I take 5 optical slices per stack, and take anywhere from 200-500 stacks, usually
with 2 channels. </span><span class="" style="font-size:14px">The files are usually 512x512 pixels. </span><span class="" style="font-size:14px">This quickly adds up to a lot of individual files, but again, I only notice the slowness when running Fiji on my
Windows computer, and when I initialize the import from the xml file.</span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class="">
<div class=""><span class="" style="font-size:14px"><b class="">Background Info: </b></span></div>
<div class=""><span class="" style="font-size:14px">I noticed whenever I used my Windows computer to run bio-formats importer, although I would set a memory limit (usually 10-15GB on a 32GB system), the memory monitor usually only showed that ImageJ was using
~500MB or less to open files and it was incredibly slow to load my hyperstack. My iMac (which also has 32GB of RAM and a slower processor) took about 30s to open the same file, whereas the Windows took multiple minutes. The iMac also utilized a good chunk
of the memory I gave to it. I tested the import on a MacBook laptop that had 16GB RAM and got the same result (it was quicker than the Windows and used the memory given to it).</span></div>
</div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">As mentioned on the forum thread, I performed a thread dump, which I have attached to this email. The people who replied to my forum post said that the following lines indicated it might be something to do
with the performance on the file system: </span></div>
<div class="">
<pre class=""><code class="x_hljs x_groovy"><span class="x_hljs-string">"Bio-Formats Importer"</span> prio=<span class="x_hljs-number">4</span> id=<span class="x_hljs-number">29</span> group=main
java.lang.Thread.<span class="x_hljs-string">State:</span> RUNNABLE
at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
at java.io.File.isHidden(File.<span class="x_hljs-string">java:</span><span class="x_hljs-number">911</span>)
at loci.common.Location.isHidden(Location.<span class="x_hljs-string">java:</span><span class="x_hljs-number">679</span>)</code></pre>
<div class=""><span class="" style="font-size:14px">They said that the Prairie’s XML file enumerates TIFF files containing image data, and that doing a file listing the directory might be slowing it down. I am not a computer expert, so I’m not even totally
sure of what that means, but I thought I’d report it here in case the info was of use. </span></div>
</div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><b class=""><span class="" style="font-size:14px">What I’ve done for troubleshooting:</span></b></div>
<div class=""><span class="" style="font-size:14px">I tried multiple things (detailed on the thread) but here are the major findings.</span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">I tried using the File —> Import —> Image Sequence option and saw comparable speeds to a Mac, and it utilized the RAM I gave to it comparable to the Mac. But this option resulted in loss of metadata, which
is not ideal. </span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">Some time later, I tried using bio-formats importer and just fed it one of the ome.tiff files instead of the XML. This was much comparable to the speeds on the MacBooks and also utilized the memory limits
I set. I also was able to keep the metadata, which was nice. <u class="">This has been my workaround for the current time being.</u></span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">Using the XML file is more convenient for writing scripts because you just parse for the name of the scan instead of a specific ome.tiff file that might different depending on how many channels and slices
you acquired that session. Mac computers have no trouble initializing from an XML file, so I’m not sure what is different between them?</span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">The last person who commented on my post was David Gualt, who said that there are other reports of slowdowns in Windows environments detailed here: <a href="https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing" class="">https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing</a></span></div>
<div class=""><br class="">
</div>
<div class=""><span class="" style="font-size:14px">I did see some comments on Windows being slow to read and write files with bio-formats, and it seems consistent with my findings that it’s slow to import and also slow to save files. I’d love to get to the
bottom of this and any help is appreciated! I built a windows desktop specifically to help me process my data faster, so it’s been a bit disappointing that even my Mac laptop is performing better with less resources available! </span></div>
<div class=""><br class="">
</div>
<a href="https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing" rel="nofollow noopener" class=""><br class="">
</a>
<div class=""><span class="" style="font-size:14px">Thank you so much, and let me know if there is anything else you would like me to send or do!</span></div>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class=""><span class="" style="font-size:14px">-Ani Varjabedian</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""></div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""></div>
<div class=""><br class="">
</div>
</div>
</div>
<span id="cid:D703A5DA90295E429DC0D253AFF1095F@namprd06.prod.outlook.com"><upload.zip></span>_______________________________________________<br class="">
ome-users mailing list<br class="">
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>