[ome-users] Slow bio-formats import on Windows machines using XML files

[ANI VARJABEDIAN]

Hello,

I posted on the ImageJ forum about an issue regarding bio-formats slowdowns on a Windows computers and was instructed to send a bug report to your mailing list. The link to the forum discussion is here: https://forum.image.sc/t/changing-memory-allocation-has-no-effect/5477

ImageJ information:

  *   I am using bio-formats 5.9.2
  *   Operating system = Windows 10 (comparing with an iMac running Sierra or a laptop running High Sierra)
  *   Java version = 1.8.0_172 [64-bit]

The problem:
I have noticed that when using ImageJ/Fiji on a Windows computer, bio-formats importer is incredibly slow to load my hyperstacks compared to when I run the same dataset on any Apple Mac computer. This problem appears only when I initialize the import using an .xml file. When I use a single TIFF file from the dataset as a seed, my Windows computer runs at comparable speeds to my Mac and utilizes all of the memory I give to it. When I use the XML file as a seed, the Windows computer rarely uses above 1-5% of the RAM I allocate and takes forever to load all of the frames (multiple minutes, compared to seconds on a Mac). On a side note, writing/saving files is also pretty slow on Windows compared to Mac, but I’m not sure if this is all related or if that issue comes from somewhere else.

Data Type:
The data I am working with is multiple ome.tiff files from a Bruker Scanning Confocal running Prairie View software. Usually I take 5 optical slices per stack, and take anywhere from 200-500 stacks, usually with 2 channels. The files are usually 512x512 pixels. This quickly adds up to a lot of individual files, but again, I only notice the slowness when running Fiji on my Windows computer, and when I initialize the import from the xml file.

Background Info:
I noticed whenever I used my Windows computer to run bio-formats importer, although I would set a memory limit (usually 10-15GB on a 32GB system), the memory monitor usually only showed that ImageJ was using ~500MB or less to open files and it was incredibly slow to load my hyperstack. My iMac (which also has 32GB of RAM and a slower processor) took about 30s to open the same file, whereas the Windows took multiple minutes. The iMac also utilized  a good chunk of the memory I gave to it. I tested the import on a MacBook laptop that had 16GB RAM and got the same result (it was quicker than the Windows and used the memory given to it).

As mentioned on the forum thread, I performed a thread dump, which I have attached to this email. The people who replied to my forum post said that the following lines indicated it might be something to do with the performance on the file system:

"Bio-Formats Importer" prio=4 id=29 group=main
   java.lang.Thread.State: RUNNABLE
        at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
        at java.io.File.isHidden(File.java:911)
        at loci.common.Location.isHidden(Location.java:679)

They said that the Prairie’s XML file enumerates TIFF files containing image data, and that doing a file listing the directory might be slowing it down. I am not a computer expert, so I’m not even totally sure of what that means, but I thought I’d report it here in case the info was of use.

What I’ve done for troubleshooting:
I tried multiple things (detailed on the thread) but here are the major findings.

I tried using the File —> Import —> Image Sequence option and saw comparable speeds to a Mac, and it utilized the RAM I gave to it comparable to the Mac. But this option resulted in loss of metadata, which is not ideal.

Some time later, I tried using bio-formats importer and just fed it one of the ome.tiff files instead of the XML. This was much comparable to the speeds on the MacBooks and also utilized the memory limits I set. I also was able to keep the metadata, which was nice. This has been my workaround for the current time being.

Using the XML file is more convenient for writing scripts because you just parse for the name of the scan instead of a specific ome.tiff file that might different depending on how many channels and slices you acquired that session. Mac computers have no trouble initializing from an XML file, so I’m not sure what is different between them?

The last person who commented on my post was David Gualt, who said that there are other reports of slowdowns in Windows environments detailed here: https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing

I did see some comments on Windows being slow to read and write files with bio-formats, and it seems consistent with my findings that it’s slow to import and also slow to save files. I’d love to get to the bottom of this and any help is appreciated! I built a windows desktop specifically to help me process my data faster, so it’s been a bit disappointing that even my Mac laptop is performing better with less resources available!


<https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing>
Thank you so much, and let me know if there is anything else you would like me to send or do!

-Ani Varjabedian




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181008/dc8bafa8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upload.zip
Type: application/zip
Size: 42844 bytes
Desc: upload.zip
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181008/dc8bafa8/attachment.zip>