[ome-users] Slow bio-formats import on Windows machines using XML files

Wed Oct 10 15:45:25 BST 2018

Hi,

Thank you for providing such a detailed breakdown of your troubleshooting, this really is very helpful for us. Improving the performance of Bio-Formats is currently one of our priorities and as you mentioned with the linked Trello card (https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing) we are currently carrying out some investigation and profiling. We also have some initial PR’s open for testing at the moment which should help improve the performance.

With Thanks,
David Gault

On 8 Oct 2018, at 22:28, ANI VARJABEDIAN <avarjabedian at wisc.edu<mailto:avarjabedian at wisc.edu>> wrote:

Hello,

I posted on the ImageJ forum about an issue regarding bio-formats slowdowns on a Windows computers and was instructed to send a bug report to your mailing list. The link to the forum discussion is here: https://forum.image.sc/t/changing-memory-allocation-has-no-effect/5477

ImageJ information:

  *   I am using bio-formats 5.9.2
  *   Operating system = Windows 10 (comparing with an iMac running Sierra or a laptop running High Sierra)
  *   Java version = 1.8.0_172 [64-bit]

The problem:
I have noticed that when using ImageJ/Fiji on a Windows computer, bio-formats importer is incredibly slow to load my hyperstacks compared to when I run the same dataset on any Apple Mac computer. This problem appears only when I initialize the import using an .xml file. When I use a single TIFF file from the dataset as a seed, my Windows computer runs at comparable speeds to my Mac and utilizes all of the memory I give to it. When I use the XML file as a seed, the Windows computer rarely uses above 1-5% of the RAM I allocate and takes forever to load all of the frames (multiple minutes, compared to seconds on a Mac). On a side note, writing/saving files is also pretty slow on Windows compared to Mac, but I’m not sure if this is all related or if that issue comes from somewhere else.

Data Type:
The data I am working with is multiple ome.tiff files from a Bruker Scanning Confocal running Prairie View software. Usually I take 5 optical slices per stack, and take anywhere from 200-500 stacks, usually with 2 channels. The files are usually 512x512 pixels. This quickly adds up to a lot of individual files, but again, I only notice the slowness when running Fiji on my Windows computer, and when I initialize the import from the xml file.

Background Info:
I noticed whenever I used my Windows computer to run bio-formats importer, although I would set a memory limit (usually 10-15GB on a 32GB system), the memory monitor usually only showed that ImageJ was using ~500MB or less to open files and it was incredibly slow to load my hyperstack. My iMac (which also has 32GB of RAM and a slower processor) took about 30s to open the same file, whereas the Windows took multiple minutes. The iMac also utilized  a good chunk of the memory I gave to it. I tested the import on a MacBook laptop that had 16GB RAM and got the same result (it was quicker than the Windows and used the memory given to it).

As mentioned on the forum thread, I performed a thread dump, which I have attached to this email. The people who replied to my forum post said that the following lines indicated it might be something to do with the performance on the file system:

"Bio-Formats Importer" prio=4 id=29 group=main
   java.lang.Thread.State: RUNNABLE
        at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
        at java.io.File.isHidden(File.java:911)
        at loci.common.Location.isHidden(Location.java:679)

They said that the Prairie’s XML file enumerates TIFF files containing image data, and that doing a file listing the directory might be slowing it down. I am not a computer expert, so I’m not even totally sure of what that means, but I thought I’d report it here in case the info was of use.

What I’ve done for troubleshooting:
I tried multiple things (detailed on the thread) but here are the major findings.

I tried using the File —> Import —> Image Sequence option and saw comparable speeds to a Mac, and it utilized the RAM I gave to it comparable to the Mac. But this option resulted in loss of metadata, which is not ideal.

Some time later, I tried using bio-formats importer and just fed it one of the ome.tiff files instead of the XML. This was much comparable to the speeds on the MacBooks and also utilized the memory limits I set. I also was able to keep the metadata, which was nice. This has been my workaround for the current time being.

Using the XML file is more convenient for writing scripts because you just parse for the name of the scan instead of a specific ome.tiff file that might different depending on how many channels and slices you acquired that session. Mac computers have no trouble initializing from an XML file, so I’m not sure what is different between them?

The last person who commented on my post was David Gualt, who said that there are other reports of slowdowns in Windows environments detailed here: https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing

I did see some comments on Windows being slow to read and write files with bio-formats, and it seems consistent with my findings that it’s slow to import and also slow to save files. I’d love to get to the bottom of this and any help is appreciated! I built a windows desktop specifically to help me process my data faster, so it’s been a bit disappointing that even my Mac laptop is performing better with less resources available!

<https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing>
Thank you so much, and let me know if there is anything else you would like me to send or do!

-Ani Varjabedian

<upload.zip>_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181010/05734358/attachment.html>