[ome-users] Bio-Formats ND2 importer bug

Fri Feb 9 16:01:54 GMT 2018

Hi Perrine and Michael,

Trying to answer a couple of issues raised on this thread.

Regarding the performance issue, similar reports began a few years ago as the size of the acquired ND2 files started to increase sharply. Bio-Formats 5.4.0 included a significant community contribution to provide a first round of performance improvement for this file format [1].

Our Bio-Formats development process includes the testing of about 200 representative ND2 filesets. The largest of ND2 file tested daily (~200G) definitely exhibited slow initialization times over a shared NFS mount prior to this performance improvements:

sbesson at necromancer ~ $ for i in $(seq 1 4); do ~/bftools-5.3.0/showinf -nopix /ome/data_repo/curated/nd2/sara/Ex9\ ALADIN\ WT\ KO\ KI\ 31_07_2013.nd2 | grep Initialization; done
Initialization took 394.658s
Initialization took 16.242s
Initialization took 16.168s
Initialization took 16.256s

Since Bio-Formats 5.4.0, the initialization time was drastically reduced so that our largest ND2 samples have been reincluded into our daily test suite, see [2] for the latest run on our current test system:

$ for i in $(seq 1 4); do ./bftools-5.7.3/showinf -nopix /ome/data_repo/curated/nd2/sara/Ex9\ ALADIN\ WT\ KO\ KI\ 31_07_2013.nd2 | grep Initialization; done
Initialization took 2.458s
Initialization took 2.514s
Initialization took 2.467s
Initialization took 2.457s

To help further with the slowness issue, it might be useful to have more specifics about the file size, mount option and performance metrics you are experiencing on your end.

Regarding the new ND2 bug, thanks as always for notifying us and uploading a representative sample file. Before we initiate more investigation, our understanding from the initial message is that the faulty files have been generated by the latest version of the acquisition software, is that correct? If this is the case, a definite possibility is that the data saved by the new acquisition software contains breaking changes to the Nikon ND2 format which invalidate assumptions of our reader. Has this issue been independently communicated to Nikon? If an update to Nikon software broke your workflow, perhaps you should report it to them?

Keeping up with new proprietary file formats or new versions of them is a perpetual challenge and not something that academic funding can support. Recent community efforts to engage with vendors led to significant contributions which have helped to reduce the burden of maintaining open-source tools essential for the accessibility and re-usability of scientific data. [3]

Best,
Sebastien

[1] https://docs.openmicroscopy.org/bio-formats/5.4.0/about/whats-new.html<http://docs.openmicroscopy.org/bio-formats/5.4.0/about/whats-new.html>
[2] https://web-proxy.openmicroscopy.org/next-ci/job/BIOFORMATS-test-folder/7982/console
[3] http://blog.openmicroscopy.org/file-formats/community/2016/08/31/bf-partnerships/

On 8 Feb 2018, at 14:59, Perrine PAUL-GILLOTEAUX <Perrine.Paul-Gilloteaux at univ-nantes.fr<mailto:Perrine.Paul-Gilloteaux at univ-nantes.fr>> wrote:

They do open, (we have long multi pos of about 200Gb each), but indeed you need to be patient ...
Data conversion/processing /analysis takes about 1 day with Fiji.

Le 08-02-2018 15:42, Cammer, Michael a écrit :
We gave up on BioFormats for opening ND2 files more than a few hundred
MB. We now have files that are 10+ GB and they don't open, or maybe
they do but we don't have the patience to wait. It's faster to export
folders of TIFs from Elements and reassemble.
We'd love to see ND2 importer working for large files.
MICHAEL CAMMER, Sr Research Scientist, DART Microscopy Laboratory
NYU Langone Health, 540 First Avenue, SK2 Microscopy Suite, New York,
NY 10016
C: 914-309-3270 Michael.Cammer at med.nyu.edu<mailto:Michael.Cammer at med.nyu.edu> http://nyulmc.org/micros
[1] http://microscopynotes.com/ [2]
FROM: ome-users [mailto:ome-users-bounces at lists.openmicroscopy.org.uk]
ON BEHALF OF Paul-Gilloteaux Perrine
SENT: Thursday, February 8, 2018 9:31 AM
TO: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
CC: francois.paris at univ-nantes.fr<mailto:francois.paris at univ-nantes.fr>;
perrine.paul-gilloteaux at univ-nantes.fr<mailto:perrine.paul-gilloteaux at univ-nantes.fr>; Isabelle.Delias at nikon.com<mailto:Isabelle.Delias at nikon.com>;
manon.pietri at univ-nantes.fr<mailto:manon.pietri at univ-nantes.fr>
SUBJECT: [ome-users] Bio-Formats ND2 importer bug
Dear Bio-Formats team,
Im also reporting a bug on nd2 with bioformat (tested on last release
5.7.3 snapshot and previous)
Disabling the chunkmap option does not solve the problem.
I ve uploaded a short example data set here:
https://www.openmicroscopy.org/qa2/qa/feedback/20512/?token=1f4ad9c32a8ea3de4f0c34d2c4add35c
[3]
In a few words (see my comments on the link above), time and positions
are shuffled, while NIS element correcty read the file metadata.
Series corresponds to time when it should be position, and time to
what should be position.
It only happens when there is several channels (no problem if the
movie is 1 channel multi-t, multi pos)
It only happens with acquisitions generated with the last version of
NIS 4.5.0.00 (but read correctly by previous version of NIS element).
Best regards,
Perrine
------------------------------------------------------------
This email message, including any attachments, is for the sole use of
the intended recipient(s) and may contain information that is
proprietary, confidential, and exempt from disclosure under applicable
law. Any unauthorized review, use, disclosure, or distribution is
prohibited. If you have received this email in error please notify the
sender by return email and delete the original message. Please note,
the recipient should check this email and any attachments for the
presence of viruses. The organization accepts no liability for any
damage caused by any virus transmitted by this email.
=================================
Links:
------
[1] http://nyulmc.org/micros
[2] http://microscopynotes.com/
[3]
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.openmicroscopy.org_qa2_qa_feedback_20512_-3Ftoken-3D1f4ad9c32a8ea3de4f0c34d2c4add35c&d=DwMFAw&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=oU_05LztNstAydlbm5L5GDu_vAdjXk3frDLx_CqKkuo&m=wOsSGQja4TLs56sXa1P8sT_q6DdHNaWJvRr50dHM4U4&s=LH49R6mHp1j5M6bwJ4YyuzFZ7TNHPps6IrWr_9OqwIs&e=

--
Perrine Paul-Gilloteaux, IR CNRS
Ingénieur en gestion et analyse de données images

SFR Santé François Bonamy UMS 3556
IRS-UN, 8 quai Moncousu, BP 70721
44007 Nantes cedex 1, France

perrine.paul-gilloteaux at univ-nantes.fr<mailto:perrine.paul-gilloteaux at univ-nantes.fr>
tel: 02 28 08 03 43
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180209/2108768f/attachment.html>