[ome-users] -|EXT|- Re: Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Jason Swedlow (Staff)
j.r.swedlow at dundee.ac.uk
Wed Aug 15 09:09:22 BST 2018
Dear All-
We’ve contacted the Andor team about this kind of support request. We’ll report back when we have a resolution.
Cheers,
Jason
From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk> On Behalf Of WAINWRIGHT James
Sent: 30 July 2018 11:29
To: David Gault (Staff) <d.gault at dundee.ac.uk>
Cc: WILDE Geraint <g.wilde at andor.com>; row_microscopy_support <row_microscopy_support at andor.com>; OME User Support List <ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] -|EXT|- Re: Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Hi David,
I did this while using netcdfAll-4.6.11.jar.
The one supplied with FIJI is not present in the jars directory (I have deleted it).
Thanks – output below:
ID is D:\FusionImages\2018-07-25\2018-07-25_08.45.37_Sample Prefix_Protocol.ims
File format is: Bitplane Imaris 5.5 (HDF)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at loci.plugins.macro.MacroFunctions.handleExtension(MacroFunctions.java:82)
at ij.macro.ExtensionDescriptor.dispatch(ExtensionDescriptor.java:288)
at ij.macro.Functions.doExt(Functions.java:4702)
at ij.macro.Functions.getStringFunction(Functions.java:275)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1360)
at ij.macro.Interpreter.getString(Interpreter.java:1338)
at ij.macro.Interpreter.doStatement(Interpreter.java:303)
at ij.macro.Interpreter.doStatements(Interpreter.java:235)
at ij.macro.Interpreter.run(Interpreter.java:118)
at ij.macro.Interpreter.run(Interpreter.java:89)
at ij.macro.Interpreter.run(Interpreter.java:100)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:145)
at ij.plugin.Macro_Runner.run(Macro_Runner.java:39)
at ij.IJ.runPlugIn(IJ.java:198)
at ij.Executer.runCommand(Executer.java:137)
at ij.Executer.run(Executer.java:66)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.io.EOFException: Reading D:/FusionImages/2018-07-25/2018-07-25_08.45.37_Sample Prefix_Protocol.ims at 4776428346718162991 file length = 4776428346718162991
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:427)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:394)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:381)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:369)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.plugins.macro.LociFunctions.setId(LociFunctions.java:427)
... 22 more
Caused by: java.io.EOFException: Reading D:/FusionImages/2018-07-25/2018-07-25_08.45.37_Sample Prefix_Protocol.ims at 4776428346718162991 file length = 4776428346718162991
at ucar.unidata.io.RandomAccessFile.readFully(RandomAccessFile.java:852)
at ucar.unidata.io.RandomAccessFile.readFully(RandomAccessFile.java:831)
at ucar.unidata.io.RandomAccessFile.readString(RandomAccessFile.java:1522)
at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:441)
at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:498)
at ucar.nc2.iosp.hdf5.FractalHeap.<init>(FractalHeap.java:182)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2477)
at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2463)
at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2294)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2129)
at ucar.nc2.iosp.hdf5.H5header.access$500(H5header.java:72)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.<init>(H5header.java:2175)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:4192)
at ucar.nc2.iosp.hdf5.H5header.access$1000(H5header.java:72)
at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2262)
at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2220)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:494)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:219)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:130)
at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1560)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:835)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:424)
... 34 more
James
James Wainwright
Global Applications Specialist - Microscopy Systems
Tel: +44 (0) 2890 237 126 ext. 2130
Skype: andor.j.wainwright
Mob: +44 (0) 7834 710 834
Web: https://www.andor.com/microscopy-systems
[logo]
From: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>
Sent: 27 July 2018 10:39
To: WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>
Cc: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>
Subject: Re: -|EXT|- Re: [ome-users] Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Thanks James,
That particular error looks to be due to not having the Netcdf upgrade, the important thing is that it has identified the file as Imaris HDF and is attempting to use the correct image reader.
Can you run the below ImageJ macro, selecting the file from the dialog box when prompted, and provide the output log:
run("Bio-Formats Macro Extensions");
id = File.openDialog("Choose a file");
Ext.getFormat(id, format)
print("ID is " + id);
print("File format is: " + format);
Ext.setId(id);
Ext.getSeriesCount(seriesCount);
for (s=0; s<seriesCount; s++) {
Ext.setSeries(s);
Ext.getSizeX(sizeX);
Ext.getSizeY(sizeY);
Ext.getSizeZ(sizeZ);
Ext.getSizeC(sizeC);
Ext.getSizeT(sizeT);
print("Series #" + s + ": image resolution is " + sizeX + " x " + sizeY);
print("Focal plane count = " + sizeZ);
print("Channel count = " + sizeC);
print("Time point count = " + sizeT);
}
Ext.close();
With Thanks,
David Gault
On 26 Jul 2018, at 12:47, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:
Hi David,
Here are the results:
Checking file format [Bitplane Imaris 5.5 (HDF)]
Initializing reader
ImarisHDFReader initializing 2018-07-25_08.45.37_Sample Prefix_Protocol.ims
Exception in thread "main" java.io.IOException: java.lang.IllegalStateException:
should equal FHIB
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1034)
at loci.formats.tools.ImageInfo.main(ImageInfo.java:1120)
Caused by: java.lang.IllegalStateException: should equal FHIB
at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:401)
at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:456)
at ucar.nc2.iosp.hdf5.FractalHeap.<init>(FractalHeap.java:186)
at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2344)
at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2332)
at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.<init>(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
... 11 more
I don’t know how to check the first 8 bytes? (I considered opening the file in notepad, but I don’t fancy trying that on a 95GB file...)
James
From: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>
Sent: 26 July 2018 11:19
To: WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>; OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>
Subject: Re: -|EXT|- Re: [ome-users] Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Hi James,
If you download the command line tools from https://www.openmicroscopy.org/bio-formats/downloads/ can you reproduce the error using:
showinf -nopix path/to/file/sample.ims
Also if you are able to check the first 8 bytes of the file that would allow us to confirm the expected signature.
With Thanks,
David Gault
On 25 Jul 2018, at 09:54, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:
Hi David,
I have now repeated the problem, which is independent of using a Virtual Stack.
The problem happens somewhere between 57GB and 95GB – for example, a 57GB file opens fine, but the 95GB file generates the following error message:
<image001.png>
This is using the latest netCDF from https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
How can I provide more information on how to troubleshoot this?
Many thanks!
James
James Wainwright
Global Applications Specialist - Microscopy Systems
Tel: +44 (0) 2890 237 126 ext. 2130
Skype: andor.j.wainwright
Mob: +44 (0) 7834 710 834
Web: https://www.andor.com/microscopy-systems
<image002.jpg>
From: David Gault (Staff) [mailto:d.gault at dundee.ac.uk]
Sent: 23 July 2018 18:54
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Cc: row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>
Subject: -|EXT|- Re: [ome-users] Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Hi James,
In the Bio-Formats importer options there is a section labelled "Memory management”, from this if you select the option “Use Virtual Stack” then it will load each frame as required rather than reading the entire file at once.
With Thanks,
David Gault
On 23 Jul 2018, at 15:15, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:
Just to add that these are IMS files and that we have “patched” FIJI with the latest netCDF file with the jar from here:
https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
James
James Wainwright
Global Applications Specialist - Microscopy Systems
Tel: +44 (0) 2890 237 126 ext. 2130
Skype: andor.j.wainwright
Mob: +44 (0) 7834 710 834
Web: https://www.andor.com/microscopy-systems
<image002.jpg>
From: row_microscopy_support
Sent: 23 July 2018 15:05
To: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
Cc: row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>
Subject: Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets
Hi,
We have a customer in Australia that is acquiring large Multi-point datasets. Can be upwards of 500GB data sets each time.
They capture full Frame, no Binning (1024x1024 pixels), dual channel ( 2 simultaneous cameras acquiring) and 60-100 Z planes on multiple X/Y positions in an overnight (timelapse) protocol. These files can be upwards of 80Gb per XY position.
The problem they have is when they try and import these large data sets to FIJI it fails, however if they perform a similar experiment, only smaller in size there is no issue.
After asking the customer to perform a couple of tests for us, this is what they discovered:
“Definitely looks to be a file size issue. Tried the smaller 1.5Gb files yesterday and it worked fine. Did a longer experiment (same experimental setup just with more z-slices and more time points so the file was around 60Gb and these are failing. The only other difference was that the small files that worked had been captured using two channel sequential while the large files that don’t work were captured with dual channel simultaneous capture.”
Any advice/workaround you can supply for handling these larger datasets would be appreciated.
Best,
Phil Mullan
Product Support Engineer (Microscopy Systems)
Europe, Middle East & Africa
Tel: +44 (0) 28 9023 7126 Ext. 2732
Fax: +44 (0) 28 9031 0792
Web: andor.com/support<http://www.andor.com/>
<image003.jpg>
___________________________________________________________________________
This e-mail is confidential and is for the addressee only. Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information. _______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
The University of Dundee is a registered Scottish Charity, No: SC015096
___________________________________________________________________________
This e-mail is confidential and is for the addressee only. Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information.
The University of Dundee is a registered Scottish Charity, No: SC015096
___________________________________________________________________________
This e-mail is confidential and is for the addressee only. Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information.
The University of Dundee is a registered Scottish Charity, No: SC015096
___________________________________________________________________________
This e-mail is confidential and is for the addressee only. Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information.
The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180815/56dba930/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 8737 bytes
Desc: image001.jpg
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180815/56dba930/attachment.jpg>
More information about the ome-users
mailing list