[ome-users] -|EXT|- Re: Ticket ID: 001-00-019862Issue importing large Multi-point scans into Fiji_Doesn't occur on smaller datasets

Fri Jul 27 10:38:49 BST 2018

Thanks James,

That particular error looks to be due to not having the Netcdf upgrade, the important thing is that it has identified the file as Imaris HDF and is attempting to use the correct image reader.
Can you run the below ImageJ macro, selecting the file from the dialog box when prompted, and provide the output log:

run("Bio-Formats Macro Extensions");
id = File.openDialog("Choose a file");
Ext.getFormat(id, format)
print("ID is " + id);
print("File format is: " + format);
Ext.setId(id);
Ext.getSeriesCount(seriesCount);
for (s=0; s<seriesCount; s++) {
  Ext.setSeries(s);
  Ext.getSizeX(sizeX);
  Ext.getSizeY(sizeY);
  Ext.getSizeZ(sizeZ);
  Ext.getSizeC(sizeC);
  Ext.getSizeT(sizeT);
  print("Series #" + s + ": image resolution is " + sizeX + " x " + sizeY);
  print("Focal plane count = " + sizeZ);
  print("Channel count = " + sizeC);
  print("Time point count = " + sizeT);
}
Ext.close();

With Thanks,
David Gault

On 26 Jul 2018, at 12:47, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:

Hi David,

Here are the results:

Checking file format [Bitplane Imaris 5.5 (HDF)]
Initializing reader
ImarisHDFReader initializing 2018-07-25_08.45.37_Sample Prefix_Protocol.ims
Exception in thread "main" java.io.IOException: java.lang.IllegalStateException:
      should equal FHIB
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:392)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:379)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:367)
        at loci.formats.services.NetCDFServiceImpl.init(NetCDFServiceImpl.java:310)
        at loci.formats.services.NetCDFServiceImpl.setFile(NetCDFServiceImpl.java:103)
        at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:246)
        at loci.formats.FormatReader.setId(FormatReader.java:1397)
        at loci.formats.ImageReader.setId(ImageReader.java:842)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.tools.ImageInfo.testRead(ImageInfo.java:1034)
        at loci.formats.tools.ImageInfo.main(ImageInfo.java:1120)
Caused by: java.lang.IllegalStateException:      should equal FHIB
        at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:401)
        at ucar.nc2.iosp.hdf5.FractalHeap.readIndirectBlock(FractalHeap.java:456)
        at ucar.nc2.iosp.hdf5.FractalHeap.<init>(FractalHeap.java:186)
        at ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2344)
        at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2332)
        at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2180)
        at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
        at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
        at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.<init>(H5header.java:2073)
        at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
        at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
        at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2153)
        at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2118)
        at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
        at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
        at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1521)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
        ... 11 more

I don’t know how to check the first 8 bytes? (I considered opening the file in notepad, but I don’t fancy trying that on a 95GB file...)

James

From: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>
Sent: 26 July 2018 11:19
To: WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>; OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>
Subject: Re: -|EXT|- Re: [ome-users] Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets

Hi James,

If you download the command line tools from https://www.openmicroscopy.org/bio-formats/downloads/ can you reproduce the error using:
showinf -nopix path/to/file/sample.ims

Also if you are able to check the first 8 bytes of the file that would allow us to confirm the expected signature.

With Thanks,
David Gault

On 25 Jul 2018, at 09:54, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:

Hi David,

I have now repeated the problem, which is independent of using a Virtual Stack.

The problem happens somewhere between 57GB and 95GB – for example, a 57GB file opens fine, but the 95GB file generates the following error message:

<image001.png>

This is using the latest netCDF from https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp

How can I provide more information on how to troubleshoot this?

Many thanks!

James

James Wainwright
Global Applications Specialist - Microscopy Systems

Tel:        +44 (0) 2890 237 126 ext. 2130
Skype:  andor.j.wainwright
Mob:     +44 (0) 7834 710 834
Web:     https://www.andor.com/microscopy-systems

<image002.jpg>

From: David Gault (Staff) [mailto:d.gault at dundee.ac.uk]
Sent: 23 July 2018 18:54
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Cc: row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>
Subject: -|EXT|- Re: [ome-users] Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets

Hi James,

In the Bio-Formats importer options there is a section labelled "Memory management”, from this if you select the option “Use Virtual Stack” then it will load each frame as required rather than reading the entire file at once.

With Thanks,
David Gault

On 23 Jul 2018, at 15:15, WAINWRIGHT James <J.Wainwright at ANDOR.COM<mailto:J.Wainwright at ANDOR.COM>> wrote:

Just to add that these are IMS files and that we have “patched” FIJI with the latest netCDF file with the jar from here:

https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp

James

James Wainwright
Global Applications Specialist - Microscopy Systems

Tel:        +44 (0) 2890 237 126 ext. 2130
Skype:  andor.j.wainwright
Mob:     +44 (0) 7834 710 834
Web:     https://www.andor.com/microscopy-systems

<image002.jpg>

From: row_microscopy_support
Sent: 23 July 2018 15:05
To: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
Cc: row_microscopy_support <row_microscopy_support at andor.com<mailto:row_microscopy_support at andor.com>>; WILDE Geraint <g.wilde at andor.com<mailto:g.wilde at andor.com>>; WAINWRIGHT James <J.Wainwright at andor.com<mailto:J.Wainwright at andor.com>>
Subject: Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets

Hi,

We have a customer in Australia that is acquiring large Multi-point datasets. Can be upwards of 500GB data sets each time.

They capture full Frame, no Binning (1024x1024 pixels), dual channel ( 2 simultaneous cameras acquiring) and 60-100 Z planes on multiple X/Y positions in an overnight (timelapse) protocol. These files can be upwards of 80Gb per XY position.

The problem they have is when they try and import these large data sets to FIJI it fails, however if they perform a similar experiment, only smaller in size there is no issue.

After asking the customer to perform a couple of tests for us, this is what they discovered:

“Definitely looks to be a file size issue.  Tried the smaller 1.5Gb files yesterday and it worked fine.  Did a longer experiment (same experimental setup just with more z-slices and more time points so the file was around 60Gb and these are failing.   The only other difference was that the small files that worked had been captured using two channel sequential while the large files that don’t work were captured with dual channel simultaneous capture.”

Any advice/workaround you can supply for handling these larger datasets would be appreciated.

Best,

Phil Mullan
Product Support Engineer (Microscopy Systems)
Europe, Middle East & Africa
Tel: +44 (0) 28 9023 7126 Ext. 2732
Fax: +44 (0) 28 9031 0792
Web:  andor.com/support<http://www.andor.com/>

<image003.jpg>

___________________________________________________________________________
This e-mail is confidential and is for the addressee only.   Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information. _______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
___________________________________________________________________________
This e-mail is confidential and is for the addressee only.   Please refer to
www.oxinst.com/email-statement<http://www.oxinst.com/email-statement> for regulatory information.

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180727/f59a3c4a/attachment.html>

[ome-users] -|EXT|- Re: Ticket ID: 001-00-019862__Issue importing large Multi-point scans into Fiji___Doesn't occur on smaller datasets

[ome-users] -|EXT|- Re: Ticket ID: 001-00-019862Issue importing large Multi-point scans into Fiji_Doesn't occur on smaller datasets