[ome-users] Error when converting large .svs or .vsi images

Wed Dec 5 08:24:31 GMT 2018

Hi,

the milestone Bio-Formats 6 release mentioned in our recent blog post [1] should contain
our current fix [2] for writing large files on non-local systems if you wanted to give it a try.

Best,
Sebastien

[1] https://blog.openmicroscopy.org/file-formats/community/2018/11/29/ometiffpyramid/
[2] https://github.com/ome/ome-common-java/pull/29

On 24 Nov 2018, at 00:58, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

I worked on it today.  The error does not occur when writing to a local disk.  I could copy the converted files back to the shared clustered filesystem (DDN Gridscaler), so the system does support the large files.  And as I mentioned, this is the primary storage system for a high performance compute cluster, which routinely manages large files.  So there must be some interaction between the OME Bio-formats software and the GPFS (aka IBM Spectrum Scale) file system.  Hard to speculate what, since GPFS is accessed with Posix calls.  Perhaps file locking latencies?  Who knows.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of Sebastien Besson (Staff)
Sent: Friday, November 23, 2018 3:43 AM
To: users OME <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Hi,

did you have a chance to test whether the issue can be reproduced when writing to a local disk
or whether this only applies to non-local file systems?

Sebastien

On 20 Nov 2018, at 14:52, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

On second thought, it is a GPFS (aka IBM Spectrum Scale) mounted disk.  It appears as a local disk (not NFS), but in actuality it is in a shared storage system (i.e. akin to a SAN, but with a shared cluster file system).  These conversions were done on our supercomputer.  It normally is used to very large datasets.

I was also getting errors when doing conversions on my laptop going to local disk.  I switched to the cluster to take advantage of larger memory sizes (128 GB HPC versus 16 GB laptop).  I could try again on the laptop to see if I get the same error, but not until tomorrow.

From: Tarbox, Lawrence
Sent: Tuesday, November 20, 2018 8:48 AM
To: users OME <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: RE: [ome-users] Error when converting large .svs or .vsi images

Local disk.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of Sebastien Besson (Staff)
Sent: Tuesday, November 20, 2018 4:58 AM
To: users OME <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Cc: Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Dear Lawrence,

re-reading this thread, one thing coming to mind is that we hit a very similar
error internally when testing large image conversions recently [1].

To make sure this is the same issue, are you writing the converted output to
a local disk or to a network file system?

Best,
Sebastien

[1] https://github.com/openmicroscopy/bioformats/pull/3189#issuecomment-410805187<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openmicroscopy_bioformats_pull_3189-23issuecomment-2D410805187&d=DwMGaQ&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=7PfGBb2SHdaGpGKqCizhwg&m=EweMJix5DTaLSZodAecdRo0mMLAECXtN4lLoeQbaLaw&s=lU99ySOWUtgtkocrUg05hdOA41P3Wh1qm8bbffva_fo&e=>

On 20 Nov 2018, at 05:39, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

I tried changing tile size.  The error still comes up.  The conversions that are failing all stop writing the TIFF files when the size of the output file hits 1,429,209,088 bytes, consistently, regardless of the size of the input file, on both .svs and .vsi files.  It does seem to be a restriction in the size of the file that the TIFF writer can write.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of Tarbox, Lawrence
Sent: Tuesday, November 6, 2018 11:22 AM
To: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>; OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

These errors came up while using a compression codec.  Here is the command that I was using:

/home/ltarbox/bftools/bfconvert  -no-upgrade -overwrite -compression JPEG \
  "${FILES[MOAB_JOBARRAYINDEX-1]}" \
  /home/ltarbox/CIR-scans/out/WSI-${MOAB_JOBARRAYINDEX}_S%s.ome.tif

Are you thinking that switching from JPEG to JPEG-2000 or LZW would work better?

The original .svs images use JPEG compression (I think) with a tile size of 256x256

The original .vsi images use JPEG compression with a tile size of 512x512.

I was somewhat hoping that if I requested -compression JPEG bfconvert would simply copy the already-JPEG-compressed tiles from the original images into the .ome.tif images.  But I’m game to try a different compression scheme.

From: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>
Sent: Tuesday, November 6, 2018 9:14 AM
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>; Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Hi Lawrence,

It does look like it is related to the file size of the converted file. Setting a compression codec should reduce the required size and might allow for easier conversion.

For example using:
bfconvert -compression JPEG-2000 /path/to/originalFile.svs /path/to/convertedFile.ome.tiff
bfconvert -compression LZW /path/to/originalFile.svs /path/to/convertedFile.ome.tiff

With Thanks,
David Gault

On 5 Nov 2018, at 20:19, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

I am using 5.9.2, from 3 Sept 2018.

The file sizes that are not converting range from 1.7 GB to 4.2 GB.  The .svs images that were smaller than about 1.3 GB converted without problem.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of David Gault (Staff)
Sent: Monday, November 5, 2018 7:10 AM
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Hi Lawrence,

That error looks as though the buffer has been unable to successfully increase its limit for some reason.
What size are the original files you are trying to convert? Also what version of Bio-Formats are you currently using?

With Thanks,
David Gault

On 2 Nov 2018, at 15:21, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

In converting large .svs or .vsi images, I get the following error:

/home/ltarbox/CIR-scans
54680.svs
SVSReader initializing 54680.svs
Reading IFDs
Populating metadata
Populating OME metadata
[Aperio SVS] -> /home/ltarbox/CIR-scans/out/WSI-2_S%s.ome.tif [OME-TIFF]
Switching to BigTIFF (by file size)
Tile size = 256 x 256
Exception in thread "main" java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:636)
        at loci.common.NIOFileHandle.writeSetup(NIOFileHandle.java:642)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:505)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:499)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:487)
        at loci.common.RandomAccessOutputStream.write(RandomAccessOutputStream.java:217)
        at loci.formats.tiff.TiffSaver.writeIFDStrips(TiffSaver.java:987)
        at loci.formats.tiff.TiffSaver.writeImageIFD(TiffSaver.java:451)
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:401)
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277)
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:275)
        at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:218)
        at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:748)
        at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:634)
        at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:563)
        at loci.formats.tools.ImageConverter.main(ImageConverter.java:884)

Smaller images convert fine.

I tripled the memory size (to 40 GB), but it stll fails

Ideas?

Lawrence Tarbox, Ph.D., Dept. of Biomedical Informatics, Univ. of Arkansas for Medical Sciences
mailto:LTarbox at uams.edu +1.314.681-2752

________________________________
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMFAg&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=7PfGBb2SHdaGpGKqCizhwg&m=r40EMcblVZoAtg0jtYbiDCk9hOS9PhHfHWfyTgcvMjE&s=AaJ76JDcXs17ypTw9JmpTKnUrJxLimF9uMYHNayZeuw&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMFAg&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=FQmnf-oLIEICrzyIHywOlUOTxJWIZf3tvJ8LUzIG0qM&m=3m3AIgFE_QbUasB7kt9_RCytADWjS30QJ6Al7TrUAQA&s=kfZIGmhU35jS7P0cZDCo9Svf9RFFIpFrGtbxDCbc37Y&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMGaQ&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=7PfGBb2SHdaGpGKqCizhwg&m=EweMJix5DTaLSZodAecdRo0mMLAECXtN4lLoeQbaLaw&s=Rjv_maVsUwoF-4xe26PyO7zqsMSENaDBHczwe-42ap0&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMGaQ&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=7PfGBb2SHdaGpGKqCizhwg&m=DxjEcXrRunsrvhixFdCIzXzgdDdKFVlnuUH-0dR7xDM&s=OynY4Q3Yi5ZtIg_9oWQs-6-aGX5hwLqFhCJ25i9vUSc&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181205/5198c7c4/attachment.html>