[ome-users] Error when converting large .svs or .vsi images

Tue Nov 20 10:57:57 GMT 2018

Dear Lawrence,

re-reading this thread, one thing coming to mind is that we hit a very similar
error internally when testing large image conversions recently [1].

To make sure this is the same issue, are you writing the converted output to
a local disk or to a network file system?

Best,
Sebastien

[1] https://github.com/openmicroscopy/bioformats/pull/3189#issuecomment-410805187

On 20 Nov 2018, at 05:39, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

I tried changing tile size.  The error still comes up.  The conversions that are failing all stop writing the TIFF files when the size of the output file hits 1,429,209,088 bytes, consistently, regardless of the size of the input file, on both .svs and .vsi files.  It does seem to be a restriction in the size of the file that the TIFF writer can write.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of Tarbox, Lawrence
Sent: Tuesday, November 6, 2018 11:22 AM
To: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>; OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

These errors came up while using a compression codec.  Here is the command that I was using:

/home/ltarbox/bftools/bfconvert  -no-upgrade -overwrite -compression JPEG \
  "${FILES[MOAB_JOBARRAYINDEX-1]}" \
  /home/ltarbox/CIR-scans/out/WSI-${MOAB_JOBARRAYINDEX}_S%s.ome.tif

Are you thinking that switching from JPEG to JPEG-2000 or LZW would work better?

The original .svs images use JPEG compression (I think) with a tile size of 256x256

The original .vsi images use JPEG compression with a tile size of 512x512.

I was somewhat hoping that if I requested -compression JPEG bfconvert would simply copy the already-JPEG-compressed tiles from the original images into the .ome.tif images.  But I’m game to try a different compression scheme.

From: David Gault (Staff) <d.gault at dundee.ac.uk<mailto:d.gault at dundee.ac.uk>>
Sent: Tuesday, November 6, 2018 9:14 AM
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>; Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Hi Lawrence,

It does look like it is related to the file size of the converted file. Setting a compression codec should reduce the required size and might allow for easier conversion.

For example using:
bfconvert -compression JPEG-2000 /path/to/originalFile.svs /path/to/convertedFile.ome.tiff
bfconvert -compression LZW /path/to/originalFile.svs /path/to/convertedFile.ome.tiff

With Thanks,
David Gault

On 5 Nov 2018, at 20:19, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

I am using 5.9.2, from 3 Sept 2018.

The file sizes that are not converting range from 1.7 GB to 4.2 GB.  The .svs images that were smaller than about 1.3 GB converted without problem.

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> On Behalf Of David Gault (Staff)
Sent: Monday, November 5, 2018 7:10 AM
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Error when converting large .svs or .vsi images

Hi Lawrence,

That error looks as though the buffer has been unable to successfully increase its limit for some reason.
What size are the original files you are trying to convert? Also what version of Bio-Formats are you currently using?

With Thanks,
David Gault

On 2 Nov 2018, at 15:21, Tarbox, Lawrence <LRTarbox at uams.edu<mailto:LRTarbox at uams.edu>> wrote:

In converting large .svs or .vsi images, I get the following error:

/home/ltarbox/CIR-scans
54680.svs
SVSReader initializing 54680.svs
Reading IFDs
Populating metadata
Populating OME metadata
[Aperio SVS] -> /home/ltarbox/CIR-scans/out/WSI-2_S%s.ome.tif [OME-TIFF]
Switching to BigTIFF (by file size)
Tile size = 256 x 256
Exception in thread "main" java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:636)
        at loci.common.NIOFileHandle.writeSetup(NIOFileHandle.java:642)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:505)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:499)
        at loci.common.NIOFileHandle.write(NIOFileHandle.java:487)
        at loci.common.RandomAccessOutputStream.write(RandomAccessOutputStream.java:217)
        at loci.formats.tiff.TiffSaver.writeIFDStrips(TiffSaver.java:987)
        at loci.formats.tiff.TiffSaver.writeImageIFD(TiffSaver.java:451)
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:401)
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277)
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:275)
        at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:218)
        at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:748)
        at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:634)
        at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:563)
        at loci.formats.tools.ImageConverter.main(ImageConverter.java:884)

Smaller images convert fine.

I tripled the memory size (to 40 GB), but it stll fails

Ideas?

Lawrence Tarbox, Ph.D., Dept. of Biomedical Informatics, Univ. of Arkansas for Medical Sciences
mailto:LTarbox at uams.edu +1.314.681-2752

________________________________
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMFAg&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=7PfGBb2SHdaGpGKqCizhwg&m=r40EMcblVZoAtg0jtYbiDCk9hOS9PhHfHWfyTgcvMjE&s=AaJ76JDcXs17ypTw9JmpTKnUrJxLimF9uMYHNayZeuw&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openmicroscopy.org.uk_mailman_listinfo_ome-2Dusers&d=DwMFAg&c=27AKQ-AFTMvLXtgZ7shZqsfSXu-Fwzpqk4BoASshREk&r=FQmnf-oLIEICrzyIHywOlUOTxJWIZf3tvJ8LUzIG0qM&m=3m3AIgFE_QbUasB7kt9_RCytADWjS30QJ6Al7TrUAQA&s=kfZIGmhU35jS7P0cZDCo9Svf9RFFIpFrGtbxDCbc37Y&e=>

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181120/fbd53179/attachment.html>