[ome-devel] TIFF pyramid support in Bio-Formats - reference files for review

[Damir Sudar]

Hi Roger,

I meanwhile found my mistake in makepyramid.ptif. Enclosed one that 
works correctly.

- Damir

On 3/23/2018 10:55, Damir Sudar wrote:
> Hi Roger,
>
> On 3/23/2018 2:46, Roger Leigh wrote:
>> On 23/03/18 01:40, Damir Sudar wrote:
>>> TIFFWriteDirectoryTagSubifd: Illegal value for SubIFD tag.
>>> TIFFWriteDirectoryTagSubifd: Illegal value for SubIFD tag.
>>> I'm using libtiff 4.0.9 and its associated tools on Ubuntu 14.04. The
>>> error does not happen with the scn files provided. Could it be related
>>> to having an offset value that is too large for a non-BigTIFF file?
>> Yes, you need to apply this change to libtiff:
>>   https://gitlab.com/libtiff/libtiff/merge_requests/25/diffs
>> Hopefully this will be in libtiff 4.0.10.  We must be the first users of
>> tiffset with BigTIFF images to run into this problem.
>>
> Indeed, with your merge request applied, it works correctly for svs 
> files and almost for my ptifs. I'm still chasing a small bug around 
> line 133 that causes the subresolutions pyramid to appear in its own 
> top-level IFD instead of all being stuck as SubIFDs into the single 
> top-level IFD. I'll keep chasing but in case you see my mistake?
>
>> I should also mention that the scripts have only been tested on Linux,
>> and might not run on other systems.  Hopefully they will be short-lived;
>> as soon as we have bfconvert supporting pyramids they can be forgotten.
>
> Yes, but for their intended purpose, these scripts are great.
> Once I get the ptif one to work correctly, I'll mock up some more 
> example files including some with larger numbers of channels, Z and t.
>
>> - also, all these examples carry some of those ancillary images such as
>>> slide labels, overview images, etc. While the scripts handled those
>>> correctly as far as I can see, a reader would have to be pretty 
>>> smart to
>>> figure out which of the contents of such an output file is the actual
>>> image and which are those ancillary images.
>>
>> Absolutely.  We will need to do something about these as a followup
>> task, to add the metadata to identify these as labels, overviews, etc.
>>
> Ok, I'll start looking at options how to do so.
>
> Cheers,
> - Damir
>>
>> Kind regards,
>> Roger
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>

-- 
Damir Sudar - Affiliate Scientist
Lawrence Berkeley Natl Laboratory / MBIB
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
http://biosciences.lbl.gov/profiles/damir-sudar-2/

Visiting Scientist, Oregon Health & Science University

-------------- next part --------------
#!/bin/bash

set -e
set -x

# Check if file is BigTIFF
# $1 file
bigtiff() {
    if [ "0000000 4949 002b" != "$(dd if="$1" iflag=count_bytes count=4 | od -x | head -n1)" ]; then
        return 1
    fi
    return 0
}

# Get fieldoffset for TIFF
# $1 file
fieldoffset() {
    if bigtiff "$1"; then
        echo 8
    else
        echo 2
    fi
}

# Get fieldsize for TIFF
# $1 file
fieldsize() {
    if bigtiff "$1"; then
        echo 20
    else
        echo 12
    fi
}

# Get IFD offsets
# $1=IFD number
# $2=file
diroffsets() {
    tiffinfo "$dest" | grep "TIFF Directory at offset" | sed -e 's;.*(\(.*\))$;\1;'
}

# Get offset for IFD
# $1=IFD number
# $2=file
diroffset() {
    diroffsets "$2" | head -n$(($1 + 1)) | tail -n1
}

# Get number of tags in directory
# $1=IFD offset
# $2=file
ntags() {
    echo "od -j $1 -N 2 -d \"$2\"" >&2
  od -j $1 -N 2 -d "$2" | head -n1 | sed -e 's;^[0-9]* *\(.*\);\1;'
}

# Offset of next pointer in IFD
# $1=IFD offset
# $2=file
nextoffset() {
    echo "$(($1 + $(fieldoffset "$2") + ($(ntags $1 "$2") * $(fieldsize "$2"))))"
}

# Write uint64 little endian value to binary file
# $1=value
# $2=destination file
# $3=offset in file
update_uint64_le() {
    if bigtiff "$2"; then
        printf "$(printf %.16x $1 | sed -e 's;\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)\(..\)\(..\);\8\7\6\5\4\3\2\1;' | sed -e 's;\(..\);\\x\1;g')" | dd of="$2" conv=notrunc,nocreat oflag=seek_bytes seek=$3
    else
        printf "$(printf %.8x $1 | sed -e 's;\(..\)\(..\)\(..\)\(..\);\4\3\2\1;' | sed -e 's;\(..\);\\x\1;g')" | dd of="$2" conv=notrunc,nocreat oflag=seek_bytes seek=$3
    fi
}

makeuuid() {
    echo "urn:uuid:$(uuidgen)"
}

src="$(ls -1 orig/*.ptif)"
mkdir -p new

for srcfile in $src; do
    base="$(basename $srcfile)"
    dest="new/${base%.ptif}-subifds.tiff"

    cp "$srcfile" "$dest"

    nifds=$(tiffinfo "$dest" | grep "TIFF Directory at offset" | wc -l)
    echo "IFD count: $nifds"

    ifds=$(seq 0 $(($nifds - 1)))
    mainifd=$(echo "$ifds" | head -n1)
#    otherifds=$(echo "$ifds" | tail -n2)
    otherifds=''
    subifds=$(echo "$ifds" | tail -n$((nifds-1)))
    nsubifds=$(echo "$subifds" | wc -l)
    echo "IFDs: $ifds"
    echo "Main IFDs: $mainifd"
    echo "Other IFDs: $otherifds"
    echo "SUBIFDs: $subifds"

    # Main header
    tiffset -d 0 -s 270 "OME Pyramid TIFF test (from $base)" "$dest"
    tiffset -d 0 -s 305 "A gnarly shell script (makepyramid-ptif)" "$dest"
    tiffset -d 0 -s 315 "Roger Leigh <rleigh at dundee.ac.uk>" "$dest"

    # NewSubFileType
    for ifd in $mainifd $otherifds; do
        tiffset -d $ifd -s 254 0 "$dest"
    done
    for ifd in $subifds; do
        tiffset -d $ifd -s 254 1 "$dest"
    done

    # SubIFDs
    subifds_diroffs=$(echo $(tiffinfo "$dest" | grep "TIFF Directory at offset" | sed -e 's;.*(\(.*\))$;\1;' | tail -n$((nifds-1))))
    echo "SubIFDs for series 0: $subifds_diroffs"
    tiffset -d 0 -s 330 $nsubifds $subifds_diroffs "$dest"
    subifds_diroffs=$(echo $(tiffinfo "$dest" | grep "TIFF Directory at offset" | sed -e 's;.*(\(.*\))$;\1;' | tail -n$((nifds-1))))
    echo "Updated SubIFDs for series 0: $subifds_diroffs"

    echo "New directories:"
    diroffsets "$dest"

    dest2="${dest%.tiff}-flat.tiff"
    cp "$dest" "$dest2"

    # Relink IFDs to elide SubIFDs
    series0dir="$(diroffset 0 "$dest")"
    echo "s0dir $series0dir"
# assumption: only one real IFD per file - this needs revision if multiple real IFDs
#    seriesndir="$(diroffset $(echo "$otherifds" | head -n1) "$dest")"
#    echo "sndir $seriesndir"

    noffset="$(nextoffset $series0dir "$dest")"
#    echo "UPDATE: $series0dir $noffset -> [$seriesndir]"
#    update_uint64_le $seriesndir "$dest" $noffset
    echo "UPDATE: $series0dir $noffset -> 0"
    update_uint64_le 0 "$dest" $noffset

    for offset in $subifds_diroffs; do
        c="$(ntags $offset "$dest")"
        echo "NTAGS: $c"
        noffset="$(nextoffset $offset "$dest")"
        echo "UPDATE: $offset $noffset -> 0"
        update_uint64_le 0 "$dest" $noffset
    done

    tiffinfo "$dest"

    # Create OME-XML metadata for the files.
    dest_ometiff="${dest%.tiff}.ome.tiff"
    dest2_ometiff="${dest2%.tiff}.ome.tiff"
    cp "$dest" "$dest_ometiff"
    cp "$dest2" "$dest2_ometiff"

    bfomexml="$(showinf -nopix -noflat -omexml -omexml-only "$srcfile")"

    # Add TiffData elements.
    uuid="$(makeuuid)"
    ome_attr="Creator=\"makepyramid-ptif\" UUID=\"${uuid}\""
    tiffdata_fmt="<TiffData FirstC=\"0\" FirstT=\"0\" FirstZ=\"0\" IFD=\"%d\" PlaneCount=\"1\"><UUID FileName=\"$(basename "${dest_ometiff}")\">${uuid}</UUID></TiffData>"
    tiffdata_fmt_flat="<TiffData FirstC=\"0\" FirstT=\"0\" FirstZ=\"0\" IFD=\"%d\" PlaneCount=\"1\"><UUID FileName=\"$(basename "${dest2_ometiff}")\">${uuid}</UUID></TiffData>"

    omexml_fmt="$(echo "$bfomexml" | sed -e "s;\(<OME.*\)\(\">\);\1\" ${ome_attr}>;" -e "s;<MetadataOnly\/>;${tiffdata_fmt};")"
    omexml_fmt_flat="$(echo "$bfomexml" | sed -e "s;\(<OME.*\)\(\">\);\1\" ${ome_attr}>;" -e "s;<MetadataOnly\/>;${tiffdata_fmt_flat};")"

# assumption: one real IFD per file - original was based on SVS file which has label and macro image as well
#    ifds="0 1 2"
    ifds="0"
    omexml="$(printf "$omexml_fmt" $ifds)"

    flatifds="$(echo $mainifd $otherifds)"
    omexml_flat="$(printf "$omexml_fmt" $flatifds)"

    tiffset -d 0 -s 270 "$omexml" "$dest_ometiff"
    tiffset -d 0 -s 270 "$omexml_flat" "$dest2_ometiff"
done