[ome-devel] [ome-files-cpp] Cannot Write Multiple ROIs to Image Plane Using SaveBytes

[Dennis Ai]

Roger,

Thanks for the explanation.  Unfortunately, I am still running into the same problem.  I created the following test example (in my C# wrapper around OME-TIFF).

OMETIFFWriter tiffWriter = new OMETIFFWriter();
OMEXMLMetadata metadata = new OMEXMLMetadata();
CoreMetadata coreMetadata = new CoreMetadata();
PixelBuffer pixelBuffer = new PixelBuffer(512, 64, 1, 1, 1, "int8", "XYZTC");

coreMetadata.SizeX = 512;
coreMetadata.SizeY = 128;
coreMetadata.SizeZ = 1;
coreMetadata.SizeT = 1;
coreMetadata.SizeC = 1;
coreMetadata.PixelType = "int8";
coreMetadata.BitsPerPixel = 8;

metadata.SetCoreMetadata(new CoreMetadata[] { coreMetadata });
tiffWriter.FileMetadata = metadata;
tiffWriter.CompressionType = "Deflate";
tiffWriter.TileSizeX = 64;
tiffWriter.TileSizeY = 64;
            
tiffWriter.Open(TestTIFFFilePath);
CopyGradientToPixelBuffer(pixelBuffer); // This copies a gradient to the pixel buffer.
tiffWriter.Series = 0;
tiffWriter.SaveBytes(0, pixelBuffer, 0, 0, 512, 64); // This call works.
tiffWriter.SaveBytes(0, pixelBuffer, 0, 64, 512, 64); // This call does not work.

When I open the image, I get the attached image.  The first row of tiles is written (from the first saveBytes call).  The second row of tiles, when you open the image, is just a repeated version of the last tile in the first row.  My assumption, when I look at the file size of the image, is that the second row hasn't actually been written, and the last tile was just displayed for all unwritten tiles.

Also, can you point me to the documentation that states that the tiles must be written in linear order?  When I was previously working with libtiff (via LibTIFFNet), I don't recall having run into that limitation?  I could write tiles in any order I wanted.  Forcing the tiles to be stored in cache could be problematic for us, as in your example:

  ┌──┬──┐
A│01│23│B
  │45│67│
  ├──┼──┤
C│89│ab│D
  │cd│ef│
  └──┴──┘

We would like to write a whole slide image, but must do it in order of A-C-B-D, where A and C make up a strip, and B and D make up a 2nd strip.  Is there any way to force flushing?

Thanks,
Dennis

-----Original Message-----
From: ome-devel <ome-devel-bounces at lists.openmicroscopy.org.uk> On Behalf Of Roger Leigh
Sent: Wednesday, June 20, 2018 2:20 AM
To: ome-devel at lists.openmicroscopy.org.uk
Subject: Re: [ome-devel] [ome-files-cpp] Cannot Write Multiple ROIs to Image Plane Using SaveBytes

On 19/06/18 22:15, Dennis Ai wrote:
> Hello OME Team,
> 
> I am working on an image acquisition pipeline that writes to a tiled 
> OME-TIFF.  Let’s say for argument’s sake, my final image is 4096 x 
> 4096.  What I would like to do is:
> 
>  1. Write the first region (2048 x 2048) I acquired to the top-left of
>     the final image.  I call /saveBytes/ with a ROI defined as (0, 0,
>     2048, 2048) [x, y, width, height].  This works okay.
>  2. Write the second region (2048 x 2048) to the right of the first
>     region.  I call /saveBytes/ with a ROI defined as (2048, 0, 2048,
>     2048).  This does not get written.
> 
> I have also tried variations where I write to the bottom of the first 
> region, which doesn’t work either.  Is there a mechanism that allows 
> me to sequentially write regions of interest to the same plane?

Dear Dennis,

What you are describing should work and is explicitly tested for by the unit tests.  It's possible there is something which doesn't work as expected, but I would consider it unlikely.  I will describe how this works internally below.

When writing out a plane of pixel data into a TIFF IFD, it is expected that every pixel on the plane be written once only by a saveBytes call. 
The API doesn't mandate any ordering requirements, and it also doesn't mandate writing on tile boundaries.  However, libtiff does require the tiles be written in a specific linear order.  And writing complete tiles in that order will have some performance benefits.  We maintain a tile cache to retain tiles which can't yet be written or which are incomplete (only partially covered by pixels), plus an R*Tree coverage map to detect overlapping writes and completed tiles.

Let's take a simple image, 4096×4096 with 1024×1024 tiles:

  ┌──┬──┐
A│01│23│B
  │45│67│
  ├──┼──┤
C│89│ab│D
  │cd│ef│
  └──┴──┘

The numbers are the order in which tiles will be written to the TIFF file. I've blocked it into the 2048×2048 regions you were writing, which I've labelled A-D.

If we write in the order A, B, C, D, the tiles will be flushed to disk in this order:

A: 01
B: 234567
C: 89
D: abcdef

This is because when we write A, it's only possible to flush two tiles. 
When we write B, it's possible to flush six.  The same pattern repeats for C and D.

If we write in reverse order B, C, B, A, we get:

D:
C:
B:
A: 01234567abcdef

This is because it's not possible to flush the tile cache until the very last saveBytes call, due to the ordering requirements.

For a different order, B, A, D, C:

B:
A: 01234567
D:
C: 89abcdef

You can see that the tile cache will be flushed to the maximum extent possible after each saveBytes call.  But the key point that I want to make is that the tile cache won't be flushed if the coverage map isn't complete, i.e. some pixels were missing from the full plane.

For your example (2): (2048, 0, 2048, 2048), did you also follow up with
   (0, 0, 2048, 2048)
   (0, 2048, 2048, 2048)
   (2048, 2048, 2048, 2048)
saveBytes calls?  If you did not, then that would possibly explain why nothing was written.  It's a requirement that the plane be completely written.  TIFF does not support sparse planes.  At least, not for a conforming valid TIFF file.


I hope that helps, and if not then it might be useful to see some of your code or a more complete description of what you are trying to accomplish.


Kind regards,
Roger
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openmicroscopy.org.uk%2Fmailman%2Flistinfo%2Fome-devel&data=01%7C01%7Cdennis.ai%40sri.com%7C207140ce91dd407ecc8408d5d68ef609%7C40779d3379c44626b8bf140c4d5e9075%7C1&sdata=KYCD5MKgmBOQ1fnoTthmxlzXN9wlKryt24iF3Hd%2BqK0%3D&reserved=0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 5dtest.ome.tiff
Type: image/tiff
Size: 4184 bytes
Desc: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20180620/a6f0ab20/attachment.tiff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5124 bytes
Desc: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20180620/a6f0ab20/attachment.p7s>