[ome-devel] Prairie View XML changes for upcoming v5.2 Release

Fri Oct 10 22:53:15 BST 2014

Hi Jimmy & everyone,

== SamplesPerPixel vs. Integration ==

Jimmy wrote:
> we often average 3 raw samples together to form 1 pixel value, and
> only write the single pixel value to Tiff.  We would like our Prairie
> XML file to still reflect SamplesPerPixel = 3 so users could go back
> and reference that value.

The term "SamplesPerPixel" has a very technical meaning in TIFF and
OME-XML: it is the number of sample values actually recorded in the file.
Since there is only one sample value recorded per pixel, SamplesPerPixel
must equal 1 for this data.

Conversely, what you describe is, I think, what the OME schema refers to as
"integration" [1]. It's not exactly "frame" averaging, but seems
conceptually equivalent. Andrew, would you agree?

== Prairie metadata parsing bugs ==

Upon further investigation, there were more bugs in the Prairie metadata
parsing relating to subindexed values. I think I have them all fixed now
[2]. I also added regression tests to ensure things work as expected [3].

== Prairie vs. OME-TIFF format detection ==

I wrote:
> if you select the .xml or .env file, the dataset will be detected as
> "Prairie TIFF" format, whereas if you select one of the .ome.tif
> files, it will be detected as "OME-TIFF" format.

Actually, there was an additional wrinkle. Selecting the first OME-TIFF
file (i.e., the one with a fully specified OME-XML block) detected as
OME-TIFF. But selecting any other OME-TIFF file (i.e., one that uses the
"BinaryOnly" mechanism to point to the first file) detected as a Prairie
TIFF. I have now fixed that bug [4], so that selecting any OME-TIFF file
results in the dataset being detected as OME-TIFF.

== SizeC inconsistency across multiple series ==

I wrote:
> I think it is indeed a bug in the Bio-Formats OME-TIFF reader that the
> first series is marked with "SizeC=1" but subsequent ones are "SizeC=5
> (effectively 1)". I will investigate if/how we can fix that on our
> end.

I did some digging, and found the issue [5]. It seems there is a hardcoded
"i == 0" test, which I am reluctant to change, since doing so might have
substantial ramifications. But I do still think we should somehow fix this
behavior.

Melissa, what do you think? I uploaded a sample dataset to QA [6].

== Support for spectral datasets ==

I wrote:
> Next on my list is to finish adding support for spectral datasets.
> Will keep everyone posted, and file a PR as soon as it is ready.

Done [7]. And PR filed [8].

Latest build available for testing from:
http://curtis.imagej.net/prairie-view-5/

Regards,
Curtis

[1]
https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#DetectorSettings_Integration

[2]
https://github.com/ctrueden/bioformats/commit/073ddc27a69fa089880e26c9bd53d5f6bac64d1b

[3]
https://github.com/ctrueden/bioformats/commit/7124247e55484b2f77ba8766d51455efd3f1d1e4

(Note to programmers: always write regression tests! I discovered and fixed
two additional bugs in the Prairie metadata logic, thanks to the creation
of these tests.)

[4]
https://github.com/ctrueden/bioformats/commit/a00d1251cd7bbfa6eef8329777f788c7d242df83

[5]
https://github.com/openmicroscopy/bioformats/blob/v5.0.5/components/formats-bsd/src/loci/formats/in/OMETiffReader.java#L573-L590

[6] http://qa.openmicroscopy.org.uk/qa/feedback/10297/

[7]
https://github.com/ctrueden/bioformats/commit/f9b6ac38bb7f71b2dc9cfc709b7321d9884c99a9

[8] https://github.com/openmicroscopy/bioformats/pull/1365

On Fri, Oct 10, 2014 at 2:53 PM, Fong, Jimmy <Jimmy.Fong at bruker-nano.com>
wrote:

>  Hi Curtis,
>
> We appreciate all your work.  Just as some clarification, the
> SamplesPerPixel field in our metadata corresponds to the number of samples
> (off of our acquisition hardware) that were already binned together to form
> the pixel value.  We do not provide the ability to write out each
> individual sample in the Tiff data, so it would make sense that the
> Bio-Formats Prairie TIFF reader would always set the SamplesPerPixel field
> in the OME metadata to 1, regardless of the value of SamplesPerPixels field
> in the Prairie metadata.
>
> For example, with our resonant scanner, we often average 3 raw samples
> together to form 1 pixel value, and only write the single pixel value to
> Tiff.  We would like our Prairie XML file to still reflect SamplesPerPixel
> = 3 so users could go back and reference that value.
>
> Thanks again and let me know if I'm just interpreting your email
> incorrectly.
>
> Regards,
> Jimmy
>
>  *Jimmy Fong*
>
> Software Engineer
>
>
>
> Bruker Nano Surfaces Division
> Jimmy.Fong at bruker-nano.com
>
> 3030 Laura Lane, Suite 140
> www.bruker.com/nano
> <https://remote.bruker-nano.com/itweb/www.bruker.com/nano>
>
> Middleton, WI 53562-0677
>
> Phone: +1 608-662-0022 x163
>  ------------------------------
>
> *We are now Bruker!*
>
> *Same great products.**  Same great technology.*
>
> *Same great people.**  New great name.*
>
>
>   ------------------------------
> *From:* ctrueden.wisc at gmail.com [ctrueden.wisc at gmail.com] on behalf of
> Curtis Rueden [ctrueden at wisc.edu]
> *Sent:* Thursday, October 09, 2014 5:21 PM
> *To:* Wussow, Mike; Fong, Jimmy
> *Cc:* j.r.swedlow at dundee.ac.uk; ome-devel at lists.openmicroscopy.org.uk;
> Staniszewski, Kevin
>
> *Subject:* Re: [ome-devel] Prairie View XML changes for upcoming v5.2
> Release
>
>   Hi Mike, Jimmy, Kevin and everyone,
>
>  > We'll test the reader when it's released and keep you posted about
> > what we find.
>
>  Thanks for getting back to me (via private mail) with your findings
> regarding the new PrairieView 5.2 file format reader. I am replying to the
> public thread to keep others in the loop regarding this work.
>
>
>  == Reading the files as a PrairieView dataset ==
>
>  It is important to note that if you select the .xml or .env file, the
> dataset will be detected as "Prairie TIFF" format, whereas if you select
> one of the .ome.tif files, it will be detected as "OME-TIFF" format.
> Completely separate reader code is used in each case. The recent changes I
> made to the PrairieReader [3] only affect the former case.
>
>  In short: please select the .xml or .env file if you wish to use the
> Bio-Formats Prairie TIFF reader, rather than the OME-TIFF reader.
>
>
>  == SamplesPerPixel issue when reading as OME-TIFF ==
>
>  Kevin Staniszewski wrote:
>  > 1. The "SizeC" attribute is incorrect for all but the first frame.
> > This acquisition only contained one channel, but every frame after the
> > first reports 5 channels. It appears that the number 5 is coming from
> > the "SamplesPerPixel" attribute, as in a separate data set, both
> > values reported 6 (using a slightly longer dwell time).
> >
>  > 2. The "SamplesPerPixel" attribute is incorrect for only the first
> > frame. The correct value for this acquisition is 5, but the first
> > frame reports a value of 1. The value of 1 may be coming from the
> > number of channels in the acquisition.
>
>  Actually, SamplesPerPixel needs to be set to 1 for the dataset you sent.
> From the OME-XML schema documentation [1]:
>
>      The number of samples the detector takes to form each pixel value.
>     [units:none] Note: This is not the same as "Frame Averaging" - see
>     Integration in DetectorSettings
>
>  SamplesPerPixel is the actual number of recorded sample values per
> pixel. If you use the "showinf" Bio-Formats command line tool on the
> dataset, the following warning is shown:
>
>      SamplesPerPixel mismatch: OME=5, TIFF=1
>
>  This indicates that the OME-XML indicates SamplesPerPixel=5, but the
> TIFF file only contains 1 sample value per pixel (as indicated by the IFD's
> SamplesPerPixel directory entry—which in this case is absent and hence
> defaults to 1 according to page 39 of the TIFF specification [2]).
>
>  All of that said: I think it is indeed a bug in the Bio-Formats OME-TIFF
> reader that the first series is marked with "SizeC=1" but subsequent ones
> are "SizeC=5 (effectively 1)". I will investigate if/how we can fix that on
> our end.
>
>
>  == Failure to extract (X, Y, Z) stage positions ==
>
>  Kevin Staniszewski wrote:
>   > The experiment that generated this data was a T series that took one
> > image at all XY locations. If I use the bio-formats importer to look
> > at the OME data, it shows up in a way such that each frame looks like
> > a different time point, instead of a different XY position.
>
>  Yes, there was a bug! Fixed now:
>
>
> https://github.com/ctrueden/bioformats/commit/ffa5e3071b2083cc0385032b6fdbcf40b3157271
>
>
>  == Spectral datasets ==
>
>  Next on my list is to finish adding support for spectral datasets. Will
> keep everyone posted, and file a PR as soon as it is ready.
>
>  Regards,
>  Curtis
>
>  [1]
> https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2013-06/ome_xsd.html#Channel_SamplesPerPixel
>
>  [2] http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf
>
>  [3] https://github.com/openmicroscopy/bioformats/pull/1306
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20141010/4912ac15/attachment.html>