[ome-users] Bug report for version 2012-06 OME-XML in Fiji, seen in grid/collection stitching
Michael Pinkert
mpinkert at wisc.edu
Thu Nov 10 15:46:53 GMT 2016
Hi Curtis,
For the Grid/collection stitching plugin I used the parameters
Type: Positions from file
Order: Defined by image metadata
In the next pop up screen I have all options unchecked. Here is the remainder:
Fusion method: Linear blending
Regression Threshold: 0.30
Max/avg displacement threshold: 2.50
Absolute displacement threshold: 3.50
Computation Parameters: Save memory (but be slower)
Image output: Fuse and display
For the Fiji + Java, I have ImageJ 2.0.0-rc-54/1.51g and for Java I have 1.8.0_66
Best regards,
Michael Pinkert
From: <ctrueden.wisc at gmail.com> on behalf of Curtis Rueden <ctrueden at wisc.edu>
Date: Thursday, November 10, 2016 at 8:34 AM
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk>
Cc: Michael Pinkert <mpinkert at wisc.edu>
Subject: Re: [ome-users] Bug report for version 2012-06 OME-XML in Fiji, seen in grid/collection stitching
Hi Roger,
> I've done some further testing with the submitted files.
Thank you very much for the further testing.
> Workarounds for broken data should never compromise the handling of
> valid data, and at some point there is a limit for what it is
> reasonable or safe for us to correct.
I absolutely agree 100%. However, two points in this case:
1) I believe these files were written by Bio-Formats itself. I would argue that under no circumstances should Bio-Formats write OME-TIFFs which are later considered to be invalid or corrupt. Even dev builds of Bio-Formats should use a stable schema version by default. If it is needed for testing to write files with "in-progress" schemas, that option should be specified via a manually triggered API call, rather than happen by default.
2) For files with an "in-progress" schema version, making a best effort to parse using nearby stable schema versions does not introduce any possibility of compromise to valid data handling. So I am glad to hear that Bio-Formats does this.
> I'd be interested to know why it wasn't working for you with your
> Fiji/ImageJ installation.
Michael: can you describe the exact steps you took? Grid/Collection Stitching? Which parameters? Which version of Fiji + Java are you using? Click the status bar to see.
Regards,
Curtis
--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/
On Thu, Nov 10, 2016 at 7:32 AM, Roger Leigh <rleigh at dundee.ac.uk<mailto:rleigh at dundee.ac.uk>> wrote:
Dear all,
I've done some further testing with the submitted files. They are
technically invalid (non-validating), but I can in practice I can:
- view them with showinf,
- convert them to 2016-06 OME-TIFF with bfconvert (fixing the validation
errors)
- import into current Fiji (with the java8 update site providing
bioformats 5.2.4)
None of these showed any errors for me.
So it looks like Bio-Formats is already doing the "best effort"
transformation in this particular case. I'd be interested to know why
it wasn't working for you with your Fiji/ImageJ installation. Were you
using the current version of Bio-Formats (5.2.4) with the Java8 update site?
I would highly recommend running bfconvert on these files to transform
them into valid OME-TIFF data.
A final comment about this type of "best effort" handling of invalid
data. Bio-Formats already makes significant efforts to cope with
various different types of invalid data and metadata so that broken,
damaged and incorrect files can be read. However, this does come at a
cost. As an example, a few weeks ago a fix to handle broken TiffData
elements in OME-TIFF was found to corrupt *valid* OME-TIFF data due to
certain assumptions being made about the structure of the data. That
is, it altered the original (and correct) intent of the data.
Workarounds for broken data should never compromise the handling of
valid data, and at some point there is a limit for what it is reasonable
or safe for us to correct. Broken data must at some point result in a
hard failure. Not notifying the user that the data is incorrect is
helpful from one point of view (getting the data read so we can get work
done), but longer term it masks serious problems from both users and
developers and delays getting them fixed. And it can also result in
software using Bio-Formats unknowingly creating buggy OME-TIFFs; the
workarounds result in problems not being picked up during testing
because it appears to "work" while actually being broken. This is bad
for interoperability, and also for the complexity of the OME-TIFF
reader, which then needs to permanently retain a whole series of
workarounds for every buggy file in existence rather than simply
implementing the standard, and that burden is already unreasonably high.
In this particular case, the workaround is (by chance) safe and
functional, but development versions of the schema should never have
been used for writing OME-TIFF; only published finalised versions of the
schema are suitable for use in the OME-TIFF file format, and trying to
use development versions for long term archival is asking for
trouble--we make no guarantees these can be read back in since
incompatible changes to the schema before release would result in
untransformable data. The transforms are only provided for and tested
with published released versions of the schema.
Regards,
Roger
On 09/11/16 20:17, Curtis Rueden wrote:
Hi,
Michael: This problem would occur if WiscScan was ever deployed to use a
non-release version of Bio-Formats. Ask the WiscScan programmers to
search the Git commit history for changes to the Bio-Formats JAR files,
to see which versions were committed when.
OME team: it would be nice if Bio-Formats could make a "best effort" to
transform these sorts of files, by attempting to treat them as one or
more released schema versions before giving up on the XSLT forward
transform.
Regards,
Curtis
--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
<http://imagej.net/User:Rueden>
Did you know ImageJ has a forum? http://forum.imagej.net/
On Wed, Nov 9, 2016 at 2:03 PM, Michael Pinkert <mpinkert at wisc.edu<mailto:mpinkert at wisc.edu>
<mailto:mpinkert at wisc.edu<mailto:mpinkert at wisc.edu>>> wrote:
Thanks for solving that Roger!
That solution makes sense, though it is rather confusing that we
would be encountering this problem now. We've been using the
software for years without problems, but I guess an update either
messed up the references or made the improper reference a
requirement when it wasn't before.
I'll have to talk to the more knowledgeable coders on this issue,
and look around to see if other people are still having this problem.
Best regards,
Michael Pinkert
-----Original Message-----
From: ome-users
[mailto:ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>
<mailto:ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>>] On Behalf Of
Roger Leigh
Sent: Wednesday, November 9, 2016 4:20 AM
To: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
<mailto:ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Bug report for version 2012-06 OME-XML in
Fiji, seen in grid/collection stitching
On 08/11/16 19:50, Michael Pinkert wrote:
> Hi Mark,
>
> I just uploaded 9 files, from a much smaller sample image set. I
tried using the ImageJ/Fiji stitching algorithm again with this
smaller sample size and it ran into the same problem, with the same
warning. It seems not to be reading the metadata correctly, because
it was interpreting different Z-slices as different T-time points.
>
> This problem occurred after a recent Fiji/ImageJ update and
applies to older sets of images that I've already stitched; I don't
think there's anything wrong with the formatting of the images.
Dear Michael,
I've taken a look at your files, and the problem is that the XML
schema is invalid. Using bftools' xmlvalid program:
% ./tools/xmlvalid ~/images/17416/FijiTester.ome.xml Parsing schema
path
https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
Validating /home/rleigh/images/17416/FijiTester.ome.xml
Error parsing schema at
https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
org.xml.sax.SAXParseException: schema_reference.4: Failed to read
schema document
'https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>',
because 1) could not find the document; 2) the document could not be
read; 3) the root element of the document is not <xsd:schema>.
% ./tools/xmlvalid ~/images/17416/FijiTester_C0_TP0_SP0_FW0.ome.tiff
Parsing schema path
https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
Validating /home/rleigh/images/17416/FijiTester_C0_TP0_SP0_FW0.ome.tiff
Error parsing schema at
https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
org.xml.sax.SAXParseException: schema_reference.4: Failed to read
schema document
'https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>',
because 1) could not find the document; 2) the document could not be
read; 3) the root element of the document is not <xsd:schema>.
The schema which is being used here is an invalid URL, and was never
a released schema. I'm not sure why any software would be using that.
You can find sample files using the old 2012-06 schema here:
http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/
<http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/>
They are using a root element like this:
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
<http://www.openmicroscopy.org/Schemas/OME/2012-06>"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
<http://www.w3.org/2001/XMLSchema-instance>"
xmlns:ROI="http://www.openmicroscopy.org/Schemas/ROI/2012-06
<http://www.openmicroscopy.org/Schemas/ROI/2012-06>"
xmlns:SA="http://www.openmicroscopy.org/Schemas/SA/2012-06
<http://www.openmicroscopy.org/Schemas/SA/2012-06>"
xmlns:SPW="http://www.openmicroscopy.org/Schemas/SPW/2012-06
<http://www.openmicroscopy.org/Schemas/SPW/2012-06>"
xmlns:Bin="http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06
<http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06>"
xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06 <http://www.openmicroscopy.org/Schemas/OME/2012-06>
http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
<http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd>">
while your samples do this:
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
<http://www.openmicroscopy.org/Schemas/OME/2012-06>"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
<http://www.w3.org/2001/XMLSchema-instance>"
xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06 <http://www.openmicroscopy.org/Schemas/OME/2012-06>
https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
<https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>">
It's the schemaLocation which is wrong, making the file invalid. If
you fix that up, and the rest of the file is valid for the 2012-06
schema, it should then work correctly.
Kind regards,
Roger
--
Dr Roger Leigh -- Open Microscopy Environment Wellcome Trust Centre
for Gene Regulation and Expression, College of Life Sciences,
University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No:
SC015096 _______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
<mailto:ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
<http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
<mailto:ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
<http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20161110/28fb1472/attachment.html>
More information about the ome-users
mailing list