[ome-users] Bug report for version 2012-06 OME-XML in Fiji, seen in grid/collection stitching
Roger Leigh
rleigh at dundee.ac.uk
Thu Nov 10 13:32:58 GMT 2016
Dear all,
I've done some further testing with the submitted files. They are
technically invalid (non-validating), but I can in practice I can:
- view them with showinf,
- convert them to 2016-06 OME-TIFF with bfconvert (fixing the validation
errors)
- import into current Fiji (with the java8 update site providing
bioformats 5.2.4)
None of these showed any errors for me.
So it looks like Bio-Formats is already doing the "best effort"
transformation in this particular case. I'd be interested to know why
it wasn't working for you with your Fiji/ImageJ installation. Were you
using the current version of Bio-Formats (5.2.4) with the Java8 update site?
I would highly recommend running bfconvert on these files to transform
them into valid OME-TIFF data.
A final comment about this type of "best effort" handling of invalid
data. Bio-Formats already makes significant efforts to cope with
various different types of invalid data and metadata so that broken,
damaged and incorrect files can be read. However, this does come at a
cost. As an example, a few weeks ago a fix to handle broken TiffData
elements in OME-TIFF was found to corrupt *valid* OME-TIFF data due to
certain assumptions being made about the structure of the data. That
is, it altered the original (and correct) intent of the data.
Workarounds for broken data should never compromise the handling of
valid data, and at some point there is a limit for what it is reasonable
or safe for us to correct. Broken data must at some point result in a
hard failure. Not notifying the user that the data is incorrect is
helpful from one point of view (getting the data read so we can get work
done), but longer term it masks serious problems from both users and
developers and delays getting them fixed. And it can also result in
software using Bio-Formats unknowingly creating buggy OME-TIFFs; the
workarounds result in problems not being picked up during testing
because it appears to "work" while actually being broken. This is bad
for interoperability, and also for the complexity of the OME-TIFF
reader, which then needs to permanently retain a whole series of
workarounds for every buggy file in existence rather than simply
implementing the standard, and that burden is already unreasonably high.
In this particular case, the workaround is (by chance) safe and
functional, but development versions of the schema should never have
been used for writing OME-TIFF; only published finalised versions of the
schema are suitable for use in the OME-TIFF file format, and trying to
use development versions for long term archival is asking for
trouble--we make no guarantees these can be read back in since
incompatible changes to the schema before release would result in
untransformable data. The transforms are only provided for and tested
with published released versions of the schema.
Regards,
Roger
On 09/11/16 20:17, Curtis Rueden wrote:
> Hi,
>
> Michael: This problem would occur if WiscScan was ever deployed to use a
> non-release version of Bio-Formats. Ask the WiscScan programmers to
> search the Git commit history for changes to the Bio-Formats JAR files,
> to see which versions were committed when.
>
> OME team: it would be nice if Bio-Formats could make a "best effort" to
> transform these sorts of files, by attempting to treat them as one or
> more released schema versions before giving up on the XSLT forward
> transform.
>
> Regards,
> Curtis
>
> --
> Curtis Rueden
> LOCI software architect - http://loci.wisc.edu/software
> ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
> <http://imagej.net/User:Rueden>
> Did you know ImageJ has a forum? http://forum.imagej.net/
>
>
> On Wed, Nov 9, 2016 at 2:03 PM, Michael Pinkert <mpinkert at wisc.edu
> <mailto:mpinkert at wisc.edu>> wrote:
>
> Thanks for solving that Roger!
>
> That solution makes sense, though it is rather confusing that we
> would be encountering this problem now. We've been using the
> software for years without problems, but I guess an update either
> messed up the references or made the improper reference a
> requirement when it wasn't before.
>
> I'll have to talk to the more knowledgeable coders on this issue,
> and look around to see if other people are still having this problem.
>
> Best regards,
> Michael Pinkert
>
> -----Original Message-----
> From: ome-users
> [mailto:ome-users-bounces at lists.openmicroscopy.org.uk
> <mailto:ome-users-bounces at lists.openmicroscopy.org.uk>] On Behalf Of
> Roger Leigh
> Sent: Wednesday, November 9, 2016 4:20 AM
> To: ome-users at lists.openmicroscopy.org.uk
> <mailto:ome-users at lists.openmicroscopy.org.uk>
> Subject: Re: [ome-users] Bug report for version 2012-06 OME-XML in
> Fiji, seen in grid/collection stitching
>
> On 08/11/16 19:50, Michael Pinkert wrote:
> > Hi Mark,
> >
> > I just uploaded 9 files, from a much smaller sample image set. I
> tried using the ImageJ/Fiji stitching algorithm again with this
> smaller sample size and it ran into the same problem, with the same
> warning. It seems not to be reading the metadata correctly, because
> it was interpreting different Z-slices as different T-time points.
> >
> > This problem occurred after a recent Fiji/ImageJ update and
> applies to older sets of images that I've already stitched; I don't
> think there's anything wrong with the formatting of the images.
>
> Dear Michael,
>
> I've taken a look at your files, and the problem is that the XML
> schema is invalid. Using bftools' xmlvalid program:
>
> % ./tools/xmlvalid ~/images/17416/FijiTester.ome.xml Parsing schema
> path
> https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
> Validating /home/rleigh/images/17416/FijiTester.ome.xml
> Error parsing schema at
> https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
> org.xml.sax.SAXParseException: schema_reference.4: Failed to read
> schema document
> 'https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>',
> because 1) could not find the document; 2) the document could not be
> read; 3) the root element of the document is not <xsd:schema>.
>
> % ./tools/xmlvalid ~/images/17416/FijiTester_C0_TP0_SP0_FW0.ome.tiff
> Parsing schema path
> https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
> Validating /home/rleigh/images/17416/FijiTester_C0_TP0_SP0_FW0.ome.tiff
> Error parsing schema at
> https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>
> org.xml.sax.SAXParseException: schema_reference.4: Failed to read
> schema document
> 'https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>',
> because 1) could not find the document; 2) the document could not be
> read; 3) the root element of the document is not <xsd:schema>.
>
>
> The schema which is being used here is an invalid URL, and was never
> a released schema. I'm not sure why any software would be using that.
>
> You can find sample files using the old 2012-06 schema here:
> http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/
> <http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/>
> They are using a root element like this:
>
> <OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
> <http://www.openmicroscopy.org/Schemas/OME/2012-06>"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> <http://www.w3.org/2001/XMLSchema-instance>"
> xmlns:ROI="http://www.openmicroscopy.org/Schemas/ROI/2012-06
> <http://www.openmicroscopy.org/Schemas/ROI/2012-06>"
> xmlns:SA="http://www.openmicroscopy.org/Schemas/SA/2012-06
> <http://www.openmicroscopy.org/Schemas/SA/2012-06>"
> xmlns:SPW="http://www.openmicroscopy.org/Schemas/SPW/2012-06
> <http://www.openmicroscopy.org/Schemas/SPW/2012-06>"
>
> xmlns:Bin="http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06
> <http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06>"
>
> xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06 <http://www.openmicroscopy.org/Schemas/OME/2012-06>
> http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
> <http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd>">
>
> while your samples do this:
>
> <OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
> <http://www.openmicroscopy.org/Schemas/OME/2012-06>"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
> <http://www.w3.org/2001/XMLSchema-instance>"
>
> xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06 <http://www.openmicroscopy.org/Schemas/OME/2012-06>
> https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd
> <https://raw.github.com/openmicroscopy/openmicroscopy/schema-2012-06/components/specification/InProgress/ome.xsd>">
>
> It's the schemaLocation which is wrong, making the file invalid. If
> you fix that up, and the rest of the file is valid for the 2012-06
> schema, it should then work correctly.
>
>
> Kind regards,
> Roger
>
> --
> Dr Roger Leigh -- Open Microscopy Environment Wellcome Trust Centre
> for Gene Regulation and Expression, College of Life Sciences,
> University of Dundee, Dow Street,
> Dundee DD1 5EH Scotland UK Tel: (01382) 386364
>
> The University of Dundee is a registered Scottish Charity, No:
> SC015096 _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> <mailto:ome-users at lists.openmicroscopy.org.uk>
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> <mailto:ome-users at lists.openmicroscopy.org.uk>
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
> <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
>
>
>
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
More information about the ome-users
mailing list