[ome-users] Bug report for version 2012-06 OME-XML in Fiji, seen in grid/collection stitching
Curtis Rueden
ctrueden at wisc.edu
Thu Nov 10 14:34:16 GMT 2016
Hi Roger,
> I've done some further testing with the submitted files.
Thank you very much for the further testing.
> Workarounds for broken data should never compromise the handling of
> valid data, and at some point there is a limit for what it is
> reasonable or safe for us to correct.
I absolutely agree 100%. However, two points in this case:
1) I believe these files were written by Bio-Formats itself. I would argue
that under no circumstances should Bio-Formats write OME-TIFFs which are
later considered to be invalid or corrupt. Even dev builds of Bio-Formats
should use a stable schema version by default. If it is needed for testing
to write files with "in-progress" schemas, that option should be specified
via a manually triggered API call, rather than happen by default.
2) For files with an "in-progress" schema version, making a best effort to
parse using nearby stable schema versions does not introduce any
possibility of compromise to valid data handling. So I am glad to hear that
Bio-Formats does this.
> I'd be interested to know why it wasn't working for you with your
> Fiji/ImageJ installation.
Michael: can you describe the exact steps you took? Grid/Collection
Stitching? Which parameters? Which version of Fiji + Java are you using?
Click the status bar to see.
Regards,
Curtis
--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/
On Thu, Nov 10, 2016 at 7:32 AM, Roger Leigh <rleigh at dundee.ac.uk> wrote:
> Dear all,
>
> I've done some further testing with the submitted files. They are
> technically invalid (non-validating), but I can in practice I can:
>
> - view them with showinf,
> - convert them to 2016-06 OME-TIFF with bfconvert (fixing the validation
> errors)
> - import into current Fiji (with the java8 update site providing
> bioformats 5.2.4)
>
> None of these showed any errors for me.
>
> So it looks like Bio-Formats is already doing the "best effort"
> transformation in this particular case. I'd be interested to know why
> it wasn't working for you with your Fiji/ImageJ installation. Were you
> using the current version of Bio-Formats (5.2.4) with the Java8 update
> site?
>
> I would highly recommend running bfconvert on these files to transform
> them into valid OME-TIFF data.
>
> A final comment about this type of "best effort" handling of invalid
> data. Bio-Formats already makes significant efforts to cope with
> various different types of invalid data and metadata so that broken,
> damaged and incorrect files can be read. However, this does come at a
> cost. As an example, a few weeks ago a fix to handle broken TiffData
> elements in OME-TIFF was found to corrupt *valid* OME-TIFF data due to
> certain assumptions being made about the structure of the data. That
> is, it altered the original (and correct) intent of the data.
> Workarounds for broken data should never compromise the handling of
> valid data, and at some point there is a limit for what it is reasonable
> or safe for us to correct. Broken data must at some point result in a
> hard failure. Not notifying the user that the data is incorrect is
> helpful from one point of view (getting the data read so we can get work
> done), but longer term it masks serious problems from both users and
> developers and delays getting them fixed. And it can also result in
> software using Bio-Formats unknowingly creating buggy OME-TIFFs; the
> workarounds result in problems not being picked up during testing
> because it appears to "work" while actually being broken. This is bad
> for interoperability, and also for the complexity of the OME-TIFF
> reader, which then needs to permanently retain a whole series of
> workarounds for every buggy file in existence rather than simply
> implementing the standard, and that burden is already unreasonably high.
> In this particular case, the workaround is (by chance) safe and
> functional, but development versions of the schema should never have
> been used for writing OME-TIFF; only published finalised versions of the
> schema are suitable for use in the OME-TIFF file format, and trying to
> use development versions for long term archival is asking for
> trouble--we make no guarantees these can be read back in since
> incompatible changes to the schema before release would result in
> untransformable data. The transforms are only provided for and tested
> with published released versions of the schema.
>
>
> Regards,
> Roger
>
> On 09/11/16 20:17, Curtis Rueden wrote:
>
>> Hi,
>>
>> Michael: This problem would occur if WiscScan was ever deployed to use a
>> non-release version of Bio-Formats. Ask the WiscScan programmers to
>> search the Git commit history for changes to the Bio-Formats JAR files,
>> to see which versions were committed when.
>>
>> OME team: it would be nice if Bio-Formats could make a "best effort" to
>> transform these sorts of files, by attempting to treat them as one or
>> more released schema versions before giving up on the XSLT forward
>> transform.
>>
>> Regards,
>> Curtis
>>
>> --
>> Curtis Rueden
>> LOCI software architect - http://loci.wisc.edu/software
>> ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
>> <http://imagej.net/User:Rueden>
>> Did you know ImageJ has a forum? http://forum.imagej.net/
>>
>>
>> On Wed, Nov 9, 2016 at 2:03 PM, Michael Pinkert <mpinkert at wisc.edu
>> <mailto:mpinkert at wisc.edu>> wrote:
>>
>> Thanks for solving that Roger!
>>
>> That solution makes sense, though it is rather confusing that we
>> would be encountering this problem now. We've been using the
>> software for years without problems, but I guess an update either
>> messed up the references or made the improper reference a
>> requirement when it wasn't before.
>>
>> I'll have to talk to the more knowledgeable coders on this issue,
>> and look around to see if other people are still having this problem.
>>
>> Best regards,
>> Michael Pinkert
>>
>> -----Original Message-----
>> From: ome-users
>> [mailto:ome-users-bounces at lists.openmicroscopy.org.uk
>> <mailto:ome-users-bounces at lists.openmicroscopy.org.uk>] On Behalf Of
>> Roger Leigh
>> Sent: Wednesday, November 9, 2016 4:20 AM
>> To: ome-users at lists.openmicroscopy.org.uk
>> <mailto:ome-users at lists.openmicroscopy.org.uk>
>> Subject: Re: [ome-users] Bug report for version 2012-06 OME-XML in
>> Fiji, seen in grid/collection stitching
>>
>> On 08/11/16 19:50, Michael Pinkert wrote:
>> > Hi Mark,
>> >
>> > I just uploaded 9 files, from a much smaller sample image set. I
>> tried using the ImageJ/Fiji stitching algorithm again with this
>> smaller sample size and it ran into the same problem, with the same
>> warning. It seems not to be reading the metadata correctly, because
>> it was interpreting different Z-slices as different T-time points.
>> >
>> > This problem occurred after a recent Fiji/ImageJ update and
>> applies to older sets of images that I've already stitched; I don't
>> think there's anything wrong with the formatting of the images.
>>
>> Dear Michael,
>>
>> I've taken a look at your files, and the problem is that the XML
>> schema is invalid. Using bftools' xmlvalid program:
>>
>> % ./tools/xmlvalid ~/images/17416/FijiTester.ome.xml Parsing schema
>> path
>> https://raw.github.com/openmicroscopy/openmicroscopy/schema-
>> 2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>
>> Validating /home/rleigh/images/17416/FijiTester.ome.xml
>> Error parsing schema at
>> https://raw.github.com/openmicroscopy/openmicroscopy/schema-
>> 2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>
>> org.xml.sax.SAXParseException: schema_reference.4: Failed to read
>> schema document
>> 'https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>',
>> because 1) could not find the document; 2) the document could not be
>> read; 3) the root element of the document is not <xsd:schema>.
>>
>> % ./tools/xmlvalid ~/images/17416/FijiTester_C0_TP0_SP0_FW0.ome.tiff
>> Parsing schema path
>> https://raw.github.com/openmicroscopy/openmicroscopy/schema-
>> 2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>
>> Validating /home/rleigh/images/17416/Fiji
>> Tester_C0_TP0_SP0_FW0.ome.tiff
>> Error parsing schema at
>> https://raw.github.com/openmicroscopy/openmicroscopy/schema-
>> 2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>
>> org.xml.sax.SAXParseException: schema_reference.4: Failed to read
>> schema document
>> 'https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>',
>> because 1) could not find the document; 2) the document could not be
>> read; 3) the root element of the document is not <xsd:schema>.
>>
>>
>> The schema which is being used here is an invalid URL, and was never
>> a released schema. I'm not sure why any software would be using that.
>>
>> You can find sample files using the old 2012-06 schema here:
>> http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/
>> <http://downloads.openmicroscopy.org/images/OME-TIFF/2012-06/>
>> They are using a root element like this:
>>
>> <OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
>> <http://www.openmicroscopy.org/Schemas/OME/2012-06>"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> <http://www.w3.org/2001/XMLSchema-instance>"
>> xmlns:ROI="http://www.openmicroscopy.org/Schemas/ROI/2012-06
>> <http://www.openmicroscopy.org/Schemas/ROI/2012-06>"
>> xmlns:SA="http://www.openmicroscopy.org/Schemas/SA/2012-06
>> <http://www.openmicroscopy.org/Schemas/SA/2012-06>"
>> xmlns:SPW="http://www.openmicroscopy.org/Schemas/SPW/2012-06
>> <http://www.openmicroscopy.org/Schemas/SPW/2012-06>"
>>
>> xmlns:Bin="http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06
>> <http://www.openmicroscopy.org/Schemas/BinaryFile/2012-06>"
>>
>> xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06
>> <http://www.openmicroscopy.org/Schemas/OME/2012-06>
>> http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd
>> <http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd>">
>>
>> while your samples do this:
>>
>> <OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2012-06
>> <http://www.openmicroscopy.org/Schemas/OME/2012-06>"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> <http://www.w3.org/2001/XMLSchema-instance>"
>>
>> xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2012-06
>> <http://www.openmicroscopy.org/Schemas/OME/2012-06>
>> https://raw.github.com/openmicroscopy/openmicroscopy/schema-
>> 2012-06/components/specification/InProgress/ome.xsd
>> <https://raw.github.com/openmicroscopy/openmicroscopy/schema
>> -2012-06/components/specification/InProgress/ome.xsd>">
>>
>> It's the schemaLocation which is wrong, making the file invalid. If
>> you fix that up, and the rest of the file is valid for the 2012-06
>> schema, it should then work correctly.
>>
>>
>> Kind regards,
>> Roger
>>
>> --
>> Dr Roger Leigh -- Open Microscopy Environment Wellcome Trust Centre
>> for Gene Regulation and Expression, College of Life Sciences,
>> University of Dundee, Dow Street,
>> Dundee DD1 5EH Scotland UK Tel: (01382) 386364
>>
>> The University of Dundee is a registered Scottish Charity, No:
>> SC015096 _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> <mailto:ome-users at lists.openmicroscopy.org.uk>
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> <mailto:ome-users at lists.openmicroscopy.org.uk>
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>> <http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users>
>>
>>
>>
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>>
>
> --
> Dr Roger Leigh -- Open Microscopy Environment
> Wellcome Trust Centre for Gene Regulation and Expression,
> College of Life Sciences, University of Dundee, Dow Street,
> Dundee DD1 5EH Scotland UK Tel: (01382) 386364
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20161110/48c26890/attachment.html>
More information about the ome-users
mailing list