[ome-users] Micromanager metadata and time increments

Wed Nov 21 21:14:33 GMT 2018

Dear Jens,

thanks for uploading a representative dataset. The issue was associated with
the parsing of invalid double values. It turns out this bug has been identified and
addressed in the development branch of Bio-Format 6 [1].

We will add your sample dataset to your QA repository to ensure we encounter
no future regression. Please let us know if it would be possible to make this data
publicly available under a CC-BY license [2]. Otherwise we will keep it private.

Best regards,
Sebastien

[1] https://github.com/openmicroscopy/bioformats/pull/3253
[2] http://downloads.openmicroscopy.org/images/Micro-Manager/

On 20 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com<mailto:jenseri at gmail.com>> wrote:

Sure thing, i have uploaded a dataset from Mircro-Manager 1-4-23-20180923 (ID:27101)
I also submitted the _metadata.txt import errorlog (ID: 27100)

Keep up the good work!

Best,

Jens

On Tue, Nov 20, 2018 at 11:35 AM Sebastien Besson (Staff) <s.besson at dundee.ac.uk<mailto:s.besson at dundee.ac.uk>> wrote:
Dear Jens,

thanks for reporting the error. The stack trace looks very similar to what has been
reported on the image.sc<http://image.sc/> forum [1] as well as the Micro-Manager mailing list [2].

For Micro-Manager 2.0.0-beta3 datasets, currently unsupported by Bio-Formats, we
are aware that the metadata format has been through several breaking changes,
including the IJType field which is the cause of the Bio-Formats parsing failure below.

For Micro-Manager 1.4.23 datasets, it might be this constitutes a variant of the
Micro-Manager 1.4 metadata format [3]. But we have no representative datasets
showing the issue so far. Would it be possible for you to upload sampled data at
http://qa.openmicroscopy.org.uk/qa/upload/?

Best,
Sebastien

[1] https://forum.image.sc/t/exception-thrown-when-importing-micro-manager-dataset-with-bioformats/11068
[2] http://micro-manager.3463995.n2.nabble.com/Exception-thrown-when-importing-Micro-Manager-2-0-dataset-with-Bioformats-td7589436.html
[3] https://micro-manager.org/wiki/Files_and_Metadata

On 19 Nov 2018, at 15:13, Jens Eriksson <jenseri at gmail.com<mailto:jenseri at gmail.com>> wrote:

I have been saving the metadata.txt with the ome-tifs, but now I realize there has been an unnoticed error at the import step. We normally import the folders and not the files directly, so there has not been any indications of this error to the user. If I import ome-tif + metadata.txt files directly, I discovered that the metadata.txt files are not processed correctly by BioFormats. The following error is logged in the OMERO-insight64 importer:

java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
at loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)

at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
at ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
at org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
at loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
... 16 more

This error is given by both MM 1.4.23-nightly & MM 2.0-beta3-nightly metadata.txt files. ome-tifs from these import without errors.

Best,

Jens

On Thu, Nov 15, 2018 at 3:13 PM Sebastien Besson (Staff) <s.besson at dundee.ac.uk<mailto:s.besson at dundee.ac.uk>> wrote:
Hi Jens,

On 13 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com<mailto:jenseri at gmail.com>> wrote:
 <snip>

I recently finished implementing an OMERO.server with a OMERO.web interface, together with the OMERO.figure & OMERO.iviewer apps (good job BTW, users and PIs LOVE these!).

Thanks for the positive feedback. Always appreciated.

However, neither OMERO.viewer or OMERO.figure display correct time points on timelapse data we generate.
It seems like OMERO/Bio-Formats is not correctly reading (or utilizing) TimeIncrement & TimeIncrementUnit from Micro-Manager generated files, even though the deltaT value for each image is displayed correctly.
We are running both MM 1.4.23-nightly and 2.0-beta-nightly to control our microscope, depending on user preference.
I occasionally upgrade the nightlies, but the problems I describe are common for all versions tested.
How can I get correct TimeIncrement & TimeIncrementUnit values set for my existing Micromanager generated data in the OMERO.server/web environment?
Question 2:
During the setup of the MM configuration files I have been very systematic and thorough, so that all relevant data (filter bandpass, product numbers, objective NAs, etc...) is captured in the Micro-Manager metadata through informative device and property names.
As a facility manager it would be very helpful for me and my users to have easy access to this data through OMERO.

There are different types of metadata and file representations created by Micro-Manager. For
versions 1.4.x, we tried to summarise the representation of the various scenarios in the table
at the bottom of this documentation page [1].

As you mentioned Micro-Manager captures extensive metadata which is stored within a TIFF tag.
This metadata can optionally be saved as a separate text file. Additionally, a subset is also
converted into OME metadata and stored in the ImageDescription tag as per the OME-TIFF
specification.

To maximise the acquisition metadata, our current recommendation is to create companion files
at acquisition containing the MM metadata and use this file when importing the data into OMERO.
This should result in the data being available as Original Metadata in the right-hand panel - see
screenshots below for an example with a sample dataset [2] created with the demo configuration.

[1] http://docs.openmicroscopy.org/bio-formats/5.9.2/users/micromanager/index.html
[2] http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/

[X]

Is there some plugin/app/script that makes the Micro-Manager metadata accessible on the OMERO.server, or at least viewable from the web interface?
If not, I can try to implement one. I have spent some time reading through tif-tags and MM metadata for other projects, but there is no need to reinvent the wheel if there is something out there.

If you cannot get the data in the representation described above or if you have data already
imported, the expectationfrom the Micro-Manager file format specification [3] is that the
extra metadata should be stored in the private TIFF tag 51123.

[3] https://micro-manager.org/wiki/Micro-Manager_File_Formats#Image_File_Directories

Comments:

It has always annoyed me that MM ome-tiff files won't display a correct frame interval when drag&dropped into ImageJ/FIJI.
On the other hand, if images are instead opened through the Bio-Formats Importer, then they loose their LUTs (but gain the lost frame intervals).
The lost LUTs is also a (minor) issue when importing MM data to OMERO.server.
I suspect that this is because ImageJ has its own TIF-tag that stores imageJ-metadata, and that there is some inconsistencies in how MM writes the tif tags here.
Anyway, thanks for great softwares everyone, let's make them even better!

All agreed. Micro-manager datasets are certainly a very interesting example of imaging data
where the overall metadata is stored according to various models (OME, ImageJ, MM-JSON)
and in different places. Preliminary discussion have been engaged on how to reconcile these
various sources of metadata when reading the files [4]. In the short run, the suggestion described
above is the one that hopefully should offer the maximal preservation of the original metadata.

[4] https://github.com/openmicroscopy/bioformats/pull/2213

Best,
Sebastien

Dr. Jens Eriksson, manager
Sellin Imaging Platform
Dep. Medical Biohecmistry and Microbiology
Uppsala University
Sweden
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
<Screen Shot 2018-11-15 at 13.56.07.png>_______________________________________________

ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181121/88cc1618/attachment.html>