[ome-users] Micromanager metadata and time increments

[Jens Eriksson]

I have been saving the metadata.txt with the ome-tifs, but now I realize
there has been an unnoticed error at the import step. We normally import
the folders and not the files directly, so there has not been any
indications of this error to the user. If I import ome-tif + metadata.txt
files directly, I discovered that the metadata.txt files are not processed
correctly by BioFormats. The following error is logged in the
OMERO-insight64 importer:

java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at
loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
at
loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at
ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
at
ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
at
org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at
org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at
org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at
org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at
org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at
org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at
org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at
org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)

at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
at
ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
at
ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
at
ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
at
org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
at
org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
at
org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
at
org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at
org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at
org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at
org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at
org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
at
org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at
loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
at
loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
at loci.formats.FormatReader.setId(FormatReader.java:1397)
at loci.formats.ImageReader.setId(ImageReader.java:842)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at loci.formats.Memoizer.setId(Memoizer.java:662)
at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
at
ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
... 16 more

This error is given by both MM 1.4.23-nightly & MM 2.0-beta3-nightly
metadata.txt files. ome-tifs from these import without errors.


Best,

Jens

On Thu, Nov 15, 2018 at 3:13 PM Sebastien Besson (Staff) <
s.besson at dundee.ac.uk> wrote:

> Hi Jens,
>
>
> On 13 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com> wrote:
>  <snip>
>
>
> I recently finished implementing an OMERO.server with a OMERO.web
> interface, together with the OMERO.figure & OMERO.iviewer apps (good job
> BTW, users and PIs LOVE these!).
>
>
> Thanks for the positive feedback. Always appreciated.
>
> However, neither OMERO.viewer or OMERO.figure display correct time points
> on timelapse data we generate.
> It seems like OMERO/Bio-Formats is not correctly reading (or utilizing)
> TimeIncrement & TimeIncrementUnit from Micro-Manager generated files, even
> though the deltaT value for each image is displayed correctly.
> We are running both MM 1.4.23-nightly and 2.0-beta-nightly to control our
> microscope, depending on user preference.
> I occasionally upgrade the nightlies, but the problems I describe are
> common for all versions tested.
>
> How can I get correct TimeIncrement & TimeIncrementUnit values set for my
> existing Micromanager generated data in the OMERO.server/web environment?
>
> Question 2:
> During the setup of the MM configuration files I have been very systematic
> and thorough, so that all relevant data (filter bandpass, product numbers,
> objective NAs, etc...) is captured in the Micro-Manager metadata through
> informative device and property names.
> As a facility manager it would be very helpful for me and my users to have
> easy access to this data through OMERO.
>
>
> There are different types of metadata and file representations created by
> Micro-Manager. For
> versions 1.4.x, we tried to summarise the representation of the various
> scenarios in the table
> at the bottom of this documentation page [1].
>
> As you mentioned Micro-Manager captures extensive metadata which is stored
> within a TIFF tag.
> This metadata can optionally be saved as a separate text file.
> Additionally, a subset is also
> converted into OME metadata and stored in the ImageDescription tag as per
> the OME-TIFF
> specification.
>
> To maximise the acquisition metadata, our current recommendation is to
> create companion files
> at acquisition containing the MM metadata and use this file when importing
> the data into OMERO.
> This should result in the data being available as Original Metadata in the
> right-hand panel - see
> screenshots below for an example with a sample dataset [2] created with
> the demo configuration.
>
> [1]
> http://docs.openmicroscopy.org/bio-formats/5.9.2/users/micromanager/index.html
>
> [2] *http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/
> <http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/>*
>
>
> Is there some plugin/app/script that makes the Micro-Manager metadata
> accessible on the OMERO.server, or at least viewable from the web interface?
> If not, I can try to implement one. I have spent some time reading through
> tif-tags and MM metadata for other projects, but there is no need to
> reinvent the wheel if there is something out there.
>
>
> If you cannot get the data in the representation described above or if you
> have data already
> imported, the expectationfrom the Micro-Manager file format specification
> [3] is that the
> extra metadata should be stored in the private TIFF tag 51123.
>
> [3]
> https://micro-manager.org/wiki/Micro-Manager_File_Formats#Image_File_Directories
>
> Comments:
>
> It has always annoyed me that MM ome-tiff files won't display a correct
> frame interval when drag&dropped into ImageJ/FIJI.
> On the other hand, if images are instead opened through the Bio-Formats
> Importer, then they loose their LUTs (but gain the lost frame intervals).
> The lost LUTs is also a (minor) issue when importing MM data to
> OMERO.server.
> I suspect that this is because ImageJ has its own TIF-tag that stores
> imageJ-metadata, and that there is some inconsistencies in how MM writes
> the tif tags here.
>
> Anyway, thanks for great softwares everyone, let's make them even better!
>
>
> All agreed. Micro-manager datasets are certainly a very interesting
> example of imaging data
> where the overall metadata is stored according to various models (OME,
> ImageJ, MM-JSON)
> and in different places. Preliminary discussion have been engaged on how
> to reconcile these
> various sources of metadata when reading the files [4]. In the short run,
> the suggestion described
> above is the one that hopefully should offer the maximal preservation of
> the original metadata.
>
> [4] https://github.com/openmicroscopy/bioformats/pull/2213
>
> Best,
> Sebastien
>
> Dr. Jens Eriksson, manager
> Sellin Imaging Platform
> Dep. Medical Biohecmistry and Microbiology
> Uppsala University
> Sweden
>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181119/c2bba9d8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-11-15 at 13.56.07.png
Type: image/png
Size: 220204 bytes
Desc: not available
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181119/c2bba9d8/attachment.png>