[ome-users] Micromanager metadata and time increments

[Jens Eriksson]

Sure thing, i have uploaded a dataset from Mircro-Manager 1-4-23-20180923
(ID:27101)
I also submitted the _metadata.txt import errorlog (ID: 27100)

Keep up the good work!

Best,

Jens

On Tue, Nov 20, 2018 at 11:35 AM Sebastien Besson (Staff) <
s.besson at dundee.ac.uk> wrote:

> Dear Jens,
>
> thanks for reporting the error. The stack trace looks very similar to what
> has been
> reported on the image.sc forum [1] as well as the Micro-Manager mailing
> list [2].
>
> For Micro-Manager 2.0.0-beta3 datasets, currently unsupported by
> Bio-Formats, we
> are aware that the metadata format has been through several breaking
> changes,
> including the IJType field which is the cause of the Bio-Formats parsing
> failure below.
>
> For Micro-Manager 1.4.23 datasets, it might be this constitutes a variant
> of the
> Micro-Manager 1.4 metadata format [3]. But we have no representative
> datasets
> showing the issue so far. Would it be possible for you to upload sampled
> data at
> http://qa.openmicroscopy.org.uk/qa/upload/?
>
> Best,
> Sebastien
>
>
> [1]
> https://forum.image.sc/t/exception-thrown-when-importing-micro-manager-dataset-with-bioformats/11068
> [2]
> http://micro-manager.3463995.n2.nabble.com/Exception-thrown-when-importing-Micro-Manager-2-0-dataset-with-Bioformats-td7589436.html
> [3] https://micro-manager.org/wiki/Files_and_Metadata
>
>
> On 19 Nov 2018, at 15:13, Jens Eriksson <jenseri at gmail.com> wrote:
>
> I have been saving the metadata.txt with the ome-tifs, but now I realize
> there has been an unnoticed error at the import step. We normally import
> the folders and not the files directly, so there has not been any
> indications of this error to the user. If I import ome-tif + metadata.txt
> files directly, I discovered that the metadata.txt files are not processed
> correctly by BioFormats. The following error is logged in the
> OMERO-insight64 importer:
>
> java.lang.NumberFormatException: For input string: ""
> at java.lang.NumberFormatException.forInputString(Unknown Source)
> at java.lang.Integer.parseInt(Unknown Source)
> at java.lang.Integer.parseInt(Unknown Source)
> at
> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
> at
> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
> at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
> at loci.formats.FormatReader.setId(FormatReader.java:1397)
> at loci.formats.ImageReader.setId(ImageReader.java:842)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.Memoizer.setId(Memoizer.java:662)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at
> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
> at
> ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
> at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
> at
> org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
> at
> org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
> at
> org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
> at
> org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
> at
> org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
> at
> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
> at
> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
> at
> org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
> at java.lang.Thread.run(Unknown Source)
>
> at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
> at
> ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
> at
> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
> at
> ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
> at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
> at
> org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
> at
> org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
> at
> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
> at
> org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
> at
> org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
> at
> org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
> at
> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
> at
> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
> at
> org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.NumberFormatException: For input string: ""
> at java.lang.NumberFormatException.forInputString(Unknown Source)
> at java.lang.Integer.parseInt(Unknown Source)
> at java.lang.Integer.parseInt(Unknown Source)
> at
> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
> at
> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
> at loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
> at loci.formats.FormatReader.setId(FormatReader.java:1397)
> at loci.formats.ImageReader.setId(ImageReader.java:842)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at loci.formats.Memoizer.setId(Memoizer.java:662)
> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
> at
> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
> ... 16 more
>
> This error is given by both MM 1.4.23-nightly & MM 2.0-beta3-nightly
> metadata.txt files. ome-tifs from these import without errors.
>
>
> Best,
>
> Jens
>
> On Thu, Nov 15, 2018 at 3:13 PM Sebastien Besson (Staff) <
> s.besson at dundee.ac.uk> wrote:
>
>> Hi Jens,
>>
>>
>> On 13 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com> wrote:
>>  <snip>
>>
>>
>> I recently finished implementing an OMERO.server with a OMERO.web
>> interface, together with the OMERO.figure & OMERO.iviewer apps (good job
>> BTW, users and PIs LOVE these!).
>>
>>
>> Thanks for the positive feedback. Always appreciated.
>>
>> However, neither OMERO.viewer or OMERO.figure display correct time points
>> on timelapse data we generate.
>> It seems like OMERO/Bio-Formats is not correctly reading (or utilizing)
>> TimeIncrement & TimeIncrementUnit from Micro-Manager generated files, even
>> though the deltaT value for each image is displayed correctly.
>> We are running both MM 1.4.23-nightly and 2.0-beta-nightly to control our
>> microscope, depending on user preference.
>> I occasionally upgrade the nightlies, but the problems I describe are
>> common for all versions tested.
>>
>> How can I get correct TimeIncrement & TimeIncrementUnit values set for my
>> existing Micromanager generated data in the OMERO.server/web environment?
>>
>> Question 2:
>> During the setup of the MM configuration files I have been very
>> systematic and thorough, so that all relevant data (filter bandpass,
>> product numbers, objective NAs, etc...) is captured in the Micro-Manager
>> metadata through informative device and property names.
>> As a facility manager it would be very helpful for me and my users to
>> have easy access to this data through OMERO.
>>
>>
>> There are different types of metadata and file representations created by
>> Micro-Manager. For
>> versions 1.4.x, we tried to summarise the representation of the various
>> scenarios in the table
>> at the bottom of this documentation page [1].
>>
>> As you mentioned Micro-Manager captures extensive metadata which is
>> stored within a TIFF tag.
>> This metadata can optionally be saved as a separate text file.
>> Additionally, a subset is also
>> converted into OME metadata and stored in the ImageDescription tag as per
>> the OME-TIFF
>> specification.
>>
>> To maximise the acquisition metadata, our current recommendation is to
>> create companion files
>> at acquisition containing the MM metadata and use this file when
>> importing the data into OMERO.
>> This should result in the data being available as Original Metadata in
>> the right-hand panel - see
>> screenshots below for an example with a sample dataset [2] created with
>> the demo configuration.
>>
>> [1]
>> http://docs.openmicroscopy.org/bio-formats/5.9.2/users/micromanager/index.html
>>
>> [2] *http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/
>> <http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/>*
>>
>>
>> Is there some plugin/app/script that makes the Micro-Manager metadata
>> accessible on the OMERO.server, or at least viewable from the web interface?
>> If not, I can try to implement one. I have spent some time reading
>> through tif-tags and MM metadata for other projects, but there is no need
>> to reinvent the wheel if there is something out there.
>>
>>
>> If you cannot get the data in the representation described above or if
>> you have data already
>> imported, the expectationfrom the Micro-Manager file format specification
>> [3] is that the
>> extra metadata should be stored in the private TIFF tag 51123.
>>
>> [3]
>> https://micro-manager.org/wiki/Micro-Manager_File_Formats#Image_File_Directories
>>
>> Comments:
>>
>> It has always annoyed me that MM ome-tiff files won't display a correct
>> frame interval when drag&dropped into ImageJ/FIJI.
>> On the other hand, if images are instead opened through the Bio-Formats
>> Importer, then they loose their LUTs (but gain the lost frame intervals).
>> The lost LUTs is also a (minor) issue when importing MM data to
>> OMERO.server.
>> I suspect that this is because ImageJ has its own TIF-tag that stores
>> imageJ-metadata, and that there is some inconsistencies in how MM writes
>> the tif tags here.
>>
>> Anyway, thanks for great softwares everyone, let's make them even better!
>>
>>
>> All agreed. Micro-manager datasets are certainly a very interesting
>> example of imaging data
>> where the overall metadata is stored according to various models (OME,
>> ImageJ, MM-JSON)
>> and in different places. Preliminary discussion have been engaged on how
>> to reconcile these
>> various sources of metadata when reading the files [4]. In the short run,
>> the suggestion described
>> above is the one that hopefully should offer the maximal preservation of
>> the original metadata.
>>
>> [4] https://github.com/openmicroscopy/bioformats/pull/2213
>>
>> Best,
>> Sebastien
>>
>> Dr. Jens Eriksson, manager
>> Sellin Imaging Platform
>> Dep. Medical Biohecmistry and Microbiology
>> Uppsala University
>> Sweden
>>
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
> <Screen Shot 2018-11-15 at 13.56.07.png>
> _______________________________________________
>
>
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181120/63eff84e/attachment.html>