[ome-users] Micromanager metadata and time increments

Thu Nov 22 14:01:27 GMT 2018

Great, good job!

 It's fine to make the data publicly available under CC-BY.

Thanks a million!

/J

On Wed, Nov 21, 2018 at 10:14 PM Sebastien Besson (Staff) <
s.besson at dundee.ac.uk> wrote:

> Dear Jens,
>
> thanks for uploading a representative dataset. The issue was associated
> with
> the parsing of invalid double values. It turns out this bug has been
> identified and
> addressed in the development branch of Bio-Format 6 [1].
>
> We will add your sample dataset to your QA repository to ensure we
> encounter
> no future regression. Please let us know if it would be possible to make
> this data
> publicly available under a CC-BY license [2]. Otherwise we will keep it
> private.
>
> Best regards,
> Sebastien
>
> [1] https://github.com/openmicroscopy/bioformats/pull/3253
> [2] http://downloads.openmicroscopy.org/images/Micro-Manager/
>
> On 20 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com> wrote:
>
> Sure thing, i have uploaded a dataset from Mircro-Manager 1-4-23-20180923
> (ID:27101)
> I also submitted the _metadata.txt import errorlog (ID: 27100)
>
> Keep up the good work!
>
> Best,
>
> Jens
>
> On Tue, Nov 20, 2018 at 11:35 AM Sebastien Besson (Staff) <
> s.besson at dundee.ac.uk> wrote:
>
>> Dear Jens,
>>
>> thanks for reporting the error. The stack trace looks very similar to
>> what has been
>> reported on the image.sc forum [1] as well as the Micro-Manager mailing
>> list [2].
>>
>> For Micro-Manager 2.0.0-beta3 datasets, currently unsupported by
>> Bio-Formats, we
>> are aware that the metadata format has been through several breaking
>> changes,
>> including the IJType field which is the cause of the Bio-Formats parsing
>> failure below.
>>
>> For Micro-Manager 1.4.23 datasets, it might be this constitutes a variant
>> of the
>> Micro-Manager 1.4 metadata format [3]. But we have no representative
>> datasets
>> showing the issue so far. Would it be possible for you to upload sampled
>> data at
>> http://qa.openmicroscopy.org.uk/qa/upload/?
>>
>> Best,
>> Sebastien
>>
>>
>> [1]
>> https://forum.image.sc/t/exception-thrown-when-importing-micro-manager-dataset-with-bioformats/11068
>> [2]
>> http://micro-manager.3463995.n2.nabble.com/Exception-thrown-when-importing-Micro-Manager-2-0-dataset-with-Bioformats-td7589436.html
>> [3] https://micro-manager.org/wiki/Files_and_Metadata
>>
>>
>> On 19 Nov 2018, at 15:13, Jens Eriksson <jenseri at gmail.com> wrote:
>>
>> I have been saving the metadata.txt with the ome-tifs, but now I realize
>> there has been an unnoticed error at the import step. We normally import
>> the folders and not the files directly, so there has not been any
>> indications of this error to the user. If I import ome-tif + metadata.txt
>> files directly, I discovered that the metadata.txt files are not processed
>> correctly by BioFormats. The following error is logged in the
>> OMERO-insight64 importer:
>>
>> java.lang.NumberFormatException: For input string: ""
>> at java.lang.NumberFormatException.forInputString(Unknown Source)
>> at java.lang.Integer.parseInt(Unknown Source)
>> at java.lang.Integer.parseInt(Unknown Source)
>> at
>> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
>> at
>> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
>> at
>> loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
>> at loci.formats.FormatReader.setId(FormatReader.java:1397)
>> at loci.formats.ImageReader.setId(ImageReader.java:842)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.Memoizer.setId(Memoizer.java:662)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at
>> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
>> at
>> ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
>> at
>> ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
>> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
>> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
>> at
>> org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
>> at
>> org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
>> at
>> org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
>> at
>> org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
>> at java.lang.Thread.run(Unknown Source)
>>
>> at org.openmicroscopy.shoola.env.data.util.Status.update(Status.java:594)
>> at
>> ome.formats.importer.ImportCandidates.safeUpdate(ImportCandidates.java:536)
>> at
>> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:471)
>> at
>> ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
>> at
>> ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:384)
>> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
>> at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
>> at
>> org.openmicroscopy.shoola.env.data.OMEROGateway.getImportCandidates(OMEROGateway.java:5989)
>> at
>> org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1070)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:73)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:48)
>> at
>> org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:97)
>> at
>> org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
>> at
>> org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
>> at
>> org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
>> at java.lang.Thread.run(Unknown Source)
>> Caused by: java.lang.NumberFormatException: For input string: ""
>> at java.lang.NumberFormatException.forInputString(Unknown Source)
>> at java.lang.Integer.parseInt(Unknown Source)
>> at java.lang.Integer.parseInt(Unknown Source)
>> at
>> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:765)
>> at
>> loci.formats.in.MicromanagerReader.parsePosition(MicromanagerReader.java:441)
>> at
>> loci.formats.in.MicromanagerReader.initFile(MicromanagerReader.java:305)
>> at loci.formats.FormatReader.setId(FormatReader.java:1397)
>> at loci.formats.ImageReader.setId(ImageReader.java:842)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at loci.formats.Memoizer.setId(Memoizer.java:662)
>> at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
>> at
>> ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
>> ... 16 more
>>
>> This error is given by both MM 1.4.23-nightly & MM 2.0-beta3-nightly
>> metadata.txt files. ome-tifs from these import without errors.
>>
>>
>> Best,
>>
>> Jens
>>
>> On Thu, Nov 15, 2018 at 3:13 PM Sebastien Besson (Staff) <
>> s.besson at dundee.ac.uk> wrote:
>>
>>> Hi Jens,
>>>
>>>
>>> On 13 Nov 2018, at 14:42, Jens Eriksson <jenseri at gmail.com> wrote:
>>>  <snip>
>>>
>>>
>>> I recently finished implementing an OMERO.server with a OMERO.web
>>> interface, together with the OMERO.figure & OMERO.iviewer apps (good job
>>> BTW, users and PIs LOVE these!).
>>>
>>>
>>> Thanks for the positive feedback. Always appreciated.
>>>
>>> However, neither OMERO.viewer or OMERO.figure display correct time
>>> points on timelapse data we generate.
>>> It seems like OMERO/Bio-Formats is not correctly reading (or utilizing)
>>> TimeIncrement & TimeIncrementUnit from Micro-Manager generated files, even
>>> though the deltaT value for each image is displayed correctly.
>>> We are running both MM 1.4.23-nightly and 2.0-beta-nightly to control
>>> our microscope, depending on user preference.
>>> I occasionally upgrade the nightlies, but the problems I describe are
>>> common for all versions tested.
>>>
>>> How can I get correct TimeIncrement & TimeIncrementUnit values set for
>>> my existing Micromanager generated data in the OMERO.server/web environment?
>>>
>>> Question 2:
>>> During the setup of the MM configuration files I have been very
>>> systematic and thorough, so that all relevant data (filter bandpass,
>>> product numbers, objective NAs, etc...) is captured in the Micro-Manager
>>> metadata through informative device and property names.
>>> As a facility manager it would be very helpful for me and my users to
>>> have easy access to this data through OMERO.
>>>
>>>
>>> There are different types of metadata and file representations created
>>> by Micro-Manager. For
>>> versions 1.4.x, we tried to summarise the representation of the various
>>> scenarios in the table
>>> at the bottom of this documentation page [1].
>>>
>>> As you mentioned Micro-Manager captures extensive metadata which is
>>> stored within a TIFF tag.
>>> This metadata can optionally be saved as a separate text file.
>>> Additionally, a subset is also
>>> converted into OME metadata and stored in the ImageDescription tag as
>>> per the OME-TIFF
>>> specification.
>>>
>>> To maximise the acquisition metadata, our current recommendation is to
>>> create companion files
>>> at acquisition containing the MM metadata and use this file when
>>> importing the data into OMERO.
>>> This should result in the data being available as Original Metadata in
>>> the right-hand panel - see
>>> screenshots below for an example with a sample dataset [2] created with
>>> the demo configuration.
>>>
>>> [1]
>>> http://docs.openmicroscopy.org/bio-formats/5.9.2/users/micromanager/index.html
>>>
>>> [2] *http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/
>>> <http://downloads.openmicroscopy.org/images/Micro-Manager/1.4.22/thomas/test/test2_stack/>*
>>>
>>>
>>> Is there some plugin/app/script that makes the Micro-Manager metadata
>>> accessible on the OMERO.server, or at least viewable from the web interface?
>>> If not, I can try to implement one. I have spent some time reading
>>> through tif-tags and MM metadata for other projects, but there is no need
>>> to reinvent the wheel if there is something out there.
>>>
>>>
>>> If you cannot get the data in the representation described above or if
>>> you have data already
>>> imported, the expectationfrom the Micro-Manager file format
>>> specification [3] is that the
>>> extra metadata should be stored in the private TIFF tag 51123.
>>>
>>> [3]
>>> https://micro-manager.org/wiki/Micro-Manager_File_Formats#Image_File_Directories
>>>
>>> Comments:
>>>
>>> It has always annoyed me that MM ome-tiff files won't display a correct
>>> frame interval when drag&dropped into ImageJ/FIJI.
>>> On the other hand, if images are instead opened through the Bio-Formats
>>> Importer, then they loose their LUTs (but gain the lost frame intervals).
>>> The lost LUTs is also a (minor) issue when importing MM data to
>>> OMERO.server.
>>> I suspect that this is because ImageJ has its own TIF-tag that stores
>>> imageJ-metadata, and that there is some inconsistencies in how MM writes
>>> the tif tags here.
>>>
>>> Anyway, thanks for great softwares everyone, let's make them even better!
>>>
>>>
>>> All agreed. Micro-manager datasets are certainly a very interesting
>>> example of imaging data
>>> where the overall metadata is stored according to various models (OME,
>>> ImageJ, MM-JSON)
>>> and in different places. Preliminary discussion have been engaged on how
>>> to reconcile these
>>> various sources of metadata when reading the files [4]. In the short
>>> run, the suggestion described
>>> above is the one that hopefully should offer the maximal preservation of
>>> the original metadata.
>>>
>>> [4] https://github.com/openmicroscopy/bioformats/pull/2213
>>>
>>> Best,
>>> Sebastien
>>>
>>> Dr. Jens Eriksson, manager
>>> Sellin Imaging Platform
>>> Dep. Medical Biohecmistry and Microbiology
>>> Uppsala University
>>> Sweden
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>> <Screen Shot 2018-11-15 at 13.56.07.png>
>> _______________________________________________
>>
>>
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20181122/0795ed97/attachment.html>