[ome-users] Support for .FLI files from Capture software (HiCAM FLUO high-speed camera)

Wed Jan 23 15:16:11 GMT 2019

Hello David,

You are welcome. I already committed some fixes because I found out that it
was not working for files > 2 GB, it should work now.

Best regards,

Johan Herz, Sales Engineer

Please visit FLIM.camera for more information about the Toggel.

Tel: +31-50-501-8461 | Skype: lambert-johan
Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
Groningen|The Netherlands
Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com

Op wo 23 jan. 2019 om 11:09 schreef David Gault (Staff) <
d.gault at dundee.ac.uk>:

> Hi Johan,
>
> Many thanks for opening the PR, external contributions to the project are
> always appreciated.
> We have included the PR for testing with our build systems and will review
> it shortly.
>
> With Thanks,
> David Gault
>
> On 22 Jan 2019, at 17:47, Johan Herz <johan at lambertinstruments.com> wrote:
>
> Hello Sebastien,
>
> Regarding UINT12, I made a pull request for updating our LiFLIMreader.java
> (https://github.com/openmicroscopy/bioformats/pull/3308)
>
> Please let me know if this is okay, many thanks in advance.
>
> Looking forward to receiving your reply.
>
> Best,
> Johan Herz
>
>
> Op ma 21 jan. 2019 09:12 schreef Sebastien Besson (Staff) <
> s.besson at dundee.ac.uk:
>
>> Good morning Johan,
>>
>> The OME handling of 12-bit data is very similar to what you do in your
>> viewer. Shortly,
>> the model allows to define the pixel type [1] but also the significant
>> number of bits [2] for
>> A dataset.
>>
>> There is no immediate plan to extend the pixel types but you can see
>> examples of using
>> These concepts to read 12-bit data for other file readers [3] [4].
>>
>> Best,
>> Sebastien
>>
>> [1]
>> https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType
>>
>> [2]
>> https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#Pixels_SignificantBits
>>
>> [3]
>> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/CanonRawReader.java#L133
>>
>> [4]
>> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/ScanrReader.java#L591
>>
>>
>> On 18 Jan 2019, at 14:16, Johan Herz <johan at lambertinstruments.com>
>> wrote:
>>
>> Hello Sebastien,
>>
>> Thank you and your welcome.
>> one of the reasons we use uint12 is that it preserves disk space. If we
>> finally present the data in our viewer we upscale it to 16bit.
>> When do you think uint12 will become available in the core metadata?
>> Would it be an option to add an 'unpacker'-function to the LiFLIMReader,
>> that unpacks and upscales the uint12 to uint16 data? But can the data still
>> be opened virtually in that case?
>>
>> Best regards,
>>
>> Johan Herz, Sales Engineer
>>
>> Please visit FLIM.camera <http://flim.camera/> for more information
>> about the Toggel.
>>
>> Tel: +31-50-501-8461 | Skype: lambert-johan
>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>> Groningen|The Netherlands
>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>
>>
>> Op di 15 jan. 2019 om 12:29 schreef Sebastien Besson (Staff) <
>> s.besson at dundee.ac.uk>:
>>
>>> Hi Johan,
>>>
>>> Thank you very much for opening the Pull Request and sending us
>>> representative
>>> samples of the issue. We will review it at earliest and hopefully have
>>> at least this fix
>>> released as part of Bio-Formats 6.
>>>
>>> One comment regarding the data format.The pixel type is part of what we
>>> call core
>>> metadata i.e. metadata mandatory for describing each image in a fileset.
>>> It can only
>>> take a limited number of values [1]. This is partly why the mapping of
>>> this particular
>>> metadata field is much stricter than other types of keys [2] and will
>>> need to be updated
>>> In order to extend support for new data types.
>>>
>>> Best,
>>> Sebastien
>>>
>>> [1] *https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType
>>> <https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType>*
>>> [2]
>>> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/LiFlimReader.java#L496
>>>
>>> On 14 Jan 2019, at 19:17, Johan Herz <johan at lambertinstruments.com>
>>> wrote:
>>>
>>> Hello Sebastien,
>>>
>>> The GB file size problem persisted, and we think the following pull
>>> request will fix it:
>>> https://github.com/openmicroscopy/bioformats/pull/3303
>>>
>>> If you have any reconmondations please let me know.
>>>
>>> Regarding implementing the unsigned 12 bit data format, please let me
>>> know if I can be of any further help.
>>>
>>> Many thabks for your support.
>>>
>>> Best,
>>> Johan
>>>
>>> Op ma 14 jan. 2019 10:57 schreef Sebastien Besson (Staff) <
>>> s.besson at dundee.ac.uk:
>>>
>>>> Hi Johan and Aaron,
>>>>
>>>> Happy New Year to you guys too. I am reincluding the Users mailing
>>>> lists in this
>>>> conversation.
>>>>
>>>> On 11 Jan 2019, at 08:11, Johan Herz <johan at lambertinstruments.com>
>>>> wrote:
>>>>
>>>> Hello Sebastien and Ponti,
>>>>
>>>> Thank you for the follow-up, all to you too a very good 2019!
>>>>
>>>> @Aaron, yes I will be available for testing, my pleasure.
>>>>
>>>> @Sebastien,  The FLI format for the Hi-speed is not yet very well
>>>> documented, it is basically based on the original FLI format with a few new
>>>> data types. We also support color cameras. But from our experience, if we
>>>> added a header entry that the bio-formats FLI reader did not recognize, it
>>>> would just ignore it. Unless it makes a collision with another key entry.
>>>>
>>>> So basically the main issues currently are the Data type and the 2gb
>>>> File size max.
>>>> It will become possible to open very big files virtually, right?
>>>> Because we can easily make files that are 50GB or bigger so it can most of
>>>> the time never fit the RAM memory.
>>>>
>>>>
>>>>
>>>> About 2GB, I assume you are referring to the issue described in this
>>>> section of the
>>>> technical documentation [1]? Briefly, we are routinely dealing with
>>>> datasets in the
>>>> GB-TB magnitude so file size per se should not be a problem.
>>>>
>>>> Am I correct in assuming you are loading data using the ImageJ/Fiji
>>>> Bio-Formats
>>>> Plugin? In that case opening as a virtual stack is definitely the first
>>>> thing to do. For
>>>> images where Individual planes exceed 2GB limit (due to dimensions or
>>>> pixel type),
>>>> you might need to use the Crop option to load images in tiles rather
>>>> than whole planes.
>>>>
>>>> [1]
>>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/about/bug-reporting.html
>>>>
>>>>
>>>> Regarding compiling Bio-formats, I probably should have given it more
>>>> time, which I did not had, unfortunately, to have given the manual a better
>>>> read. What IDE are you using or are you compiling command line?
>>>>
>>>>
>>>> Generally this documentation page is the landing page for everything
>>>> related to building
>>>> Bio-Formats from source [2]. In general, people will use either the
>>>> command-line or an IDE,
>>>> Eclipse being probably the most popular one. We should be able to offer
>>>> guidance for both
>>>> paths.
>>>>
>>>> Best,
>>>> Sebastien
>>>>
>>>> [2]
>>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/developers/building-bioformats.html
>>>>
>>>>
>>>> I am looking forward to receiving the fix
>>>>
>>>> Best regards,
>>>>
>>>> Johan Herz, Sales Engineer
>>>>
>>>> Please visit FLIM.camera <http://flim.camera/> for more information
>>>> about the Toggel.
>>>>
>>>> Tel: +31-50-501-8461 | Skype: lambert-johan
>>>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>>>> Groningen|The Netherlands
>>>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>>>
>>>>
>>>> Op di 8 jan. 2019 om 18:04 schreef Sebastien Besson (Staff) <
>>>> s.besson at dundee.ac.uk>:
>>>>
>>>>> Hi Johan and Aaron,
>>>>>
>>>>> Thanks for making example images available, we will copy them
>>>>> internally for testing purposes.
>>>>> Is an amended version of the technical documentation described the
>>>>> format changes also available?
>>>>>
>>>>> So far we have captured this new variant of the .fli format in the
>>>>> general Bio-Formats backlog [1].
>>>>> In terms of priority, the academically-funded team will primarily
>>>>> focus on next-generation open formats
>>>>> In 2019, starting with the release of Bio-Formats 6 [2]. Although we
>>>>> will not be able to prioritise such work,
>>>>> we will happily review any direct community contribution to update the
>>>>> reader. An alternate possibility solution
>>>>> would be to get in touch with Glencoe Software and commission the
>>>>> reader update.
>>>>>
>>>>> Trying to work in incremental steps and solve the immediate blockers,
>>>>> Johan, could you give us more details
>>>>> into the type of issues you ran into while compiling Bio-Formats?
>>>>>
>>>>> Best,
>>>>> Sebastien
>>>>>
>>>>>
>>>>> [1] https://trello.com/c/OGGEzj7X/305-li-flim-12-bit-format
>>>>> [2]
>>>>> https://blog.openmicroscopy.org/file-formats/community/2018/11/29/ometiffpyramid/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 7 Jan 2019, at 09:28, Ponti Aaron <aaron.ponti at bsse.ethz.ch> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Happy new year, everyone! Sorry for the long silence about the FMI
>>>>> issue. I don’t think I will soon find the time to implement support for
>>>>> 12-but FLI files in bio-formats myself. I see that Johan replied to the
>>>>> thread and provided example images. Would you guys work together to sort
>>>>> out the issue? That’d be great!
>>>>>
>>>>> Thanks a lot!
>>>>> a2
>>>>>
>>>>> ----
>>>>> Dr. Aaron Ponti
>>>>> Software and Data Management Engineer
>>>>> Image Analysis Specialist
>>>>> Single Cell Facility
>>>>> Department of Biosystems Science and Engineering (D-BSSE)
>>>>> ETH Zürich
>>>>> Office 2.30
>>>>> Mattenstrasse 26
>>>>> 4058 Basel, Switzerland
>>>>> aaron.ponti at bsse.ethz.ch
>>>>> T: +41 61 387 33 74
>>>>> F: +41 61 387 39 93
>>>>>
>>>>> *From:* Johan Herz <johan at lambertinstruments.com>
>>>>> *Sent:* Mittwoch, 19. Dezember 2018 17:19
>>>>> *To:* OME User Support List <ome-users at lists.openmicroscopy.org.uk>
>>>>> *Cc:* Lummen Tom <tom.lummen at bsse.ethz.ch>; Ponti Aaron <
>>>>> aaron.ponti at bsse.ethz.ch>
>>>>> *Subject:* Re: [ome-users] Support for .FLI files from Capture
>>>>> software (HiCAM FLUO high-speed camera)
>>>>>
>>>>> Hello Sebastien,
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> I have 2 files, 6 GB each.
>>>>> They are made with 2 different versions of Capture our Hi-speed
>>>>> recording software.
>>>>> We make use of the .fli format and we tried to be as compatible with
>>>>> the Bio-formats as possible.
>>>>> Unfortunately, we could not work around the 12 bit, hence the
>>>>> unsupported data format.
>>>>>
>>>>> Personally, I have tried to get bio-formats to compile on my one
>>>>> machine, I made a checkout, but I ran in too many problems. It seems that
>>>>> you are planning to have a new release.
>>>>> Because I wanted to fix the problem of not being able to open files
>>>>> having a filesize bigger than 2 GB.
>>>>>
>>>>> So could you take this in to account as well? To open our files
>>>>> virtual, FiJi supports this, right?
>>>>>
>>>>> Is it okay for you if I share the files through Google Drive, as they
>>>>> are too big to upload to your server?
>>>>>
>>>>>
>>>>> * Recording_Capture_1.1.fli
>>>>> <https://drive.google.com/a/lambertinstruments.com/file/d/1xw43lsucOssXb97ia0JJCCZ9467hV3iO/view?usp=drive_web>*
>>>>>
>>>>> * Recording_Capture_1.0.4.fli
>>>>> <https://drive.google.com/a/lambertinstruments.com/file/d/19NwEUveiphMxUf9QSjtSHSMBthlaPxqW/view?usp=drive_web>*
>>>>>
>>>>> Also have attached a Recording with our New FLIM system, because
>>>>> originally the .fli was for our FLIM system.
>>>>> Those files are still compatible. So for testing purposes, I included
>>>>> that such that you can make sure it still works.
>>>>> See attachment.
>>>>>
>>>>> Please let me know if you have any further questions.
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Johan Herz, Sales Engineer
>>>>> Please visit FLIM.camera <http://flim.camera/> for more information
>>>>> about the Toggel.
>>>>>
>>>>> Tel: +31-50-501-8461 | Skype: lambert-johan
>>>>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>>>>> Groningen|The Netherlands
>>>>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>>>>
>>>>>
>>>>> Op ma 17 dec. 2018 om 15:41 schreef Sebastien Besson (Staff) <
>>>>> s.besson at dundee.ac.uk>:
>>>>>
>>>>> Hi Aaron,
>>>>>
>>>>>
>>>>> On 14 Dec 2018, at 10:00, Ponti Aaron <aaron.ponti at bsse.ethz.ch>
>>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> we have .FLI files from Lambert Instruments' Capture software (version
>>>>> 1.0.4.0) for the high-speed camera HiCAM FLUO. Images are streamed at high
>>>>> speed and stored as UINT12 (2 pixels in 3 bytes) or optionally as UINT8.
>>>>> The FLIM reader in bio-formats (
>>>>> https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/in/LiFlimReader.java)
>>>>> does not seem to support 12-bit intensities.
>>>>>
>>>>>
>>>>> From the internal specification of the LI-FLIM file format which we
>>>>> used for the reader [1]:
>>>>>
>>>>> "The datatype key can have one of the following 8 values: UINT8, INT8,
>>>>> UINT16, INT16,
>>>>> UINT32, INT32, REAL32 or REAL64.”
>>>>>
>>>>> A file that would use UINT12 would probably be a new variant of the
>>>>> file format for which we
>>>>> have currently neither a format specification nor representative
>>>>> samples
>>>>>
>>>>>
>>>>> Also, when asking bio-formats to return the metadata, the whole file
>>>>> is returned (which suggests that the metadata of standard FLIM .FLI files
>>>>> is organized -- or terminated -- differently).
>>>>>
>>>>>
>>>>> Understood. This would be consistent with the pixel type issue
>>>>> reported above i.e. that could
>>>>> indeed reflect an internal layout different from the one Bio-Formats
>>>>> is expecting..
>>>>>
>>>>>
>>>>>  The file is binary with the first bytes containing metadata
>>>>> information in readable text format. One field is 'hasDarkImage'. If
>>>>> hasDarkImage == 1, the very last frame is a dark image that is subtracted
>>>>> on-the-fly in the software when displaying the frames (but the images are
>>>>> stored raw in the file). The 'exposureTime' is in micro seconds. Finally,
>>>>> the software seems to support 'compression', although there is no setting
>>>>> for turning it on in the software. The software supports binning.
>>>>>
>>>>>
>>>>>
>>>>>  Would you be interested in adding support for these files to
>>>>> bio-formats? We have example files that we could upload.
>>>>>
>>>>>
>>>>> Over the past years, we have been working on ways to have more
>>>>> sustainable support for new file
>>>>> formats including any variants of existing file formats. We published
>>>>> a blog post summarizing our
>>>>> experience a few years ago [2] and since then more manufacturers have
>>>>> been engaging in this
>>>>> process.
>>>>>
>>>>> Have you been in touch with Lambert Instruments to report this issue?
>>>>> Are they aware that the latest files
>>>>> produced by their software are no longer readable by Bio-Formats and
>>>>> any software based upon it?
>>>>>
>>>>> Representative samples allowing to reproduce the issue are always
>>>>> welcome. Ideally public samples licensed
>>>>> under CC-BY license [3] are the most effective way to help spontaneous
>>>>> or sponsored contributions to the
>>>>> project and facilitate both the immediate software work as well as the
>>>>> long-term maintenance.
>>>>> You can upload datasets under 2GB to our QA system [4] and we can
>>>>> provide FTP credentials for large samples.
>>>>>
>>>>> Best,
>>>>> Sebastien
>>>>>
>>>>>
>>>>> [1]
>>>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/formats/lambert-instruments-flim.html
>>>>> [2]
>>>>> https://blog.openmicroscopy.org/file-formats/community/2016/08/31/bf-partnerships/
>>>>> [3] https://downloads.openmicroscopy.org/images/
>>>>> [4] http://qa.openmicroscopy.org.uk/qa/upload/
>>>>>
>>>>>
>>>>>
>>>>>  Thanks a lot,
>>>>> a2
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----
>>>>> Dr. Aaron Ponti
>>>>> Software and Data Management Engineer
>>>>> Image Analysis Specialist
>>>>> Single Cell Facility
>>>>> Department of Biosystems Science and Engineering (D-BSSE)
>>>>> ETH Zürich
>>>>> Office 2.30
>>>>> Mattenstrasse 26
>>>>> 4058 Basel, Switzerland
>>>>> aaron.ponti at bsse.ethz.ch
>>>>> T: +41 61 387 33 74
>>>>> F: +41 61 387 39 93
>>>>>
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>>
>>>>>
>>>>>
>>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>>
>>>>> _______________________________________________
>>>>> ome-users mailing list
>>>>> ome-users at lists.openmicroscopy.org.uk
>>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>>
>>>>>
>>>>>
>>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>>
>>>>
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20190123/43c85e54/attachment.html>