[ome-users] Support for .FLI files from Capture software (HiCAM FLUO high-speed camera)

Tue Jan 22 17:47:19 GMT 2019

Hello Sebastien,

Regarding UINT12, I made a pull request for updating our LiFLIMreader.java (
https://github.com/openmicroscopy/bioformats/pull/3308)

Please let me know if this is okay, many thanks in advance.

Looking forward to receiving your reply.

Best,
Johan Herz

Op ma 21 jan. 2019 09:12 schreef Sebastien Besson (Staff) <
s.besson at dundee.ac.uk:

> Good morning Johan,
>
> The OME handling of 12-bit data is very similar to what you do in your
> viewer. Shortly,
> the model allows to define the pixel type [1] but also the significant
> number of bits [2] for
> A dataset.
>
> There is no immediate plan to extend the pixel types but you can see
> examples of using
> These concepts to read 12-bit data for other file readers [3] [4].
>
> Best,
> Sebastien
>
> [1]
> https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType
>
> [2]
> https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#Pixels_SignificantBits
>
> [3]
> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/CanonRawReader.java#L133
>
> [4]
> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/ScanrReader.java#L591
>
>
> On 18 Jan 2019, at 14:16, Johan Herz <johan at lambertinstruments.com> wrote:
>
> Hello Sebastien,
>
> Thank you and your welcome.
> one of the reasons we use uint12 is that it preserves disk space. If we
> finally present the data in our viewer we upscale it to 16bit.
> When do you think uint12 will become available in the core metadata? Would
> it be an option to add an 'unpacker'-function to the LiFLIMReader, that
> unpacks and upscales the uint12 to uint16 data? But can the data still be
> opened virtually in that case?
>
> Best regards,
>
> Johan Herz, Sales Engineer
>
> Please visit FLIM.camera <http://flim.camera/> for more information about
> the Toggel.
>
> Tel: +31-50-501-8461 | Skype: lambert-johan
> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
> Groningen|The Netherlands
> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>
>
> Op di 15 jan. 2019 om 12:29 schreef Sebastien Besson (Staff) <
> s.besson at dundee.ac.uk>:
>
>> Hi Johan,
>>
>> Thank you very much for opening the Pull Request and sending us
>> representative
>> samples of the issue. We will review it at earliest and hopefully have at
>> least this fix
>> released as part of Bio-Formats 6.
>>
>> One comment regarding the data format.The pixel type is part of what we
>> call core
>> metadata i.e. metadata mandatory for describing each image in a fileset.
>> It can only
>> take a limited number of values [1]. This is partly why the mapping of
>> this particular
>> metadata field is much stricter than other types of keys [2] and will
>> need to be updated
>> In order to extend support for new data types.
>>
>> Best,
>> Sebastien
>>
>> [1] *https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType
>> <https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType>*
>> [2]
>> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/LiFlimReader.java#L496
>>
>> On 14 Jan 2019, at 19:17, Johan Herz <johan at lambertinstruments.com>
>> wrote:
>>
>> Hello Sebastien,
>>
>> The GB file size problem persisted, and we think the following pull
>> request will fix it:
>> https://github.com/openmicroscopy/bioformats/pull/3303
>>
>> If you have any reconmondations please let me know.
>>
>> Regarding implementing the unsigned 12 bit data format, please let me
>> know if I can be of any further help.
>>
>> Many thabks for your support.
>>
>> Best,
>> Johan
>>
>> Op ma 14 jan. 2019 10:57 schreef Sebastien Besson (Staff) <
>> s.besson at dundee.ac.uk:
>>
>>> Hi Johan and Aaron,
>>>
>>> Happy New Year to you guys too. I am reincluding the Users mailing lists
>>> in this
>>> conversation.
>>>
>>> On 11 Jan 2019, at 08:11, Johan Herz <johan at lambertinstruments.com>
>>> wrote:
>>>
>>> Hello Sebastien and Ponti,
>>>
>>> Thank you for the follow-up, all to you too a very good 2019!
>>>
>>> @Aaron, yes I will be available for testing, my pleasure.
>>>
>>> @Sebastien,  The FLI format for the Hi-speed is not yet very well
>>> documented, it is basically based on the original FLI format with a few new
>>> data types. We also support color cameras. But from our experience, if we
>>> added a header entry that the bio-formats FLI reader did not recognize, it
>>> would just ignore it. Unless it makes a collision with another key entry.
>>>
>>> So basically the main issues currently are the Data type and the 2gb
>>> File size max.
>>> It will become possible to open very big files virtually, right? Because
>>> we can easily make files that are 50GB or bigger so it can most of the time
>>> never fit the RAM memory.
>>>
>>>
>>>
>>> About 2GB, I assume you are referring to the issue described in this
>>> section of the
>>> technical documentation [1]? Briefly, we are routinely dealing with
>>> datasets in the
>>> GB-TB magnitude so file size per se should not be a problem.
>>>
>>> Am I correct in assuming you are loading data using the ImageJ/Fiji
>>> Bio-Formats
>>> Plugin? In that case opening as a virtual stack is definitely the first
>>> thing to do. For
>>> images where Individual planes exceed 2GB limit (due to dimensions or
>>> pixel type),
>>> you might need to use the Crop option to load images in tiles rather
>>> than whole planes.
>>>
>>> [1]
>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/about/bug-reporting.html
>>>
>>>
>>> Regarding compiling Bio-formats, I probably should have given it more
>>> time, which I did not had, unfortunately, to have given the manual a better
>>> read. What IDE are you using or are you compiling command line?
>>>
>>>
>>> Generally this documentation page is the landing page for everything
>>> related to building
>>> Bio-Formats from source [2]. In general, people will use either the
>>> command-line or an IDE,
>>> Eclipse being probably the most popular one. We should be able to offer
>>> guidance for both
>>> paths.
>>>
>>> Best,
>>> Sebastien
>>>
>>> [2]
>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/developers/building-bioformats.html
>>>
>>>
>>> I am looking forward to receiving the fix
>>>
>>> Best regards,
>>>
>>> Johan Herz, Sales Engineer
>>>
>>> Please visit FLIM.camera <http://flim.camera/> for more information
>>> about the Toggel.
>>>
>>> Tel: +31-50-501-8461 | Skype: lambert-johan
>>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>>> Groningen|The Netherlands
>>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>>
>>>
>>> Op di 8 jan. 2019 om 18:04 schreef Sebastien Besson (Staff) <
>>> s.besson at dundee.ac.uk>:
>>>
>>>> Hi Johan and Aaron,
>>>>
>>>> Thanks for making example images available, we will copy them
>>>> internally for testing purposes.
>>>> Is an amended version of the technical documentation described the
>>>> format changes also available?
>>>>
>>>> So far we have captured this new variant of the .fli format in the
>>>> general Bio-Formats backlog [1].
>>>> In terms of priority, the academically-funded team will primarily focus
>>>> on next-generation open formats
>>>> In 2019, starting with the release of Bio-Formats 6 [2]. Although we
>>>> will not be able to prioritise such work,
>>>> we will happily review any direct community contribution to update the
>>>> reader. An alternate possibility solution
>>>> would be to get in touch with Glencoe Software and commission the
>>>> reader update.
>>>>
>>>> Trying to work in incremental steps and solve the immediate blockers,
>>>> Johan, could you give us more details
>>>> into the type of issues you ran into while compiling Bio-Formats?
>>>>
>>>> Best,
>>>> Sebastien
>>>>
>>>>
>>>> [1] https://trello.com/c/OGGEzj7X/305-li-flim-12-bit-format
>>>> [2]
>>>> https://blog.openmicroscopy.org/file-formats/community/2018/11/29/ometiffpyramid/
>>>>
>>>>
>>>>
>>>>
>>>> On 7 Jan 2019, at 09:28, Ponti Aaron <aaron.ponti at bsse.ethz.ch> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Happy new year, everyone! Sorry for the long silence about the FMI
>>>> issue. I don’t think I will soon find the time to implement support for
>>>> 12-but FLI files in bio-formats myself. I see that Johan replied to the
>>>> thread and provided example images. Would you guys work together to sort
>>>> out the issue? That’d be great!
>>>>
>>>> Thanks a lot!
>>>> a2
>>>>
>>>> ----
>>>> Dr. Aaron Ponti
>>>> Software and Data Management Engineer
>>>> Image Analysis Specialist
>>>> Single Cell Facility
>>>> Department of Biosystems Science and Engineering (D-BSSE)
>>>> ETH Zürich
>>>> Office 2.30
>>>> Mattenstrasse 26
>>>> 4058 Basel, Switzerland
>>>> aaron.ponti at bsse.ethz.ch
>>>> T: +41 61 387 33 74
>>>> F: +41 61 387 39 93
>>>>
>>>> *From:* Johan Herz <johan at lambertinstruments.com>
>>>> *Sent:* Mittwoch, 19. Dezember 2018 17:19
>>>> *To:* OME User Support List <ome-users at lists.openmicroscopy.org.uk>
>>>> *Cc:* Lummen Tom <tom.lummen at bsse.ethz.ch>; Ponti Aaron <
>>>> aaron.ponti at bsse.ethz.ch>
>>>> *Subject:* Re: [ome-users] Support for .FLI files from Capture
>>>> software (HiCAM FLUO high-speed camera)
>>>>
>>>> Hello Sebastien,
>>>>
>>>> Thank you for your reply.
>>>>
>>>> I have 2 files, 6 GB each.
>>>> They are made with 2 different versions of Capture our Hi-speed
>>>> recording software.
>>>> We make use of the .fli format and we tried to be as compatible with
>>>> the Bio-formats as possible.
>>>> Unfortunately, we could not work around the 12 bit, hence the
>>>> unsupported data format.
>>>>
>>>> Personally, I have tried to get bio-formats to compile on my one
>>>> machine, I made a checkout, but I ran in too many problems. It seems that
>>>> you are planning to have a new release.
>>>> Because I wanted to fix the problem of not being able to open files
>>>> having a filesize bigger than 2 GB.
>>>>
>>>> So could you take this in to account as well? To open our files
>>>> virtual, FiJi supports this, right?
>>>>
>>>> Is it okay for you if I share the files through Google Drive, as they
>>>> are too big to upload to your server?
>>>>
>>>>
>>>> * Recording_Capture_1.1.fli
>>>> <https://drive.google.com/a/lambertinstruments.com/file/d/1xw43lsucOssXb97ia0JJCCZ9467hV3iO/view?usp=drive_web>*
>>>>
>>>> * Recording_Capture_1.0.4.fli
>>>> <https://drive.google.com/a/lambertinstruments.com/file/d/19NwEUveiphMxUf9QSjtSHSMBthlaPxqW/view?usp=drive_web>*
>>>>
>>>> Also have attached a Recording with our New FLIM system, because
>>>> originally the .fli was for our FLIM system.
>>>> Those files are still compatible. So for testing purposes, I included
>>>> that such that you can make sure it still works.
>>>> See attachment.
>>>>
>>>> Please let me know if you have any further questions.
>>>>
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> Johan Herz, Sales Engineer
>>>> Please visit FLIM.camera <http://flim.camera/> for more information
>>>> about the Toggel.
>>>>
>>>> Tel: +31-50-501-8461 | Skype: lambert-johan
>>>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>>>> Groningen|The Netherlands
>>>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>>>
>>>>
>>>> Op ma 17 dec. 2018 om 15:41 schreef Sebastien Besson (Staff) <
>>>> s.besson at dundee.ac.uk>:
>>>>
>>>> Hi Aaron,
>>>>
>>>>
>>>> On 14 Dec 2018, at 10:00, Ponti Aaron <aaron.ponti at bsse.ethz.ch> wrote:
>>>>
>>>> Hello,
>>>>
>>>> we have .FLI files from Lambert Instruments' Capture software (version
>>>> 1.0.4.0) for the high-speed camera HiCAM FLUO. Images are streamed at high
>>>> speed and stored as UINT12 (2 pixels in 3 bytes) or optionally as UINT8.
>>>> The FLIM reader in bio-formats (
>>>> https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/in/LiFlimReader.java)
>>>> does not seem to support 12-bit intensities.
>>>>
>>>>
>>>> From the internal specification of the LI-FLIM file format which we
>>>> used for the reader [1]:
>>>>
>>>> "The datatype key can have one of the following 8 values: UINT8, INT8,
>>>> UINT16, INT16,
>>>> UINT32, INT32, REAL32 or REAL64.”
>>>>
>>>> A file that would use UINT12 would probably be a new variant of the
>>>> file format for which we
>>>> have currently neither a format specification nor representative samples
>>>>
>>>>
>>>> Also, when asking bio-formats to return the metadata, the whole file is
>>>> returned (which suggests that the metadata of standard FLIM .FLI files is
>>>> organized -- or terminated -- differently).
>>>>
>>>>
>>>> Understood. This would be consistent with the pixel type issue reported
>>>> above i.e. that could
>>>> indeed reflect an internal layout different from the one Bio-Formats is
>>>> expecting..
>>>>
>>>>
>>>>  The file is binary with the first bytes containing metadata
>>>> information in readable text format. One field is 'hasDarkImage'. If
>>>> hasDarkImage == 1, the very last frame is a dark image that is subtracted
>>>> on-the-fly in the software when displaying the frames (but the images are
>>>> stored raw in the file). The 'exposureTime' is in micro seconds. Finally,
>>>> the software seems to support 'compression', although there is no setting
>>>> for turning it on in the software. The software supports binning.
>>>>
>>>>
>>>>
>>>>  Would you be interested in adding support for these files to
>>>> bio-formats? We have example files that we could upload.
>>>>
>>>>
>>>> Over the past years, we have been working on ways to have more
>>>> sustainable support for new file
>>>> formats including any variants of existing file formats. We published a
>>>> blog post summarizing our
>>>> experience a few years ago [2] and since then more manufacturers have
>>>> been engaging in this
>>>> process.
>>>>
>>>> Have you been in touch with Lambert Instruments to report this issue?
>>>> Are they aware that the latest files
>>>> produced by their software are no longer readable by Bio-Formats and
>>>> any software based upon it?
>>>>
>>>> Representative samples allowing to reproduce the issue are always
>>>> welcome. Ideally public samples licensed
>>>> under CC-BY license [3] are the most effective way to help spontaneous
>>>> or sponsored contributions to the
>>>> project and facilitate both the immediate software work as well as the
>>>> long-term maintenance.
>>>> You can upload datasets under 2GB to our QA system [4] and we can
>>>> provide FTP credentials for large samples.
>>>>
>>>> Best,
>>>> Sebastien
>>>>
>>>>
>>>> [1]
>>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/formats/lambert-instruments-flim.html
>>>> [2]
>>>> https://blog.openmicroscopy.org/file-formats/community/2016/08/31/bf-partnerships/
>>>> [3] https://downloads.openmicroscopy.org/images/
>>>> [4] http://qa.openmicroscopy.org.uk/qa/upload/
>>>>
>>>>
>>>>
>>>>  Thanks a lot,
>>>> a2
>>>>
>>>>
>>>>
>>>>
>>>> ----
>>>> Dr. Aaron Ponti
>>>> Software and Data Management Engineer
>>>> Image Analysis Specialist
>>>> Single Cell Facility
>>>> Department of Biosystems Science and Engineering (D-BSSE)
>>>> ETH Zürich
>>>> Office 2.30
>>>> Mattenstrasse 26
>>>> 4058 Basel, Switzerland
>>>> aaron.ponti at bsse.ethz.ch
>>>> T: +41 61 387 33 74
>>>> F: +41 61 387 39 93
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>>
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>> _______________________________________________
>>>> ome-users mailing list
>>>> ome-users at lists.openmicroscopy.org.uk
>>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>>
>>>>
>>>>
>>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20190122/8b2ceed8/attachment.html>