[ome-users] Support for .FLI files from Capture software (HiCAM FLUO high-speed camera)

Fri Jan 18 14:16:22 GMT 2019

Hello Sebastien,

Thank you and your welcome.
one of the reasons we use uint12 is that it preserves disk space. If we
finally present the data in our viewer we upscale it to 16bit.
When do you think uint12 will become available in the core metadata? Would
it be an option to add an 'unpacker'-function to the LiFLIMReader, that
unpacks and upscales the uint12 to uint16 data? But can the data still be
opened virtually in that case?

Best regards,

Johan Herz, Sales Engineer

Please visit FLIM.camera for more information about the Toggel.

Tel: +31-50-501-8461 | Skype: lambert-johan
Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
Groningen|The Netherlands
Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com

Op di 15 jan. 2019 om 12:29 schreef Sebastien Besson (Staff) <
s.besson at dundee.ac.uk>:

> Hi Johan,
>
> Thank you very much for opening the Pull Request and sending us
> representative
> samples of the issue. We will review it at earliest and hopefully have at
> least this fix
> released as part of Bio-Formats 6.
>
> One comment regarding the data format.The pixel type is part of what we
> call core
> metadata i.e. metadata mandatory for describing each image in a fileset.
> It can only
> take a limited number of values [1]. This is partly why the mapping of
> this particular
> metadata field is much stricter than other types of keys [2] and will need
> to be updated
> In order to extend support for new data types.
>
> Best,
> Sebastien
>
> [1] *https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType
> <https://www.openmicroscopy.org/Schemas/Documentation/Generated/OME-2016-06/ome_xsd.html#PixelType>*
> [2]
> https://github.com/openmicroscopy/bioformats/blob/v5.9.2/components/formats-gpl/src/loci/formats/in/LiFlimReader.java#L496
>
> On 14 Jan 2019, at 19:17, Johan Herz <johan at lambertinstruments.com> wrote:
>
> Hello Sebastien,
>
> The GB file size problem persisted, and we think the following pull
> request will fix it:
> https://github.com/openmicroscopy/bioformats/pull/3303
>
> If you have any reconmondations please let me know.
>
> Regarding implementing the unsigned 12 bit data format, please let me know
> if I can be of any further help.
>
> Many thabks for your support.
>
> Best,
> Johan
>
> Op ma 14 jan. 2019 10:57 schreef Sebastien Besson (Staff) <
> s.besson at dundee.ac.uk:
>
>> Hi Johan and Aaron,
>>
>> Happy New Year to you guys too. I am reincluding the Users mailing lists
>> in this
>> conversation.
>>
>> On 11 Jan 2019, at 08:11, Johan Herz <johan at lambertinstruments.com>
>> wrote:
>>
>> Hello Sebastien and Ponti,
>>
>> Thank you for the follow-up, all to you too a very good 2019!
>>
>> @Aaron, yes I will be available for testing, my pleasure.
>>
>> @Sebastien,  The FLI format for the Hi-speed is not yet very well
>> documented, it is basically based on the original FLI format with a few new
>> data types. We also support color cameras. But from our experience, if we
>> added a header entry that the bio-formats FLI reader did not recognize, it
>> would just ignore it. Unless it makes a collision with another key entry.
>>
>> So basically the main issues currently are the Data type and the 2gb File
>> size max.
>> It will become possible to open very big files virtually, right? Because
>> we can easily make files that are 50GB or bigger so it can most of the time
>> never fit the RAM memory.
>>
>>
>>
>> About 2GB, I assume you are referring to the issue described in this
>> section of the
>> technical documentation [1]? Briefly, we are routinely dealing with
>> datasets in the
>> GB-TB magnitude so file size per se should not be a problem.
>>
>> Am I correct in assuming you are loading data using the ImageJ/Fiji
>> Bio-Formats
>> Plugin? In that case opening as a virtual stack is definitely the first
>> thing to do. For
>> images where Individual planes exceed 2GB limit (due to dimensions or
>> pixel type),
>> you might need to use the Crop option to load images in tiles rather than
>> whole planes.
>>
>> [1]
>> https://docs.openmicroscopy.org/bio-formats/5.9.2/about/bug-reporting.html
>>
>>
>> Regarding compiling Bio-formats, I probably should have given it more
>> time, which I did not had, unfortunately, to have given the manual a better
>> read. What IDE are you using or are you compiling command line?
>>
>>
>> Generally this documentation page is the landing page for everything
>> related to building
>> Bio-Formats from source [2]. In general, people will use either the
>> command-line or an IDE,
>> Eclipse being probably the most popular one. We should be able to offer
>> guidance for both
>> paths.
>>
>> Best,
>> Sebastien
>>
>> [2]
>> https://docs.openmicroscopy.org/bio-formats/5.9.2/developers/building-bioformats.html
>>
>>
>> I am looking forward to receiving the fix
>>
>> Best regards,
>>
>> Johan Herz, Sales Engineer
>>
>> Please visit FLIM.camera <http://flim.camera/> for more information
>> about the Toggel.
>>
>> Tel: +31-50-501-8461 | Skype: lambert-johan
>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>> Groningen|The Netherlands
>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>
>>
>> Op di 8 jan. 2019 om 18:04 schreef Sebastien Besson (Staff) <
>> s.besson at dundee.ac.uk>:
>>
>>> Hi Johan and Aaron,
>>>
>>> Thanks for making example images available, we will copy them internally
>>> for testing purposes.
>>> Is an amended version of the technical documentation described the
>>> format changes also available?
>>>
>>> So far we have captured this new variant of the .fli format in the
>>> general Bio-Formats backlog [1].
>>> In terms of priority, the academically-funded team will primarily focus
>>> on next-generation open formats
>>> In 2019, starting with the release of Bio-Formats 6 [2]. Although we
>>> will not be able to prioritise such work,
>>> we will happily review any direct community contribution to update the
>>> reader. An alternate possibility solution
>>> would be to get in touch with Glencoe Software and commission the reader
>>> update.
>>>
>>> Trying to work in incremental steps and solve the immediate blockers,
>>> Johan, could you give us more details
>>> into the type of issues you ran into while compiling Bio-Formats?
>>>
>>> Best,
>>> Sebastien
>>>
>>>
>>> [1] https://trello.com/c/OGGEzj7X/305-li-flim-12-bit-format
>>> [2]
>>> https://blog.openmicroscopy.org/file-formats/community/2018/11/29/ometiffpyramid/
>>>
>>>
>>>
>>>
>>> On 7 Jan 2019, at 09:28, Ponti Aaron <aaron.ponti at bsse.ethz.ch> wrote:
>>>
>>> Hi,
>>>
>>> Happy new year, everyone! Sorry for the long silence about the FMI
>>> issue. I don’t think I will soon find the time to implement support for
>>> 12-but FLI files in bio-formats myself. I see that Johan replied to the
>>> thread and provided example images. Would you guys work together to sort
>>> out the issue? That’d be great!
>>>
>>> Thanks a lot!
>>> a2
>>>
>>> ----
>>> Dr. Aaron Ponti
>>> Software and Data Management Engineer
>>> Image Analysis Specialist
>>> Single Cell Facility
>>> Department of Biosystems Science and Engineering (D-BSSE)
>>> ETH Zürich
>>> Office 2.30
>>> Mattenstrasse 26
>>> 4058 Basel, Switzerland
>>> aaron.ponti at bsse.ethz.ch
>>> T: +41 61 387 33 74
>>> F: +41 61 387 39 93
>>>
>>> *From:* Johan Herz <johan at lambertinstruments.com>
>>> *Sent:* Mittwoch, 19. Dezember 2018 17:19
>>> *To:* OME User Support List <ome-users at lists.openmicroscopy.org.uk>
>>> *Cc:* Lummen Tom <tom.lummen at bsse.ethz.ch>; Ponti Aaron <
>>> aaron.ponti at bsse.ethz.ch>
>>> *Subject:* Re: [ome-users] Support for .FLI files from Capture software
>>> (HiCAM FLUO high-speed camera)
>>>
>>> Hello Sebastien,
>>>
>>> Thank you for your reply.
>>>
>>> I have 2 files, 6 GB each.
>>> They are made with 2 different versions of Capture our Hi-speed
>>> recording software.
>>> We make use of the .fli format and we tried to be as compatible with the
>>> Bio-formats as possible.
>>> Unfortunately, we could not work around the 12 bit, hence the
>>> unsupported data format.
>>>
>>> Personally, I have tried to get bio-formats to compile on my one
>>> machine, I made a checkout, but I ran in too many problems. It seems that
>>> you are planning to have a new release.
>>> Because I wanted to fix the problem of not being able to open files
>>> having a filesize bigger than 2 GB.
>>>
>>> So could you take this in to account as well? To open our files
>>> virtual, FiJi supports this, right?
>>>
>>> Is it okay for you if I share the files through Google Drive, as they
>>> are too big to upload to your server?
>>>
>>>
>>> * Recording_Capture_1.1.fli
>>> <https://drive.google.com/a/lambertinstruments.com/file/d/1xw43lsucOssXb97ia0JJCCZ9467hV3iO/view?usp=drive_web>*
>>>
>>> * Recording_Capture_1.0.4.fli
>>> <https://drive.google.com/a/lambertinstruments.com/file/d/19NwEUveiphMxUf9QSjtSHSMBthlaPxqW/view?usp=drive_web>*
>>>
>>> Also have attached a Recording with our New FLIM system, because
>>> originally the .fli was for our FLIM system.
>>> Those files are still compatible. So for testing purposes, I included
>>> that such that you can make sure it still works.
>>> See attachment.
>>>
>>> Please let me know if you have any further questions.
>>>
>>>
>>>
>>> Best regards,
>>>
>>>
>>> Johan Herz, Sales Engineer
>>> Please visit FLIM.camera <http://flim.camera/> for more information
>>> about the Toggel.
>>>
>>> Tel: +31-50-501-8461 | Skype: lambert-johan
>>> Lambert Instruments BV|Leonard Springerlaan 19 (5th floor)|9727 KB
>>> Groningen|The Netherlands
>>> Dutch Chamber of Commerce nr.: 52396940 | www.lambertinstruments.com
>>>
>>>
>>> Op ma 17 dec. 2018 om 15:41 schreef Sebastien Besson (Staff) <
>>> s.besson at dundee.ac.uk>:
>>>
>>> Hi Aaron,
>>>
>>>
>>> On 14 Dec 2018, at 10:00, Ponti Aaron <aaron.ponti at bsse.ethz.ch> wrote:
>>>
>>> Hello,
>>>
>>> we have .FLI files from Lambert Instruments' Capture software (version
>>> 1.0.4.0) for the high-speed camera HiCAM FLUO. Images are streamed at high
>>> speed and stored as UINT12 (2 pixels in 3 bytes) or optionally as UINT8.
>>> The FLIM reader in bio-formats (
>>> https://github.com/openmicroscopy/bioformats/blob/develop/components/formats-gpl/src/loci/formats/in/LiFlimReader.java)
>>> does not seem to support 12-bit intensities.
>>>
>>>
>>> From the internal specification of the LI-FLIM file format which we used
>>> for the reader [1]:
>>>
>>> "The datatype key can have one of the following 8 values: UINT8, INT8,
>>> UINT16, INT16,
>>> UINT32, INT32, REAL32 or REAL64.”
>>>
>>> A file that would use UINT12 would probably be a new variant of the file
>>> format for which we
>>> have currently neither a format specification nor representative samples
>>>
>>>
>>> Also, when asking bio-formats to return the metadata, the whole file is
>>> returned (which suggests that the metadata of standard FLIM .FLI files is
>>> organized -- or terminated -- differently).
>>>
>>>
>>> Understood. This would be consistent with the pixel type issue reported
>>> above i.e. that could
>>> indeed reflect an internal layout different from the one Bio-Formats is
>>> expecting..
>>>
>>>
>>>  The file is binary with the first bytes containing metadata information
>>> in readable text format. One field is 'hasDarkImage'. If hasDarkImage == 1,
>>> the very last frame is a dark image that is subtracted on-the-fly in the
>>> software when displaying the frames (but the images are stored raw in the
>>> file). The 'exposureTime' is in micro seconds. Finally, the software seems
>>> to support 'compression', although there is no setting for turning it on in
>>> the software. The software supports binning.
>>>
>>>
>>>
>>>  Would you be interested in adding support for these files to
>>> bio-formats? We have example files that we could upload.
>>>
>>>
>>> Over the past years, we have been working on ways to have more
>>> sustainable support for new file
>>> formats including any variants of existing file formats. We published a
>>> blog post summarizing our
>>> experience a few years ago [2] and since then more manufacturers have
>>> been engaging in this
>>> process.
>>>
>>> Have you been in touch with Lambert Instruments to report this issue?
>>> Are they aware that the latest files
>>> produced by their software are no longer readable by Bio-Formats and any
>>> software based upon it?
>>>
>>> Representative samples allowing to reproduce the issue are always
>>> welcome. Ideally public samples licensed
>>> under CC-BY license [3] are the most effective way to help spontaneous
>>> or sponsored contributions to the
>>> project and facilitate both the immediate software work as well as the
>>> long-term maintenance.
>>> You can upload datasets under 2GB to our QA system [4] and we can
>>> provide FTP credentials for large samples.
>>>
>>> Best,
>>> Sebastien
>>>
>>>
>>> [1]
>>> https://docs.openmicroscopy.org/bio-formats/5.9.2/formats/lambert-instruments-flim.html
>>> [2]
>>> https://blog.openmicroscopy.org/file-formats/community/2016/08/31/bf-partnerships/
>>> [3] https://downloads.openmicroscopy.org/images/
>>> [4] http://qa.openmicroscopy.org.uk/qa/upload/
>>>
>>>
>>>
>>>  Thanks a lot,
>>> a2
>>>
>>>
>>>
>>>
>>> ----
>>> Dr. Aaron Ponti
>>> Software and Data Management Engineer
>>> Image Analysis Specialist
>>> Single Cell Facility
>>> Department of Biosystems Science and Engineering (D-BSSE)
>>> ETH Zürich
>>> Office 2.30
>>> Mattenstrasse 26
>>> 4058 Basel, Switzerland
>>> aaron.ponti at bsse.ethz.ch
>>> T: +41 61 387 33 74
>>> F: +41 61 387 39 93
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>>
>>>
>>>
>>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>>
>>
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20190118/a81288be/attachment.html>