[ome-devel] Uploading ICS files into OMERO and OMERO client API questions

[Nick Perry]

Hi Curtis,

After talking it over on our side and with Sander, we just thought it might
be useful to chip in our ideas about the new OME-XML specification. Before I
get into that, however, could you fill us in on exactly what the plans are
as of now for the new OME-XML/OME-TIFF specifications? Or if nothing is
really concrete what the main ideas are that are being mulled around?

Our ideas about the new OME-XML/OME-TIFF:

1. Should be 'N' dimensional. The obvious perk of this system is that it
allows for any sort of microscopy in the future which could hypothetically
utilize dimensions unknown at this time. Thus we will avoid having similar
'FLIM' compatibility issues like this in the future. One point that I know
Sander wanted to mention is that it is both possible and trivial to
efficiently extract ANY 2-dimensional plane from the N-dimensional data. To
be able to do this, all the data file would need to have stored somewhere
is:

1. The total number of dimensions
2. The size of every dimension
3. The data type of a single data element

Having this information stored somewhere in the data file would allow both
extraction of planes as well as the ability to display the images (for
insight, for example). Again this just means even if the data model is
expanded N-dimensionally it's still possible to easily create the 2d images
in OMERO simply by extracting 2 planes. No knowledge of what the two planes
mean is necessary.

2. In order to have any useful analysis, we think that OMERO should specify
some "standard" dimensions. This specification would require client software
to implement this basic set of dimensions, for example X, Y, Z, Time,
Channel. With what's mentioned above, this software could still display
unknown dimensions even if it doesn't know what they are (for example, it
could still display images with something like phase even if it doesn't know
what to do with it).

3. Sander also suggested that it be possible for vendors to register
dimensions. For example, somewhere in the meta data of the new OME-TIFF
should be the following:

Dimension 0 has size 2 and is the "Channel"
Dimension 1 has size 100 and is the "X"-dimension (width of image)
Dimension 2 has size 200 and is the "Y"-dimension
Dimension 3 has size 12 and is the "Phase" dimension
Dimension 4 has size 30 and is the "Time" dimension

Dimensions 0, 1, 2, 4 are 'standard.' However, Dimension 3 would be
registered, for example, and either the file spec or the website would say
that the dimension is registered and who it is registered by. The point of
this is so that there is an official and standardized way for companies to
introduce new dimensions into OME and not have overlap or have each company
creating it's own special dimensional name for the same type of data. That
way new types of microscopy that enters and wants to be OME compatible will
see the officially registered dimension names, which companies registered
them (to see if they mean 'Phase' in the same way, for example) and either
use that already registered dimension name or register their own if they
need something different.

Also, here are the answers you specifically had for LI-FLIM data:

1. Can LI-FLIM reference/sample files ever have more than 1 channel? Curtis
brought up the example of having spectral lifetime that used multiple
channels.
In practice, Reference and Sample data only have a single channel right now,
because there currently are no LIFA camera's / systems in the field that
produce "multi-color" images. However, LI-FLIM should not have a problem
loading multi-channel reference and sample stacks. It probably won't
calculate lifetimes as I believe I put an artificial check in the software
that only allows single channel source data for lifetime
calculations. (multi-channel-data is currently never produced by current
LIFA systems, so that case is assumed to be an error). I can imagine this
'artificial check' being removed in the future, allowing for multi-spectral
FLIM.

2. Does the lifetime file have anything about maximum intensity at each
pixel, or background at each pixel? I haven't seen either so I think not but
I thought I'd ask.

No. It does use the "NaN" (not-a-number) floating point value for
pixels that should be ignored (because the average intensity fell below
threshold for instance). And we know that lifetimes should roughly be in the
0 ... 150 ns range (units are nanoseconds in the lifetime image data) for
LIFA lifetimes, and, say 150 - 10000 ns for the LIFA-X. The 'intensity'
channel is usually between 0 and 65535.

If those need clarification, let me know.

Thanks again,
Nick

On Wed, Aug 26, 2009 at 12:44 AM, Nick Perry <nperry at stanford.edu> wrote:

> Wow, thank you for that PHENOMENAL response.
>
> On Tue, Aug 25, 2009 at 11:02 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
>
>> Hi Nick,
>>
>> 1. I noticed that I can successfully import .ics files for FLIM data.
>>> However, they can't be visualized within the OMERO server (understandably
>>> since I imagine the algorithm for reconstruction images isn't design for
>>> FLIM data).
>>
>>
>>>
>> ICS is a flexible file format, capable of representing many dimensional
>> organizations. While OMERO (Bio-Formats) is capable of importing ICS data,
>> it may not get the dimensional structure right in every case. In the case of
>> FLIM data, it is assuredly wrong, because the OME data model does not yet
>> support the FLIM dimensions of phase & frequency (in the case of frequency
>> domain) and time bin (in the case of time domain).
>>
>> You can use the Bio-Formats command line tools (
>> http://www.loci.wisc.edu/ome/formats-tools.html) to study how metadata is
>> being mapped from the original structure into the standardized OME model.
>> Try this command:
>>
>>   showinf myData.ics -nopix -omexml
>>
>> The -nopix flag instructs Bio-Formats to skip reading any pixel data, and
>> the -omexml flag instructs it to standardize the metadata into the OME data
>> model.
>>
>> Towards the end of the output, you will see a block of text bracketed by
>> "<OME>" and "</OME>" that describes the standardized dimensional structure.
>> You can also check the SizeC, SizeZ and SizeT values in the "core metadata"
>> output near the top. My guess is that at the moment, Bio-Formats is
>> inappropriately compacting the phase and frequency dimensions into one or
>> more of the Z, C and T dimensions, or else dropping them completely. Either
>> way, the information subsequently pulled into the OMERO database is invalid.
>>
>
> First, in terms of the command line tool: I tried it out earlier today
> trying to see if I could convert ics 1.0 and ics 2.0 files of the lifetime
> data into ome-tifs and just put them in the OMERO server (a la 'smart
> server' concept). However, both times the created ome-tif files were created
> (1 from ics 1.0, the other from ics 2.0), both files were uploaded
> successfully to the server (ie, no errors) but neither of them ever showed
> up in importer. Also, I ran the validation tool provided by the command line
> tool, and neither file passed due to xml tags being missing and such. if
> you're interested (perhaps a bug?) i could send you the error otherwise i'll
> assume it's because it was the lifetime FLIM data and there are some missing
> necessary meta-data needed to constitute a 'readable' ome-tif file. Note:
> the file i used contained X, Y, Z, T, and Channel. There was no
> phase/frequency in this file. So i would of thought it could be converted
> correctly. Interestingly, Heygens could create a readable ome-xml file from
> this and i could as least see it in OMERO with insight.
>
>
>>
>>  Now, until FLIM data, which is 7 dimensional, is somehow fit into the
>>> OME-TIF data model (which would likely require a change in the OME-TIF
>>> specification itself), I wonder if the FLIM data can just be stored on OMERO
>>> servers as .ics files?
>>
>>
>> OMERO is capable of preserving the original data files. However, its
>> primary function is to digest the data (both pixels and metadata) into a
>> standardized form. Bio-Formats converts the metadata into an OME data
>> structure, which is then used to populate the OMERO database. It also
>> decodes the pixels and stores them into a raw pixels repository on disk, for
>> efficient access later.
>>
>> Since the OME data model does not yet support the FLIM-specific
>> dimensions, this digestion will mangle the dimensional parameters of your
>> data
>>
>
> Right. I was thinking more along the lines of skipping the bioformats stage
> at this point until the ome-xml is expanded to allow more dimensions (for
> reasons I discuss below)
>
>
>>
>> Does the OMERO server do anything to change these files as it tries to
>>> interpret them? Or does it just store it like any data storage server would?
>>> Essentially, if I put the FLIM data which is incompatible to any sort of
>>> function OMERO uses right now into .ics files and upload them (they upload
>>> successfully according to importer) would OMERO just store them as they are
>>> and not tamper with the files in any way?
>>
>>
>> Hopefully the above explanation answers these questions. If not, please
>> ask for clarification and we will be happy to explain in more detail.
>>
>>
>
> Am I correct in my understanding that if skip bioformats my raw data is NOT
> digested, and thus i get the unmodified ics version on OMERO?
>
>
>>  If so, could I then access these files through API with analysis software
>>> acting as an OMERO client and edit the files themselves on the server?
>>>
>>
>> With the OMERO API, you can: A) access the digested data for further
>> analysis with your software; and B) access the original, untouched ICS data.
>>
>> In the case of A, the digested data is invalid, due to the lack of support
>> for FLIM. While your software could theoretically untangle any mangling that
>> occurred during import, it would make more sense to wait until OMERO
>> properly supports the FLIM dimensions.
>>
>> In the case of B, your software could process the returned ICS data stream
>> exactly like an ICS file on disk. But in that case, it seems unnecessary to
>> go through the rigamarole of storing the files in OMERO in the first place.
>> Why not use a regular networked file system instead?
>>
>
> Because figuring out how to play with the API furthers my goals even if I'm
> working with incorrect data. Essentially, I imagine I could get complete
> access to the LI-FLIM analysis software, which can work with both ics AND
> eventually the new ome-xml/tif files. So I just need to get the connection
> hooked up and that's half of the battle.
>
>
>>
>> 2. If I have software available that analyzes the specific FLIM data we
>>> use, how long would it take (best guess would do here; I just need some sort
>>> of concept of the scale of this type of project) to interface the OMERO
>>> server with this software via the OMERO API?
>>>
>>
>> It depends on the software, and the type of integration desired. If it is
>> closed source, proprietary software from a vendor, then you have no control
>> over its feature set. However, assuming you are talking about in-house
>> and/or open source software, you have a way forward. One of the cool things
>> about OMERO is its use of Ice (http://www.zeroc.com/ice.html), allowing
>> cross-platform client access to the server from a number of different
>> programming paradigms including C++, Java, .NET, Python, PHP, Ruby, and
>> Objective-C.
>>
>> There are different kinds of clients, as well. The server is capable of
>> delivering compressed, rendered image planes on demand, reducing the
>> processing burden on the client as well as the necessary network bandwidth
>> -- this approach is how the OMERO.insight client works. But the server can
>> also deliver raw image planes for the client to use with completely custom
>> processing.
>>
>
> See above.
>
>
>>
>> I would also like to respond to some points you raised in an email to
>> Kevin:
>>
>> Part 1) Get LI-FLIM data inside OMERO (what you called the smart file
>>> server).
>>> Part 2) Add support for LI-FLIM analysis be in it the form of a module,
>>> direct implementation into the source code of OMERO, etc (I am more unclear
>>> at this point how this portion would work).
>>>
>>
>> I agree that these two steps are what is needed. However, accomplishing
>> Part 1 requires an extension to the OMERO data model, as described above,
>> since OMERO does more than just storing original data.
>>
>
> In an email to Sander, I re-broke-down how I see the problem:
>
> 1. The OME-TIF data model would be fixed to allow reference/sample data.
> 2. LI-FLIM software would output reference/sample data as OME-TIF.
> 3. LI-FLIM software would output lifetime data as OME-TIF.
> 4. Visualization software would be implemented inside of OMERO so that
> reference/sample/lifetime images can be correctly visualized once connected
> to the server (internally, this is all the server can do; it just shows the
> picture with basic image editing functions; any true image analysis happens
> outside the server with user built clients).
> 5. Use the OMERO server API to connect the EXISTING LI-FLIM software to the
> OMERO server to perform analysis (new idea I just had that I'm pretty sure
> works but need to confirm).
>
> Essentially, I mention that (1) must happen before (2) can happen. (3) can
> happen without (1) but would eventually change as (1) changes. However, I
> don't see working on 3 being a step sideways - i imagine coding that
> solution would expedite getting lifetime data into the new ome-xml
> specification when that happens too, so it' just a small step forwards but
> forwards nonetheless. (4) would probably need to happen after 1-3, and (5)
> is something I think could happen now as well (the API exists, I have the
> LI-FLIM software)
>
>
>>
>> Part 1 is ideally what I could contribute to during the rest of my stay.
>>> Here is a more in-depth explanation of what I'm thinking:
>>>
>>> When you use the LI-FLIM system, you take both Sample and Reference
>>> images (stored in two separate files sample.fli and reference.fli). Each of
>>> these images is 7 dimensional in terms of the file format.
>>>
>>
>> The first step on our end is to add support for the LI-FLIM (.fli) file
>> format. We have the specification and some sample data from Sander de Jong,
>> so it is only a matter of time until we have an initial implementation. I am
>> planning to work on it later this week, actually.
>>
>> However, Bio-Formats must be able to successfully map the additional
>> dimensions into the OME data model before the files can be robustly imported
>> into an OMERO server. This means extending the OME-XML data model. The OMERO
>> developers have been actively discussing how to do so in recent weekly
>> meetings, passing proposals back and forth, etc., and will share these with
>> the community as soon as we have something reasonable.
>>
>> Once you have collected the sample and reference files, right now you load
>>> them into the LI-FLIM analysis software and IMMEDIATELY a lifetime image is
>>> created. No analysis is done at this point. From this point, everything you
>>> need for analysis is contained within the lifetime file. This lifetime file
>>> is also the same file format as the sample and reference files were.
>>>
>>> On a side-note, just for completeness, it seems that the reason the
>>> LI-FLIM file format is 7 dimensions is so that the format is broad enough to
>>> hold reference/sample data and lifetime data, which are fundamentally
>>> different.
>>>
>>
>> Right; I have not yet seen a file that actually uses all 7 dimensions
>> simultaneously.
>>
>> Reference/sample data only use X, Y, Z, T, Phase, and Frequency (channel
>>> is always 1, at least as far as I can tell with my demo software, so Sander
>>> should confirm). These files are therefore 6 dimensional.
>>>
>>
>> In the case of combined spectral lifetime (SLIM) data, where the emission
>> data is split across multiple wavelengths, channels could be greater than
>> one. Not sure if any Lambert systems are capable of this, but we (LOCI) have
>> a home-built system based on the Becker & Hickl time domain approach that
>> does SLIM.
>>
>
> I am not sure but I will get back to you on this.
>
>
>>
>>
>> Lifetime data uses only X, Y, Z, T, Channel (phase and frequency are
>>> always 1, but again need confirmation). So lifetime.fli files are 5
>>> dimensional, and use the same dimensions as the OME-TIF. So as far as
>>> CRUCIAL metadata is concerned, the OME-TIF format suffices.
>>>
>>
>> Right. Fundamentally, "lifetime data" means that the pixel values have
>> been computed to match the expected lifetime values. Presumably, in the case
>> of multiple lifetime values (e.g., two-component), the lifetime data files
>> would have multiple channels? Also, what about other parameters such as
>> maximum intensity at each pixel, or background at each pixel? (These values
>> are more applicable to time domain; I am still getting my head around
>> frequency domain so perhaps they are irrelevant to the frequency approach.)
>>
>
> Yes. In the lifetime files I received from Sander (and I believe you did as
> well) the extended files had lifetime 1 and lifetime 2, as well as some
> other channels. I don't believe there is background and I'm not sure about
> max intensity, so I will get back to you with answers to those as well as
> soon as I confirm.
>
>
>>
>>
>> The reason I thought it is useful to be able to put lifetime data into
>>> OMERO and be able to analyze that, as opposed to reference/sample data, is
>>> that as far as I can tell, the computer connected to the LI-FLIM hardware
>>> itself needs to run the LI-FLIM software.
>>
>>
>> There is no reason you shouldn't be able to perform analysis of the
>> reference/sample data after the fact. So ideally we would like to support
>> storing the reference/sample data into the OMERO system.
>>
>
> Yeah I completely misunderstood how analysis of FLIM works. Sander
> corrected me :)
>
>
>>
>> I don't get the impression OMERO can control microscopes, but perhaps I'm
>>> wrong.
>>
>>
>> You are correct. OMERO is not intended as a tool for acquisition, though
>> we have made an effort to support data from many different acquisition
>> systems, including the open source Micro-Manager software.
>>
>> As such, because we still need to be able to go through the LI-FLIM
>>> software at least initially, why not just let that software compile the raw
>>> sample/reference data into the lifetime data, which can readily fit into the
>>> existing OME data model since it uses the same 5 dimensions? This would save
>>> the hassle (at least for now) of re-designing the entire datamodel for
>>> OME-TIF.
>>
>>
>> Sure, makes sense.
>>
>>
>>> Once the lifetime image is collected on the LI-FLIM software running the
>>> hardware, then, that software could just save the lifetime data into the
>>> OME-TIF format since that's the only important file for image analysis at
>>> this point (per Sander's confirmation), and save the reference/sample data
>>> as .ics. Then all 3 files could be put into OMERO which would ultimately
>>> develop analysis tools (Sander could help here?) for analyzing the lifetime
>>> file.
>>
>>
>> Storing the reference/sample data as ICS and into the OMERO server is
>> asking for trouble at the moment, as explained above.
>>
>
> BUT, it would also work for now if we have need of storing files
> immediately (right?).
>
>
>>
>>  That's how I figured the workflow would work anyway - the LI-FLIM
>>> software connected to the FLIM hardware just outputs the lifetime file,
>>> which would go into OMERO for storage and analysis, and which stores
>>> reference/sample .ics files for completeness.
>>
>>
>> There is an attachment mechanism in OMERO, to store arbitrary binary data
>> (such as PDFs, PPTs, etc.) along with an image. If you are intent on this
>> workflow, perhaps it would make more sense to store the original FLI files
>> as binary attachments, rather than parsing them in any way at the moment.
>> Then they would still be recoverable for processing with the LI-FLIM
>> software at a later time.
>>
>
> I think this workflow might make sense since at least I would be
> contributing to forward progress.
>
>
>>
>> This also solves the metadata storage issue because all the metadata from
>>> the reference/sample files is in the lifetime file as well. And this data
>>> can easily just be put into the XML part of the OME-TIF (it's hardware stuff
>>> only, like something=x, something=y, etc.).
>>>
>>
>> We still need to define a mapping from the FLI hardware metadata into the
>> standard OME data model. Best would be if the LI-FLIM software could export
>> to OME-TIFF -- but it doesn't do that now, does it? We will also want to do
>> the mapping in Bio-Formats, as an FLI file format reader, so that OMERO (and
>> other software packages that use Bio-Formats) can import the original FLI
>> files properly.
>>
>
> Is there not a CustomAttribute tag? I thought I saw one earlier today. I
> recognize that this is not the permanent solution but if I were to work on
> getting the lifetime files into ome-tif form I could just throw LI-FLIM
> specific metadata for hardware into that tag, yes?
>
> Basically, to summarize all of the above (since it was getting a little
> choppy):
>
> The OME-XML format needs to change to accommodate additional dimensions
> (for FLIM in this case, phase and frequency). However, this solution will
> NOT happen in my timeframe; I leave in 3 weeks. To at least do SOMETHING to
> help, I thought it would be beneficial to:
>
> 1) Help Sander output lifetime FLIM files in OME-TIF as the file format is
> specified currently, since this code would also probably be useful in
> converting lifetime FLIM files into the new OME-XML whenever it is released
> (and thus represents forward progress even if it's not a complete solution).
>
> 2) Work on hooking up the LI-FLIM analysis software to the OMERO server via
> the provided API. This would need to happen anyway to analyze the LI-FLIM
> data.
>
> The extra perk of both of these in addition to providing useful code is
> that no matter how we store the FLIM data right now (since the OME-XML might
> take a while) the Plateforme can start using OMERO to store data and run
> analysis off of it. So yes, the reference/sample data will be in either
> ICS/binary, and the lifetime will be in an eventually out-dated OME-XML, but
> we can at least use the benefits of the 'smart file server' provided by
> OMERO while at the same time create code that could ultimately be used to
> help output OME-TIF files directly from LI-FLIM software as well as work
> with the API for analysis clients for FLIM.
>
>
>>
>>
>> HTH,
>> Curtis
>>
>
> Again, thank you SO MUCH for the response. It was incredibly helpful!
>
> Nick
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20090826/6b40a687/attachment-0001.htm