Hi Nick,<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>
First, in terms of the command line tool: I tried it out earlier today
trying to see if I could convert ics 1.0 and ics 2.0 files of the
lifetime data into ome-tifs and just put them in the OMERO server (a la
'smart server' concept).</div></div></blockquote><div><br>There should be no reason to convert from ICS to OME-TIFF before
uploading to OMERO, because OMERO supports the ICS format directly.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote">
<div>However, both times the created ome-tif files
were created (1 from ics 1.0, the other from ics 2.0), both files were
uploaded successfully to the server (ie, no errors) but neither of them
ever showed up in importer.</div></div></blockquote><div><br>Do you mean "neither of them ever showed up in the OMERO.insight client"? If so, I am not sure why Insight would fail to display the successfully imported files.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div> Also, I ran the validation tool provided by
the command line tool, and neither file passed due to xml tags being
missing and such. if you're interested (perhaps a bug?) i could send
you the error otherwise i'll assume it's because it was the lifetime
FLIM data and there are some missing necessary meta-data needed to
constitute a 'readable' ome-tif file. Note: the file i used contained
X, Y, Z, T, and Channel. There was no phase/frequency in this file. So
i would of thought it could be converted correctly. Interestingly,
Heygens could create a readable ome-xml file from this and i could as
least see it in OMERO with insight.<br></div></div></blockquote><div> </div><div>Most of those validation errors are due to discrepancies in the order that elements appear in the XML. The schema requires that the elements be output in a particular order, but Bio-Formats does not respect that order. So there are validation errors. It is a known problem we are working to address in the coming months.<br>
<br>It is unlikely that the validation errors had much to do with the fact that ICS files were lifetime image data.<br><br>Huygens is a solid software package, and their developers have done careful work to ensure that the OME-XML they produce is valid. So it is no surprise that their OME-XML export is working well for you. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Right. I was thinking more along the lines of skipping the bioformats stage at this point until the ome-xml is expanded to allow more dimensions (for reasons I discuss below)<br>
</div></div></blockquote><div><br>To be clear: there is no such thing as "skipping the Bio-Formats stage." OMERO uses Bio-Formats internally to import the data. The root of the OME data model is the OME-XML schema. On top of that is the OME-XML Java library, and atop that, Bio-Formats. These are core layers of code that everything we have produced uses.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Am I correct in my understanding that if skip bioformats my raw data is NOT digested, and thus i get the unmodified ics version on OMERO?<br>
</div></div></blockquote><div><br>The only way to avoid any digestion is to use the "file attachment" mechanism I discussed before. One of the OMERO developers can comment further on how to do this, if you are interested. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div>Why not use a regular networked file system instead?<br>
</div></div></blockquote></div><div><br></div></div></blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>
Because figuring out how to play with the API furthers my goals even if I'm working with incorrect data. Essentially, I imagine I could get complete access to the LI-FLIM analysis software, which can work with both ics AND eventually the new ome-xml/tif files. So I just need to get the connection hooked up and that's half of the battle.<br>
</div></div></blockquote><div><br>Fair enough; that makes sense. I see from another thread you have begun the long, fun journey into C++ land. Best of luck with that! :-)<br> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div>In an email to Sander, I re-broke-down how I see the problem:<br><br>1. The OME-TIF data model would be fixed to allow reference/sample data.<br>
2. LI-FLIM software would output reference/sample data as OME-TIF.<br>3. LI-FLIM software would output lifetime data as OME-TIF.<br></div></div></blockquote><div><br>All good goals!<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div>4.
Visualization software would be implemented inside of OMERO so that
reference/sample/lifetime images can be correctly visualized once
connected to the server (internally, this is all the server can do; it
just shows the picture with basic image editing functions; any true
image analysis happens outside the server with user built clients).<br>
</div></div></blockquote><div><br>Presumably by "inside of OMERO" you mean "on the server side." Yes, in an ideal world, the server would provide basic lifetime analysis, delivering rendered lifetime images to OMERO.insight and other clients.<br>
<br>We (LOCI) have an open source lifetime curve fitting package (for time domain analysis) in the works, though it will be a while before it matures. We also want to provide support for processing frequency domain data. Once this package is ready, it could be integrated into the OMERO server to provide this kind of functionality. But we're looking at a 1-to-3 year timeline for that, most likely.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>5. Use the OMERO server API to connect the EXISTING LI-FLIM software to
the OMERO server to perform analysis (new idea I just had that I'm
pretty sure works but need to confirm).<br></div></div></blockquote><div><br>If you have access to the LI-FLIM source code, you can pursue this, yes. And we would be excited to assist with any third party software adding direct OMERO connectivity!<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Essentially, I mention that (1) must happen before (2) can happen. (3) can happen without (1) but would eventually change as (1) changes. However, I don't see working on 3 being a step sideways - i imagine coding that solution would expedite getting lifetime data into the new ome-xml specification when that happens too, so it' just a small step forwards but forwards nonetheless.</div>
</div></blockquote><div><br>Yes, step #3 could happen now and I agree that it is not a step sideways, but a step forward.<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div>(4) would probably need to happen after 1-3, and (5) is something I think could happen now as well (the API exists, I have the LI-FLIM software)<br>
</div></div></blockquote><div><br>Yes, you can begin work on #5, but there will be limits to what the API can accomplish. At the least, you should be able to store and retrieve the processed lifetime images from the OMERO server using the API. But until the OMERO server itself has lifetime analysis capabilities (as I mentioned above), there will be a limit to how functional the connection can be.<br>
</div><div> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Yes. In the lifetime files I received from Sander (and I believe you did as well) the extended files had lifetime 1 and lifetime 2, as well as some other channels. I don't believe there is background and I'm not sure about max intensity, so I will get back to you with answers to those as well as soon as I confirm.<br>
</div></div></blockquote><div><br>Yes, we have the sample data as well, and I plan to work on a LI-FLIM reader this week. Glad to know the data contains multiple lifetime values per pixel.<br> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Storing the reference/sample data as ICS and into the OMERO server is asking for trouble at the moment, as explained above.<br>
</div></div>
</blockquote></div><div><br>BUT, it would also work for now if we have need of storing files immediately (right?).<br></div></div></blockquote><div><br>No, it would make more sense to store them as file attachments, rather than allowing the server to digest them incorrectly.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div>There is an attachment mechanism in OMERO, to store arbitrary binary data (such as PDFs, PPTs, etc.) along with an image. If you are intent on this workflow, perhaps it would make more sense to store the original FLI files as binary attachments, rather than parsing them in any way at the moment. Then they would still be recoverable for processing with the LI-FLIM software at a later time.<br>
</div></div></blockquote></div><div><br>I think this workflow might make sense since at least I would be contributing to forward progress.<br></div></div></blockquote><div><br>OK, feel free to send questions to the list asking about the attachment mechanism if you need more information. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>Is there not a CustomAttribute tag? I thought I saw one earlier today. I recognize that this is not the permanent solution but if I were to work on getting the lifetime files into ome-tif form I could just throw LI-FLIM specific metadata for hardware into that tag, yes?<br>
</div></div></blockquote><div><br>Yes, for LI-FLIM-specific metadata when importing lifetime files, you could use the StructuredAnnotations (SAs) to record the information as custom fields ("CustomAttributes" are the old, deprecated way to do this). Let the list know if you need any help with SAs.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>The OME-XML format needs to change to accommodate additional dimensions (for FLIM in this case, phase and frequency). However, this solution will NOT happen in my timeframe; I leave in 3 weeks. To at least do SOMETHING to help, I thought it would be beneficial to:<br>
<br>1) Help Sander output lifetime FLIM files in OME-TIF as the file format is specified currently, since this code would also probably be useful in converting lifetime FLIM files into the new OME-XML whenever it is released (and thus represents forward progress even if it's not a complete solution).<br>
</div></div></blockquote><div><br>Yes, just identifying where in OME-XML each piece of FLIM metadata should go is a big step forward. Chances are you will develop a list of metadata fields that have no proper place to go; such a list would be very useful to us in extending the model to accommodate FLIM data. <br>
</div><div> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="gmail_quote"><div>2) Work on hooking up the LI-FLIM analysis software to the OMERO server via the provided API. This would need to happen anyway to analyze the LI-FLIM data. <br>
</div></div></blockquote><div><br>Yes, having the ability to access the processed lifetime data (or any regular image data) from OMERO, importing it into the LI-FLIM software, would be beneficial to whoever continues to maintain the LI-FLIM software after you leave. It would serve as an example of connecting to OMERO, reducing future development time when adding a complete link to the reference/sample data in OMERO (once it is being digested properly).<br>
</div></div><br>-Curtis<br><br>