<div dir="ltr">Thanks, Simon.<div><br></div><div>Our general approach is to assume that CellProfiler measurements are only reproducible using the same analysis which includes the same software revision, pipeline and image data. Using a classifier across experiments often introduces an unacceptable level of variability anyway (biological protocol, robotics, microscope setup, etc), so we usually apply the same statistical methods using the new data. In cases where we do try to compare across experimental batches, we always (I think) use the same software revision on both batches and will rerun the analysis on old data if we need to change software versions.</div>
<div><br></div><div>More discussion below...<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 22, 2014 at 9:41 AM, Simon Li <span dir="ltr"><<a href="mailto:s.p.li@dundee.ac.uk" target="_blank">s.p.li@dundee.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi all<br>
<br>
I've been working with Chris Coletta on getting the latest version of WND-CHARM to work with OMERO, and I've stumbled into the problem of how to version objects such as calculated features in OMERO. I thought this might be relevant for others on the list, and also ties in with the ongoing OMERO.features discussion. </blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Example use cases:<br>
<br>
1. I've calculated and stored some features with v1.5 of WND-CHARM. Later I upgrade WND-CHARM to v2.0. How do I know that my v1.5 features may need to be either converted or re-calculated to v2.0.</blockquote><div>We always recalculate. Rerunning an analysis isn't very expensive for us. Conversion generally doesn't work - the effects of the version change can't be reduced to a mapping of old to new values.</div>
<div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> Similarly for a trained classifier.<br></blockquote>
<div><br></div><div>Often the ground truth / biologist annotations can be reapplied to the new data to train new classifiers. For WND-CHARM, I'm guessing the ground truth is positive and negative controls and it should be very easy to retrain.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
2. I've upgraded WND-CHARM. Can I run a script which finds all WND-CHARM 1.5 objects and converts them into v2.0 format?<br>
<br></blockquote><div>I'd hope that WND-CHARM has annotated each measurement column with the version. It should be possible to iterate over the columns and determine which measurements require update. For both CellProfiler and WND-CHARM, the cost of updating a single measurement might</div>
<div>be of the same order of magnitude or identical to rerunning the entire analysis and in general, either all or none of the measurements will require update. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
We might also want to think about versioning the same object from a user's perspective, e.g. if they redo their analysis with different parameters and save it into OMERO they might want to keep their previous versions.<br>
</blockquote><div><br></div><div>Yup yup - with CP, I'd probably do a two-level annotation and give each measurement an experiment ID and store both the software version and parameterization as measurements indexed by experiment ID. That would give you only a single value to check. </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
A few initial ideas:<br>
* Namespace. Annotations in OMERO have a namespace (string) field, e.g. If the WND-CHARM results are stored in a file attached to an OMERO object this will be in the form of a FileAnnotation, so the version could be incorporated, e.g.:<br>
<br>
namespace=/Classifer/WndCharm/1.5<br></blockquote><div>I like this. Two cases - informal matching, you can match just the head of the string (did I use WND-CHARM or CellProfiler? give me all the CellProfiler measurements). Formal matching - string must match exactly (I want my experiment analyzed with /Classifier/WndCharm/2.0, is it?) </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* TextAnnotation. Can be added to anything, and could be used as a key-value pair. Note this differs from the previous in that the TextAnnotation is a completely separate object which is linked to the saved WND-CHARM object:<br>
<br>
namespace=/Classifer/WndCharm/Version<br>
value=1.5<br></blockquote><div>Meh... you can just have a table with columns of namespace, version, software name, etc. It's OK with me, just prefer the namespace. And in our environment, it's rare that we do an analysis with a release version - namespace = 26c38412a1779b8135585fc014495f1488956601 means "CellProfiler 2.1.0.rc12". Rats - I tried googling the GIT hash, didn't work.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* Key-Value pairs in OMERO. This work is currently ongoing.<br>
<br>
Thinking further ahead Gianluigi Zanetti's group have been working on bringing a graph database into OMERO [1], which would allow us to store the full history of all calculations (source data, all transformations, parameters, multiple runs, etc).<br>
</blockquote><div><br></div><div>It would be cool, but I'm not sure if we can put the work into it to populate it or use it to do partial analyses on our end (no time). </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Cheers<br>
<br>
Simon<br>
<br>
[1] <a href="https://www.openmicroscopy.org/site/support/partner/omero.biobank/" target="_blank">https://www.openmicroscopy.org/site/support/partner/omero.biobank/</a><br>
<br>
<br>
<br>
<br>
<br>
The University of Dundee is a registered Scottish Charity, No: SC015096<br>
_______________________________________________<br>
ome-devel mailing list<br>
<a href="mailto:ome-devel@lists.openmicroscopy.org.uk">ome-devel@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel</a><br>
</blockquote></div><br></div></div></div>