[ome-devel] Request for feature development for FIGURE - for discussion by the community
William Moore (Staff)
W.Moore at dundee.ac.uk
Thu Jun 7 13:52:02 BST 2018
Hi Josh,
Great. So how well does this set-up work for you to organise figures (images-of-figures) in the webclient?
I guess you may need to try Parade before you can test the full filtering options.
Cheers,
Will.
On 6 Jun 2018, at 15:32, Joshua S Titlow <joshua.titlow at bioch.ox.ac.uk<mailto:joshua.titlow at bioch.ox.ac.uk>> wrote:
The script works as advertised.
Thanks Will!
j
On Tue, Jun 5, 2018 at 4:30 PM, William Moore (Staff) <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>> wrote:
Hi Josh (T),
I wrote a little script that will take the sample tab-separated-value text-file you gave us and use it to add
Map Annotations and Tags in OMERO.
https://gist.github.com/will-moore/8b6575c279ea3fdc3aff4c84e803056e
If you have a Dataset with images within it that correspond to the names in the “image” column of your table and run
this script with the Dataset ID, it will create a Map Annotation with all the row data for that Image.
Links to figure etc are clickable and you can filter by all the map annotations in OMERO.parade, which works nicely.
However, when you choose to filter by e.g. Gene, you’re not given a list of genes to choose from in Parade (you just have to
type in Gene names to filter).
So, I also added some lines to create a Tag from each Gene name.
This gives you the ability to filter by Gene (Tag), which is also well-supported in webclient without needing Parade.
I also add the URL to figure in the Image Description.
Give it a try and let us know what works for you.
Hopefully you can get Parade installed somewhere to try out, but even without it, you can do the filtering by Gene.
The TSV to Map-Annotation code could be turned into a general-purpose OMERO script since it is quite generic.
(needs to support CSV too first).
Then other users could run this script on TSV/CSV files they’ve added to a Dataset (similar to populate_metadata.py)
without needing to run script client-side.
Cheers,
Will.
On 5 Jun 2018, at 11:27, William Moore (Staff) <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>> wrote:
Hi Ilan,
Jean-Marie and I had a good discussion with Josh and MK at the OME meeting. I think we have a good understanding of what’s needed now.
To summarise the discussion / feature-requests so far:
- Need to be able to organise figures in much the same way as Images: Tag, Project/Dataset (or Folder?) and also to filter by metadata (as in OMERO.parade).
- Various export features (e.g. export all figures with a certain Tag, export as png etc) but not sure how important these are if everything happens within OMERO (no export to Zegami)?
In order for the webclient to organise figures, we could try to improve the handling of Files (File-Annotations) which are supported better in Insight.
However, to do this as well as is currently supported for Images would be a huge amount of work. Also, we’d still miss a visual representation of each Figure. Browsing the figure JSON files without any thumbnail or view of the figure would be tricky.
I believe the solution is to export figures to create Images in OMERO, so that every figure is also represented as an OMERO Image.
These images can then be organised as desired into Datasets, Tagged or annotated with Key-Value pairs (Map Annotations) and/or assigned attributes in OMERO.tables.
This means that we don’t need to organise/tag/manage the actual Figure files - we simply use the existing webclient functionality for managing Images, and each Image-of-a-figure links to the figure itself.
We can already export figures to Images in OMERO, so the first steps will be to help you to adopt this workflow, either starting with figures where you’ve previously exported pngs for zegami, or with new figures.
We can also start to design the changes that are needed to facilitate this workflow for other users, to make it the standard way of organising figures in OMERO.
First Steps (starting with previously exported pngs):
If you have a collection of pngs in a directory, you can import them all into a single Dataset (ID 123), either via command line:
$ bin/omero import -d 123 path/to/directory/
or via Insight.
Then, use a Python script to parse the text file that Zegami uses and for each row in the table:
- find the imported png in the Dataset by name, and add the URL to figure in its description
- use other metadata in the text file to add Map annotations to the image for searching/filtering
Now you should be able to use OMERO.parade to do similar searches and filtering as you have been doing in Zegami.
First Steps (starting with figures not yet exported):
If you need to batch export figures, you can use David’s script with “OMERO” as the export option. TODO:
- We need to add the Figure URL to the description of new images, so that we can link back to the figure from the image.
- We need a ‘Dataset_IDs’ parameter to save the figure to chosen Dataset(s) (we had this working already at the OME meeting).
- Then we’d need some way to add the map annotations as above. If you have these in a CSV file already, then simply use the same script as above. Or use a script that could create map annotations directly from data read from FlyMine?
- Ideally we could make this process accessible to users without needing a developer to run scripts. E.g. if they know the Gene name, they could run a server-side script to create all the map annotations from this (getting data from FlyMine) etc.
The main remaining issue would be how to update the exported Image in OMERO when the figure changes. Initially you could repeat the initial export (add annotations etc) and simply delete the previously created Image.
Workflow improvement:
Once we are happy that the basic idea of each Figure linked to an Image-of-a-Figure is a valid way to organise your figures,
we can look to improve the workflow for other users. There are several new features needed, but
instead of adding to this e-mail, I’ve moved these to a design issue at https://github.com/openmicroscopy/design/issues/96
Regards,
Will.
On 22 May 2018, at 21:45, Ilan Davis <ilandavis at me.com<mailto:ilandavis at me.com>> wrote:
Hi Jason
thanks. I guess I will answer this a bit philosophically or strategically,
We are developing this ourselves, but my strong feeling is that we cannot really do a proper job of creating very good tools in the way we are doing it by patching our way between two non intra-operable systems and also without getting into the guts of the OMERO code.
I also think it is wrong to think of our workflows as specialised ones for a particular one off purpose. They are workflows that should be adopted by virtually all cell biology labs that I know of. Have you surveyed how people are using OMERO in non-specialist labs?
The way I see it, and perhaps failing to explain it properly, is that what we are doing is creating a general purpose workflow that should be a next-generation way of dealing with data transparently within a team. The team and individuals can define a way to browse their data that contains implicit information in its organisation and metadata and collection of panels and choice of “typical data” that allow members of teams to understand each others data. It also allows users to accumulate over time a history of the meaning of their data, as they acquire it bit by bit. So my PhD students for example, essentially have their figures made for their thesis as they go along.
I realise it maybe hard to grasp what I am talking about, as this is a bit of a sociological / anthropological point. It is not a revolutionary scientific idea. It is a workflow that I think everyone should adopt as standard. I suspect that eventually this will be considered mandatory for data transparency and integrity. Just having a collection of raw data is good, but not sufficient.
Strategically, from the point of view of the OMERO project using a work flow of this sort, would make the whole project adoptable by everyone who does microscopy rather than just the more advanced users with particularly skilled staff.
I am convinced that the OMERO team can create a better work flow than we can for a base level method of making all data that everyone acquires (not only high content screens) transparent and understandable by anyone.
I will respond to Will’s helpful and useful comments separately, as they are much more specific.
Regards
ilan
ps As steve and I will not be there, perhaps you (and Will and Josh M) can schedule time with Josh and MK to go through the wokflow, the scripts (that Steve wrote and Josh modified) as well as see the cloud version of Zegami in operation. I think it is easier to understand my point above by just seeing examples.
____________________________________________________________________________
Prof. Ilan Davis ilan.davis at bioch.ox.ac.uk<mailto:ilan.davis at bioch.ox.ac.uk> http://www.ilandavis.com<http://www.ilandavis.com/>
https://twitter.com/ilandavis
Department of Biochemistry, The University of Oxford, South Parks Road, OXFORD OX1 3QU, UK
Direct Office No: (44) (0)1865 613265 Office Fax: (44) (0)1865 613340
Lab manager: Dr Darragh Ennis darragh.ennis at bioch.ox.ac.uk<mailto:darragh.ennis at bioch.ox.ac.uk> (44) (0)1865 613271 / 613272
PA: jolanta.parkinson at bioch.ox.ac.uk<mailto:jolanta.parkinson at bioch.ox.ac.uk> (44) (0)1865 613218
____________________________________________________________________________
On 22 May 2018, at 19:30, Jason Swedlow (Staff) <j.r.swedlow at dundee.ac.uk<mailto:j.r.swedlow at dundee.ac.uk>> wrote:
Hi Ilan
My €$£0.02.
We really appreciate all these comments. As you will have seen, we just released OMERO.figure 4.0, where we add support for “big” images (really large X-Y planes), which has been requested over and over again and required a lot of work to get right (or at least roughly so).
The broader point of putting Figures under management in a serious way is a very important point. Your lab clearly has some very interesting workflows, and I look forward to hearing more about them at the upcoming Users Meeting. The whole point of an open project is people take technology in all sorts of ways that we won’t have thought of. As Will stated, a few of these are on the list, and several others are new ideas—so thank you!!!
We’ll discuss at the upcoming Users meeting, take in more feedback, and come up with a plan for taking these forward.
Huge thanks again.
Cheers,
Jason
From: William Moore <W.Moore at dundee.ac.uk<mailto:W.Moore at dundee.ac.uk>>
Date: Tuesday, 22 May 2018 at 15:41
To: OME Development <ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>>
Cc: Jason Swedlow <j.r.swedlow at dundee.ac.uk<mailto:j.r.swedlow at dundee.ac.uk>>, Joshua Titlow <joshua.titlow at bioch.ox.ac.uk<mailto:joshua.titlow at bioch.ox.ac.uk>>, mkaythomp_gmail_com <mkaythomp at gmail.com<mailto:mkaythomp at gmail.com>>, Darragh Ennis <darragh.ennis at bioch.ox.ac.uk<mailto:darragh.ennis at bioch.ox.ac.uk>>, Richard Parton <richard.parton at bioch.ox.ac.uk<mailto:richard.parton at bioch.ox.ac.uk>>, David Susano Pinto <david.pinto at bioch.ox.ac.uk<mailto:david.pinto at bioch.ox.ac.uk>>
Subject: Re: [ome-devel] Request for feature development for FIGURE - for discussion by the community
Hi Illan,
Thanks for the all the feedback.
It’s great to hear that OMERO.figure is being so useful for you and to get a clear idea of what improvements are needed.
I’ll respond to the various requests below in-line...
On 16 May 2018, at 16:30, Ilan Davis <ilan.davis at bioch.ox.ac.uk<mailto:ilan.davis at bioch.ox.ac.uk>> wrote:
Dear OME developer community
We love OMERO-figure. It has become a central way we use OMERO and we believe it is an under appreciated tool that everyone should use for all their day to day imaging. We would like to see the OMERO team embracing this and putting a small amount of effort in getting it even better. Please please please put these request high up on a priority list…
Motivation
Figure was developed to make it possible to make figures for papers directly from OMERO and bypassing illustrator and the like.
However, for us it has enabled an unintended additional use that we believe is more significant. It allows every time a user has an imaging session on a microscope to summarise the typical result of that session. It also allows a systematic record of a medium scale screen, again summarising how the user thinks the data should be viewed. Viewing raw data from a screen is rather difficult in comparison. We have been using Zegami, as a visualisation tool that can organise and slice and dice the figures at scale using rich n-dimensional meta-data of our choice including bioinformatics.
The following are specific requests framed in general terms and have mostly been posted previously over an extended period by various members of my group and Micron in Oxford. David Pinto, Josh Titlow, MK Thompson will be at the next OMERO meeting in Dundee and can discuss these points in more detail.
Development requests
1) Make some small improvements to Figure so it is sufficient for making figures for papers (other than drawing diagrams and plots). e.g. scalable text and better text labels. Does not have to be extensive and complex. Just one labelling approach that is good.
We have an existing feature request for “Stand-alone labels” (labels that can be placed anywhere on the page) https://trello.com/c/wtu7IVuN/10-stand-alone-labels.
Does this cover your needs or are there other improvements to the existing panel labels that you’d like to see?
You can set font-sizes on labels already. Does “scalable text” include some other functionality or more flexible sizes?
2) Create a hierarchy of folders housing figures so that figures can be organised in a sensible way over time, at scale.
The folders feature request is listed at https://trello.com/c/LwAVYUjm/165-save-figures-in-folders with some discussion as to options and timescale
(which is currently limited by the OMERO model for Folders - so this needs to change first in OMERO).
3) Use some kind of tagging system, similar to Auto-tag (or repurpose the Auto-tag code) - extracting tags from the folder and figure names.
A tagging system would consist of 2 main features.
- ‘Add Tags’ dialog/panel - similar to that in webclient, and/or extend Auto-Tag (if users have lots of tokens in their figure names).
- Browse Tag hierarchy from within the figure app.
This could be a fair bit of work to do this properly (e.g. webclient Tag dialog is quite complex, so is webtagging). If we had “folders for figures”, would you still want Tagging for figures? Do you need both?
NB: currently webclient (or Tag-searcher) will not display Tagged File-Annotations (figures).
Card created at https://trello.com/c/X2lvut2c/192-add-tags-to-figures-browse-by-tag
4) Allow batch export of figures from pull down menus as well as CLI. At the moment we are pulling the figures from json files using our own bespoke python scripts.
Batch export of Figures was previously being discussed at https://github.com/ome/omero-figure/issues/151 (Josh Titlow).
However, one of the main issues has been identified by David Pinto https://github.com/ome/omero-figure/issues/289 which is the fact that the export creates a File Annotation on OMERO.
We really need to split the Figure Export functionality out of the Figure_To_Pdf.py script so that we can generate figures directly from Python without requiring the scripting service. Some of that work was started at https://github.com/ome/omero-figure/pull/240 to allow the web app to generate figures without the Scripting service. I abandoned this when it became clear that this wouldn’t work from the web, but it would still be useful in your case (and others) to allow batch exports etc.
5) Allow more format of figure export especially .png
PNG export should be quite straightforward to implement. Created card at https://trello.com/c/N0PQjMKJ/189-export-as-png
6) Ideally export metadata with the figures including the tags and automate publishing of figure collections through Zegami with metadata (via .png and csv files).
Is there particular metadata other than Tags you’d like to export? Exporting an extra csv file would require export of a zip instead of PDF/TIFF, which would complicate
the workflow a bit and we’d probably want to make it optional in that case.
It may be that the best option here is to create a custom Python script that exports all the metadata you want in a format of your choice.
This gives you the most flexibility and can be updated according to your needs without the UI and workflow changes for other users.
Thanks again for your input.
I look forward to more discussions at the OME meeting next week,
Regards,
Will.
PS. Another release of OMERO.figure just went out today: http://www.openmicroscopy.org/2018/05/22/figure-4-0-0.html
I welcome comments, clarifications, questions or counterpoints.
with best regards
Ilan
For transparency:
I am on the SAB of Zegami and on the SAB of OMERO-IDR
p.s.If you would like to know more about Zegami, please look up. https://zegami.com/
____________________________________________________________________________
Prof. Ilan Davis ilan.davis at bioch.ox.ac.uk<mailto:ilan.davis at bioch.ox.ac.uk> http://www.ilandavis.com<http://www.ilandavis.com/>
https://twitter.com/ilandavis
Department of Biochemistry, The University of Oxford, South Parks Road, OXFORD OX1 3QU, UK
Direct Office No: (44) (0)1865 613265 Office Fax: (44) (0)1865 613340
Lab manager: Dr Darragh Ennis darragh.ennis at bioch.ox.ac.uk<mailto:darragh.ennis at bioch.ox.ac.uk> (44) (0)1865 613271 / 613272
PA: jolanta.parkinson at bioch.ox.ac.uk<mailto:jolanta.parkinson at bioch.ox.ac.uk> (44) (0)1865 613218
____________________________________________________________________________
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096 _______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20180607/da57d57c/attachment.html>
More information about the ome-devel
mailing list