[ome-devel] Request for feature development for FIGURE - for discussion by the community

[Joshua S Titlow]

The script works as advertised.
Thanks Will!
j


On Tue, Jun 5, 2018 at 4:30 PM, William Moore (Staff) <W.Moore at dundee.ac.uk>
wrote:

>
> Hi Josh (T),
>
>  I wrote a little script that will take the sample tab-separated-value
> text-file you gave us and use it to add
> Map Annotations and Tags in OMERO.
> https://gist.github.com/will-moore/8b6575c279ea3fdc3aff4c84e803056e
>
> If you have a Dataset with images within it that correspond to the names
> in the “image” column of your table and run
> this script with the Dataset ID, it will create a Map Annotation with all
> the row data for that Image.
>
> Links to figure etc are clickable and you can filter by all the map
> annotations in OMERO.parade, which works nicely.
>
> However, when you choose to filter by e.g. Gene, you’re not given a list
> of genes to choose from in Parade (you just have to
> type in Gene names to filter).
>
> So, I also added some lines to create a Tag from each Gene name.
> This gives you the ability to filter by Gene (Tag), which is also
> well-supported in webclient without needing Parade.
>
> I also add the URL to figure in the Image Description.
>
>
> Give it a try and let us know what works for you.
> Hopefully you can get Parade installed somewhere to try out, but even
> without it, you can do the filtering by Gene.
>
> The TSV to Map-Annotation code could be turned into a general-purpose
> OMERO script since it is quite generic.
> (needs to support CSV too first).
>
> Then other users could run this script on TSV/CSV files they’ve added to a
> Dataset (similar to populate_metadata.py)
> without needing to run script client-side.
>
>  Cheers,
>
>   Will.
>
>
>
> On 5 Jun 2018, at 11:27, William Moore (Staff) <W.Moore at dundee.ac.uk>
> wrote:
>
> Hi Ilan,
>
> Jean-Marie and I had a good discussion with Josh and MK at the OME
> meeting. I think we have a good understanding of what’s needed now.
>
> To summarise the discussion / feature-requests so far:
>  - Need to be able to organise figures in much the same way as Images:
> Tag, Project/Dataset (or Folder?) and also to filter by metadata (as in
> OMERO.parade).
>  - Various export features (e.g. export all figures with a certain Tag,
> export as png etc) but not sure how important these are if everything
> happens within OMERO (no export to Zegami)?
>
>
> In order for the webclient to organise figures, we could try to improve
> the handling of Files (File-Annotations) which are supported better in
> Insight.
> However, to do this as well as is currently supported for Images would be
> a huge amount of work. Also, we’d still miss a visual representation of
> each Figure. Browsing the figure JSON files without any thumbnail or view
> of the figure would be tricky.
>
> I believe the solution is to export figures to create Images in OMERO, so
> that every figure is also represented as an OMERO Image.
> These images can then be organised as desired into Datasets, Tagged or
> annotated with Key-Value pairs (Map Annotations) and/or assigned attributes
> in OMERO.tables.
> This means that we don’t need to organise/tag/manage the actual Figure
> files - we simply use the existing webclient functionality for managing
> Images, and each Image-of-a-figure links to the figure itself.
>
> We can already export figures to Images in OMERO, so the first steps will
> be to help you to adopt this workflow, either starting with figures where
> you’ve previously exported pngs for zegami, or with new figures.
> We can also start to design the changes that are needed to facilitate this
> workflow for other users, to make it the standard way of organising figures
> in OMERO.
>
> First Steps (starting with previously exported pngs):
>
> If you have a collection of pngs in a directory, you can import them all
> into a single Dataset (ID 123), either via command line:
> $ bin/omero import -d 123 path/to/directory/
> or via Insight.
> Then, use a Python script to parse the text file that Zegami uses and for
> each row in the table:
>  - find the imported png in the Dataset by name, and add the URL to figure
> in its description
>  - use other metadata in the text file to add Map annotations to the image
> for searching/filtering
>
> Now you should be able to use OMERO.parade to do similar searches and
> filtering as you have been doing in Zegami.
>
> First Steps (starting with figures not yet exported):
>
> If you need to batch export figures, you can use David’s script with
> “OMERO” as the export option. TODO:
>  - We need to add the Figure URL to the description of new images, so that
> we can link back to the figure from the image.
>  - We need a ‘Dataset_IDs’ parameter to save the figure to chosen
> Dataset(s) (we had this working already at the OME meeting).
>  - Then we’d need some way to add the map annotations as above. If you
> have these in a CSV file already, then simply use the same script as above.
> Or use a script that could create map annotations directly from data read
> from FlyMine?
>  - Ideally we could make this process accessible to users without needing
> a developer to run scripts. E.g. if they know the Gene name, they could run
> a server-side script to create all the map annotations from this (getting
> data from FlyMine) etc.
>
> The main remaining issue would be how to update the exported Image in
> OMERO when the figure changes. Initially you could repeat the initial
> export (add annotations etc) and simply delete the previously created Image.
>
>
> Workflow improvement:
>
> Once we are happy that the basic idea of each Figure linked to an
> Image-of-a-Figure is a valid way to organise your figures,
> we can look to improve the workflow for other users. There are several new
> features needed, but
> instead of adding to this e-mail, I’ve moved these to a design issue at
> https://github.com/openmicroscopy/design/issues/96
>
>
>  Regards,
>
>   Will.
>
>
>
> On 22 May 2018, at 21:45, Ilan Davis <ilandavis at me.com> wrote:
>
> Hi Jason
>
> thanks. I guess I will answer this a bit philosophically or strategically,
> We are developing this ourselves, but my strong feeling is that we cannot
> really do a proper job of creating very good tools in the way we are doing
> it by patching our way between two non intra-operable systems and also
> without getting into the guts of the OMERO code.
>
> I also think it is wrong to think of our workflows as specialised ones for
> a particular one off purpose. They are workflows that should be adopted by
> virtually all cell biology labs that I know of. Have you surveyed how
> people are using OMERO in non-specialist labs?
>
> The way I see it, and perhaps failing to explain it properly, is that what
> we are doing is creating a general purpose workflow that should be a
> next-generation way of dealing with data transparently within a team. The
> team and individuals can define a way to browse their data that contains
> implicit information in its organisation and metadata and collection of
> panels and choice of “typical data” that allow members of teams to
> understand each others data. It also allows users to accumulate over time a
> history of the meaning of their data, as they acquire it bit by bit. So my
> PhD students for example, essentially have their figures made for their
> thesis as they go along.
>
> I realise it maybe hard to grasp what I am talking about, as this is a bit
> of a sociological / anthropological point. It is not a revolutionary
> scientific idea. It is a workflow that I think everyone should adopt as
> standard. I suspect that eventually this will be considered mandatory for
> data transparency and integrity. Just having a collection of raw data is
> good, but not sufficient.
>
> Strategically, from the point of view of the OMERO project using a work
> flow of this sort, would make the whole project adoptable by everyone who
> does microscopy rather than just the more advanced users with particularly
> skilled staff.
>
> I am convinced that the OMERO team can create a better work flow than we
> can for a base level method of making all data that everyone acquires (not
> only high content screens) transparent and understandable by anyone.
>
> I will respond to Will’s helpful and useful comments separately, as they
> are much more specific.
>
> Regards
> ilan
>
> ps As steve and I will not be there, perhaps you (and Will and Josh M) can
> schedule time with Josh and MK to go through the wokflow, the scripts (that
> Steve wrote and Josh modified) as well as see the cloud version of Zegami
> in operation. I think it is easier to understand my point above by just
> seeing examples.
>
>
> ____________________________________________________________
> ________________
> *Prof. Ilan Davis   ilan.davis at bioch.ox.ac.uk <ilan.davis at bioch.ox.ac.uk>
>   http://www.ilandavis.com <http://www.ilandavis.com/>*
> *https://twitter.com/ilandavis <https://twitter.com/ilandavis>*
> Department of Biochemistry, The University of Oxford, South Parks
> Road, OXFORD OX1 3QU, UK
> Direct Office No: (44) (0)1865 613265  Office Fax: (44) (0)1865 613340
> Lab manager: Dr Darragh Ennis darragh.ennis at bioch.ox.ac.uk (44) (0)1865
> 613271 / 613272
> PA:  jolanta.parkinson at bioch.ox.ac.uk  (44) (0)1865 613218
> ____________________________________________________________
> ________________
>
>
>
>
>
>
>
>
>
> On 22 May 2018, at 19:30, Jason Swedlow (Staff) <j.r.swedlow at dundee.ac.uk>
> wrote:
>
> Hi Ilan
>
> My €$£0.02.
>
> We really appreciate all these comments.  As you will have seen, we just
> released OMERO.figure 4.0, where we add support for “big” images (really
> large X-Y planes), which has been requested over and over again and
> required a lot of work to get right (or at least roughly so).
>
> The broader point of putting Figures under management in a serious way is
> a very important point. Your lab clearly has some very interesting
> workflows, and I look forward to hearing more about them at the upcoming
> Users Meeting. The whole point of an open project is people take technology
> in all sorts of ways that we won’t have thought of.  As Will stated, a few
> of these are on the list, and several others are new ideas—so thank you!!!
>
> We’ll discuss at the upcoming Users meeting, take in more feedback, and
> come up with a plan for taking these forward.
>
> Huge thanks again.
>
> Cheers,
>
> Jason
>
>
> *From: *William Moore <W.Moore at dundee.ac.uk>
> *Date: *Tuesday, 22 May 2018 at 15:41
> *To: *OME Development <ome-devel at lists.openmicroscopy.org.uk>
> *Cc: *Jason Swedlow <j.r.swedlow at dundee.ac.uk>, Joshua Titlow <
> joshua.titlow at bioch.ox.ac.uk>, mkaythomp_gmail_com <mkaythomp at gmail.com>,
> Darragh Ennis <darragh.ennis at bioch.ox.ac.uk>, Richard Parton <
> richard.parton at bioch.ox.ac.uk>, David Susano Pinto <
> david.pinto at bioch.ox.ac.uk>
> *Subject: *Re: [ome-devel] Request for feature development for FIGURE -
> for discussion by the community
>
> Hi Illan,
>
>  Thanks for the all the feedback.
> It’s great to hear that OMERO.figure is being so useful for you and to get
> a clear idea of what improvements are needed.
>
> I’ll respond to the various requests below in-line...
>
>
> On 16 May 2018, at 16:30, Ilan Davis <ilan.davis at bioch.ox.ac.uk> wrote:
>
> Dear OME developer community
>
> We love OMERO-figure. It has become a central way we use OMERO and we
> believe it is an under appreciated tool that everyone should use for all
> their day to day imaging. We would like to see the OMERO team embracing
> this and putting a small amount of effort in getting it even better. Please
> please please put these request high up on a priority list…
>
> *Motivation*
> Figure was developed to make it possible to make figures for papers
> directly from OMERO and bypassing illustrator and the like.
> However, for us it has enabled an unintended additional use that we
> believe is more significant. It allows every time a user has an imaging
> session on a microscope to summarise the typical result of that session. It
> also allows a systematic record of a medium scale screen, again summarising
> how the user thinks the data should be viewed. Viewing raw data from a
> screen is rather difficult in comparison. We have been using Zegami, as a
> visualisation tool that can organise and slice and dice the figures at
> scale using rich n-dimensional meta-data of our choice including
> bioinformatics.
>
> The following are specific requests framed in general terms and have
> mostly been posted previously over an extended period by various members of
> my group and Micron in Oxford.  David Pinto, Josh Titlow, MK Thompson will
> be at the next OMERO meeting in Dundee and can discuss these points in more
> detail.
>
> *Development requests*
>
> 1) Make some small improvements to Figure so it is sufficient for making
> figures for papers (other than drawing diagrams and plots). e.g. scalable
> text and better text labels. Does not have to be extensive and complex.
> Just one labelling approach that is good.
>
>
> We have an existing feature request for “Stand-alone labels” (labels that
> can be placed anywhere on the page) https://trello.com/c/
> wtu7IVuN/10-stand-alone-labels.
> Does this cover your needs or are there other improvements to the existing
> panel labels that you’d like to see?
> You can set font-sizes on labels already. Does “scalable text” include
> some other functionality or more flexible sizes?
>
>
> 2) Create a hierarchy of folders housing figures so that figures can be
> organised in a sensible way over time, at scale.
>
>
> The folders feature request is listed at https://trello.com/c/
> LwAVYUjm/165-save-figures-in-folders with some discussion as to options
> and timescale
> (which is currently limited by the OMERO model for Folders - so this needs
> to change first in OMERO).
>
>
> 3) Use some kind of tagging system, similar to Auto-tag (or repurpose the
> Auto-tag code) - extracting tags from the folder and figure names.
>
>
> A tagging system would consist of 2 main features.
> - ‘Add Tags’ dialog/panel - similar to that in webclient, and/or extend
> Auto-Tag (if users have lots of tokens in their figure names).
> - Browse Tag hierarchy from within the figure app.
> This could be a fair bit of work to do this properly (e.g. webclient Tag
> dialog is quite complex, so is webtagging). If we had “folders for
> figures”, would you still want Tagging for figures? Do you need both?
> NB: currently webclient (or Tag-searcher) will not display Tagged
> File-Annotations (figures).
> Card created at https://trello.com/c/X2lvut2c/192-add-tags-to-
> figures-browse-by-tag
>
>
> 4) Allow batch export of figures from pull down menus as well as CLI. At
> the moment we are pulling the figures from json files using our own bespoke
> python scripts.
>
>
> Batch export of Figures was previously being discussed at
> https://github.com/ome/omero-figure/issues/151 (Josh Titlow).
> However, one of the main issues has been identified by David Pinto
> https://github.com/ome/omero-figure/issues/289 which is the fact that the
> export creates a File Annotation on OMERO.
>
> We really need to split the Figure Export functionality out of the
> Figure_To_Pdf.py script so that we can generate figures directly from
> Python without requiring the scripting service. Some of that work was
> started at https://github.com/ome/omero-figure/pull/240 to allow the web
> app to generate figures without the Scripting service. I abandoned this
> when it became clear that this wouldn’t work from the web, but it would
> still be useful in your case (and others) to allow batch exports etc.
>
>
> 5) Allow more format of figure export especially .png
>
>
> PNG export should be quite straightforward to implement. Created card at
> https://trello.com/c/N0PQjMKJ/189-export-as-png
>
>
> 6) Ideally export metadata with the figures including the tags and
> automate publishing of figure collections through Zegami with metadata (via
> .png and csv files).
>
>
> Is there particular metadata other than Tags you’d like to export?
> Exporting an extra csv file would require export of a zip instead of
> PDF/TIFF, which would complicate
> the workflow a bit and we’d probably want to make it optional in that case.
> It may be that the best option here is to create a custom Python script
> that exports all the metadata you want in a format of your choice.
> This gives you the most flexibility and can be updated according to your
> needs without the UI and workflow changes for other users.
>
>
>  Thanks again for your input.
>
>  I look forward to more discussions at the OME meeting next week,
>
>  Regards,
>
>
>   Will.
>
>
> PS. Another release of OMERO.figure just went out today: http://www.
> openmicroscopy.org/2018/05/22/figure-4-0-0.html
>
>
>
>
> I welcome comments, clarifications, questions or counterpoints.
>
> with best regards
> Ilan
> *For transparency:*
> *I am on the SAB of Zegami and on the SAB of OMERO-IDR*
>
> p.s.If you would like to know more about Zegami, please look up.
> https://zegami.com/
>
>
>
> ____________________________________________________________
> ________________
> *Prof. Ilan Davis   **ilan.davis at bioch.ox.ac.uk*
> <ilan.davis at bioch.ox.ac.uk>   *http://www.ilandavis.com*
> <http://www.ilandavis.com/>
> *https://twitter.com/ilandavis* <https://twitter.com/ilandavis>
> Department of Biochemistry, The University of Oxford, South Parks
> Road, OXFORD OX1 3QU, UK
> Direct Office No: (44) (0)1865 613265  Office Fax: (44) (0)1865 613340
> Lab manager: Dr Darragh Ennis darragh.ennis at bioch.ox.ac.uk (44) (0)1865
> 613271 / 613272
> PA:  jolanta.parkinson at bioch.ox.ac.uk  (44) (0)1865 613218
> ____________________________________________________________
> ________________
>
>
>
>
>
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
> _______________________________________________
> ome-devel mailing list
> ome-devel at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-devel/attachments/20180606/8f8b11de/attachment.html>