[ome-devel] CLI render - and a philosophic question re: rendering settings

Damir Sudar dsudar at lbl.gov
Fri Aug 11 17:15:58 BST 2017 

Hi Josh,

On 8/11/2017 8:32, Josh Moore wrote:

Happy Friday night to you!

> I'm finally gearing up to upgrade to 5.3.3 for all its great new
>> capabilities and was trying some stuff on my 5.3.3 test server. One of those
>> was the render.py script that, per below, wasn't really usable in vanilla
>> 5.2.x. Unfortunately I get the exact same exceptions with 5.3.3
> i.e. you're getting the "getWindowMin" error when running `info` and
> "no attribute '_closeRE" on `copy`?
Yes, exactly those same errors:
----
omero at quid:~$ omero render info Image:85547
Using session 19d44614-0173-4c8c-b297-03f3ba96a357 
(dsudar at localhost:4064). Idle timeout: 10 min. Current group: LaBargeLab
Traceback (most recent call last):
   File "/usr/local/bin/omero", line 125, in <module>
     rv = omero.cli.argv()
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1507, 
in argv
     cli.invoke(args[1:])
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 992, in 
invoke
     stop = self.onecmd(line, previous_args)
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1069, 
in onecmd
     self.execute(line, previous_args)
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1151, 
in execute
     args.func(args)
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 343, in info
     ro = RenderObject(img)
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 207, in __init__
     image.getChannels(noRE=True))
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 206, in <lambda>
     self.channels = map(lambda x: ChannelObject(x),
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 109, in __init__
     self.init_from_channel(channel)
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 117, in init_from_channel
     self.min = channel.getWindowMin()
   File "/home/omero/OMERO.server/lib/python/omero/gateway/__init__.py", 
line 7370, in getWindowMin
     return si.getGlobalMin().val
   File "/home/omero/OMERO.server/lib/python/omero_model_StatsInfoI.py", 
line 138, in getGlobalMin
     self.errorIfUnloaded()
   File "/home/omero/OMERO.server/lib/python/omero_model_StatsInfoI.py", 
line 39, in errorIfUnloaded
     raise _omero.UnloadedEntityException("Object unloaded:"+str(self))
omero.UnloadedEntityException: Object unloaded:object #0 
(::omero::model::StatsInfo)
{
     _id = object #1 (::omero::RLong)
     {
         _val = 210111
     }
     _details = <nil>
     _loaded = False
     _version = <nil>
     _globalMin = <nil>
     _globalMax = <nil>
}
omero at quid:~$ omero render copy Image:84367 Image:84360
Using session 19d44614-0173-4c8c-b297-03f3ba96a357 
(dsudar at localhost:4064). Idle timeout: 10 min. Current group: LaBargeLab
Traceback (most recent call last):
   File "/usr/local/bin/omero", line 125, in <module>
     rv = omero.cli.argv()
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1507, 
in argv
     cli.invoke(args[1:])
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 992, in 
invoke
     stop = self.onecmd(line, previous_args)
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1069, 
in onecmd
     self.execute(line, previous_args)
   File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1151, 
in execute
     args.func(args)
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 361, in copy
     self._copy(gateway, args.object, args.target, args.skipthumbs)
   File "/home/omero/OMERO.server/lib/python/omero/plugins/render.py", 
line 373, in _copy
     src_img._closeRE()
   File "/home/omero/OMERO.server/lib/python/omero/gateway/__init__.py", 
line 1306, in __getattr__
     % (self._obj.__class__.__name__, attr))
AttributeError: 'ImageI' object has no attribute '_closeRE'
----
>> so I guess I should wait for 5.4?
> I don't know of any of these issues having been fixed in 5.4, so not
> particularly.
>
>
> Or is the situation with 5.3 more solvable?
>
> Perhaps, but I don't think so.
That's too bad.
>> Now the philosophical question: as in IDR, we have a very large image
>> dataset that we are quantitatively analyzing and also want to show visually.
>> This is part of a large project to asses the effects of a set of perturbing
>> agents (perturbagens) on a panel cell lines, where in our case a large panel
>> of different microenvironment components and drugs are used as the
>> perturbagens. During import into OMERO each image gets its rendering
>> settings from the min/max values in the image itself so all images have
>> different settings and the unsuspecting viewer may see all kinds of
>> interesting intensity effects that are mostly fake while real effects may be
>> masked. So I'm looking for a general way to set the rendering settings
>> across the entire experiment to something that allows comparing images but
>> doesn't generate loads of black and over-exposed-looking images. I was
>> thinking that you guys encountered this same issue with many of the IDR
>> datasets so was hoping for some wise thoughts.
> We definitely do but per study, and typically we can only define that
> after visual inspection since they are external datasets. You're
> looking to set something *before import*?

No, after import makes more sense since one doesn't really know what the 
expected min and max values and desired range setting are going to be, 
just as you said. We have been playing with ideas such as:
- we could look at the min and max values per channel across a "set" of 
wells/plates (tbd what a "set" means) and set the rendering settings to 
be at 2% - 98% (or something) of those values. I'd need to write a 
script to extract such min/max values for a "set" of plates and then we 
could test what those percent values should be for a pleasing result.
- question to answer then is: what is a "set" and we're back to 
different rendering settings when you do want to compare images across 
"sets".
- one option is to let an external query/visualization tool set the 
rendering settings. The OMERO webgateway allows you to query the min/max 
(via a JSON query) and set the rendering settings via the UR. So that 
would rely on an external visualization software do the "right" thing.
Just to give you an idea of the data we're dealing with, check out our 
AWS-hosted public server at: 
https://omero.lincsclarion.org/webclient/userdata/?experimenter=52
The front-end portal with query/visualization tools is still being built.
>> Anyway, this is why I'm so interested in the render.py utility and thus my
>> hope it will make it to production-level soon.
> I don't have a timeline for you on the production status of render.py,
> but what we *would* like to do for 5.4 is make it separately
> installable, so that you can upgrade just render.py as we releases
> fixes without needing to update all of OMERO.

That sounds like a great approach.
>
> P.S. from your Jul 12th email while I was on vacation:
>
>> Meanwhile, I had kludged together a regular OMERO script (HCS_Render_Settings.py) that does a number of the things that the render.py plugin does but much much less elegantly and very slowly. I'll use that for now. That script is on: https://github.com/dsudar/OMEROscripts
> There's no inherent reason a script should be slower that the CLI. Do
> you have specifics?

I think it's mostly that my script is not at all efficiently implemented 
where it does repeated calls to objects to first get metadata (such as 
value ranges) and then again to set values and probably causes multiple 
thumbnail creation calls.

Cheers,
- Damir
>> On 7/10/2017 6:45, Josh Moore wrote:
>>>
>>>> Next I tried a few of the basic commands of the render plugin:
>>>> 1 ---
>>> ...
>>>>     File
>>>> "/home/omero_user/OMERO.server/lib/python/omero/plugins/render.py",
>>>> line 117, in init_from_channel
>>>>       self.min = channel.getWindowMin()
>>> ...
>>>> omero.UnloadedEntityException: Object unloaded:object #0
>>> Vanilla 5.2 is not loading the necessary metadata for the plugin to
>>> work. Unfortunately, solving this isn't as simple as copying a new
>>> file into your installation. In fact, I haven't yet found a smoking
>>> gun between the versions.
>>>
>>>
>>>> 2---
>>>> AttributeError: 'ImageI' object has no attribute '_closeRE'
>>>> ---
>>> Here again, changes from 5.3 are not available in your version, though
>>> at least these I know where they come from.
>>>
>>>
>>>> Rather than trying more of this, I thought to check with you first
>>>> whether
>>>> I'm missing something fundamental.
>>> 5.3 :) (or the upcoming 5.4)
>>>
>>> Fundamentally, the plugin is not production quality and has only been
>>> used in isolated situations. We certainly intend to get it into the
>>> testing pipeline for 5.4 but that's not likely to uncover what's going
>>> on with 5.2. If you or anyone on your side wants to continue digging,
>>> we're happy to evaluate your changes, but there's a question of how
>>> far you want to go in modifying your local installation.
>>>
>>> Cheers,
>>> ~Josh.
>>>
>>>
>>>
>>>> Thanks,
>>>> - Damir
>>>>
>>>> On 6/2/2017 2:35, Simon Li wrote:
>>>>
>>>> The render plugin (not tested on 5.3):
>>>>
>>>> https://github.com/openmicroscopy/openmicroscopy/blob/metadata53/components/tools/OmeroPy/src/omero/plugins/render.py
>>>> Try dropping this into OMERO.server/lib/python/omero/plugins/
>>>>
>>>> Example YAML render settings file:
>>>>
>>>> https://github.com/IDR/idr-metadata/blob/master/idr0015-UNKNOWN-taraoceans/screenA/idr0015-screenA-renderdef.yml
>>>>
>>>> omero render edit --help
>>>> omero render edit --copy Plate:1 renderdef.yml
>>>> omero render copy Image:SrcID Plate:TargetID

-- 
Damir Sudar - Affiliate Scientist
Lawrence Berkeley Natl Laboratory / MBIB
One Cyclotron Road, MS 977, Berkeley, CA 94720, USA
T: 510/486-5346 - F: 510/486-5586 - E: DSudar at lbl.gov
http://biosciences.lbl.gov/profiles/damir-sudar-2/

Visiting Scientist, Oregon Health & Science University

More information about the ome-devel mailing list