[ome-devel] Problem with float point image channel min/max range

[Josh Moore]

On Tue, Jun 23, 2015 at 3:24 PM, José Salavert Torres <josator at ebi.ac.uk> wrote:
> Dear Josh,

Hi José,


> We were not able to identify this failure until now. We used the pre-release
> version because we needed support for floating point images.

Understood. Floating-point support has certainly been further improved
in later versions. Especially
https://github.com/openmicroscopy/openmicroscopy/pull/3835, I think,
would be critical since this causes the exact kind of behavior you are
describing.


> Regarding your question, the server is installed at EBI in an environment in
> which it took us more than a week to be able to set up an omero compilation
> / installation environment. So performing updates is not an easy /
> reasonable task and we continued using that version when we saw it worked
> properly.

Upgrading between the pre-releases and the point releases like 5.1.2
should be much easier than doing a full install.


> Please be comprehensive about our infrastructure: all the tools / libraries
> / ***ice dependencies*** and python files must be manually installed without
> root privileges in our machines, setting the path variables and python
> environment, finding the binaries and libXXX.so files etc, and the same
> stands for the build chain. Making a git pull can put us in a situation in
> which we would need to spend more days manually updating our environment.

What version of Python and Ice are you using? Or is there another
barrier to your using the provided OMERO.server zips? It shouldn't be
necessary to rebuild OMERO.server for an upgrade.


> Right now a service is running with all PDBe entries in the OMERO database,
> so in order to upgrade the server we need to upgrade the database also. If
> not we will need to drop the entire database and import all the entries
> again. This takes around a week because it is a very i/o demanding task.
> Also, this is aggravated by continued failures in the import process related
> to session tokens (that force us to clean the OMERO.data directory hidden
> lock files manually and restart the server). This may also stop our release
> cycle of the service.

> Is it possible to upgrade an existing omero database to a new version schema
> without dropping the database?

It is, yes. The version of the database you are using is printed to
var/log/Blitz-0.log or is in the dbpatch table of the database, e.g.

$ grep OMERO5 dist/var/log/Blitz-0.log
2015-06-22 16:05:31,347 INFO  [
ome.services.util.DBPatchCheck] (      main) Verified database patch:
OMERO5.1__1

OR

$ psql omero -c "select currentversion, currentpatch from dbpatch
order by id desc limit 1"
 currentversion | currentpatch
----------------+--------------
 OMERO5.1       |            1
(1 row)

The scripts for upgrading from the pre-release versions are under the
sql/psql directory:
https://github.com/openmicroscopy/openmicroscopy/tree/v5.1.2/sql/psql



> Now that I answered your question, again, please be comprehensive. Before we
> try to update the server which may (or may not) make us spend about a week,
> please can you have a quick  look and see if the problems we are
> experiencing are still reproduced in the current version?
>
> Please, can you simply import entry EMD-2796:
>
> ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-2796/map/emd_2796.map.gz
>
> Into a running instance of OMERO and ask for:
>
> …
> /webgateway/render_image_region_quality/43450/99/0/100?c=1|0:0&region=26,29,169,169
> (service down)
> …
> /webgateway/render_image_region_quality/43450/99/0/100?c=1|-0.000475:-0.000476&region=26,29,169,169
> (too much compute time)
>
> to the web gateway and check both issues.

Using both of these queries (as updated in your subsequent email)
worked fine for me on http://demo.openmicroscopy.org (5.1.2)


> Sincerely, I expected a different response, I can also check the amount of
> commits between our version and the current one with a simple git command,
> but that does not solve the issue for us. If you can not confirm that the
> problem is already solved we can not risk updating at this moment.

Sorry for the misunderstanding, but your previous email sounded quite
like you wanted us to reproduce under m4. Before anyone on the team
spent their time installing it, I think it was fair to ask for
clarification.

In general, we don't and can't support pre-release builds, but
verifying against a current release (which was much easier with your
download -- thanks!) certainly is possible.

That being said, you might consider making use of demo.o.org yourself,
grabbing the VM, or even using our docker containers for having a
local test setup of the latest release.


> We put a lot of effort into making the OMERO service run and takes us days
> to do it.

Let us know if there's anything we can do to reduce that burden.

All the best,
~Josh.


> On 23 Jun 2015, at 12:48, Josh Moore <josh at glencoesoftware.com> wrote:
>
> On Tue, Jun 23, 2015 at 1:13 PM, José Salavert Torres <josator at ebi.ac.uk>
> wrote:
>
> Dear OMERO developers,
>
>
> Hi José,
>
>
> I am on commit “c6f9a94b0a16ac352e1d7287954d08c0d881eb06”.
> When asking for images through the Webserver gateway, I have experienced a
> problem that hangs the server.
>
>
> If I send an URL like:
>
> ... /webgateway/render_image_region/35203/99/0?c=1|0:0&region=29,0,169,169”
>
> with the channel min and max values equal to zero:
>
> ?c=1|0:0
>
> the server starts an intensive computation and stops returning any other
> image requests (the service is down due the overloading until you reset it).
>
> Also, if the minimum / maximum range is smaller than the precision this has
> an impact in the performance that is greater the narrower this range is. i.e
> given :
>
> min = -10
> max = 30
>
> A call with:
>
> ?c=1|20.00:22.01
>
> Will trigger an image calculation that will take a lot of time and
> resources, slowing down the server and making it unresponsive for several
> minutes.
>
> Can you reproduce these issues?
>
>
> Before we start trying to reproduce, is there a reason you're using a
> pre-release version (5.1.0-m4)? There are some 2000+ commits between
> what you're working on and the current release tag ("v5.1.2").
>
> Cheers,
> ~Josh.
>
> José