[ome-devel] Problem with float point image channel min/max range

[Josh Moore]

Hi José,

On Tue, Jun 23, 2015 at 6:19 PM, José Salavert Torres <josator at ebi.ac.uk> wrote:
> Thank you very much for your detailed reply.

You're welcome.


> I will upgrade to head and test it on a local machine first.

There are a few fixes on the develop branch, but I'd suggest working
from the latest release tag in general (v5.1.2)


> We need to build it because we add a web service to the web gateway that
> allows us to decrease the jpeg quality when rendering image regions.

A webservice in the Django layer? Often it's possible to add such
extensions by having them on the PYTHONPATH and registering them via:

  bin/omero config append omero.web.apps '"<your-app>"'

More info is available under
http://www.openmicroscopy.org/site/support/omero5.1/developers/Web/CreateApp.html
if that's something we could help you with.


> Best wishes,
> José

Cheers,
~Josh.


> On 23 Jun 2015, at 16:52, Josh Moore <josh at glencoesoftware.com> wrote:
>
> On Tue, Jun 23, 2015 at 3:24 PM, José Salavert Torres <josator at ebi.ac.uk>
> wrote:
>
> Dear Josh,
>
>
> Hi José,
>
>
> We were not able to identify this failure until now. We used the pre-release
> version because we needed support for floating point images.
>
>
> Understood. Floating-point support has certainly been further improved
> in later versions. Especially
> https://github.com/openmicroscopy/openmicroscopy/pull/3835, I think,
> would be critical since this causes the exact kind of behavior you are
> describing.
>
>
> Regarding your question, the server is installed at EBI in an environment in
> which it took us more than a week to be able to set up an omero compilation
> / installation environment. So performing updates is not an easy /
> reasonable task and we continued using that version when we saw it worked
> properly.
>
>
> Upgrading between the pre-releases and the point releases like 5.1.2
> should be much easier than doing a full install.
>
>
> Please be comprehensive about our infrastructure: all the tools / libraries
> / ***ice dependencies*** and python files must be manually installed without
> root privileges in our machines, setting the path variables and python
> environment, finding the binaries and libXXX.so files etc, and the same
> stands for the build chain. Making a git pull can put us in a situation in
> which we would need to spend more days manually updating our environment.
>
>
> What version of Python and Ice are you using? Or is there another
> barrier to your using the provided OMERO.server zips? It shouldn't be
> necessary to rebuild OMERO.server for an upgrade.
>
>
> Right now a service is running with all PDBe entries in the OMERO database,
> so in order to upgrade the server we need to upgrade the database also. If
> not we will need to drop the entire database and import all the entries
> again. This takes around a week because it is a very i/o demanding task.
> Also, this is aggravated by continued failures in the import process related
> to session tokens (that force us to clean the OMERO.data directory hidden
> lock files manually and restart the server). This may also stop our release
> cycle of the service.
>
>
> Is it possible to upgrade an existing omero database to a new version schema
> without dropping the database?
>
>
> It is, yes. The version of the database you are using is printed to
> var/log/Blitz-0.log or is in the dbpatch table of the database, e.g.
>
> $ grep OMERO5 dist/var/log/Blitz-0.log
> 2015-06-22 16:05:31,347 INFO  [
> ome.services.util.DBPatchCheck] (      main) Verified database patch:
> OMERO5.1__1
>
> OR
>
> $ psql omero -c "select currentversion, currentpatch from dbpatch
> order by id desc limit 1"
> currentversion | currentpatch
> ----------------+--------------
> OMERO5.1       |            1
> (1 row)
>
> The scripts for upgrading from the pre-release versions are under the
> sql/psql directory:
> https://github.com/openmicroscopy/openmicroscopy/tree/v5.1.2/sql/psql
>
>
>
> Now that I answered your question, again, please be comprehensive. Before we
> try to update the server which may (or may not) make us spend about a week,
> please can you have a quick  look and see if the problems we are
> experiencing are still reproduced in the current version?
>
> Please, can you simply import entry EMD-2796:
>
> ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-2796/map/emd_2796.map.gz
>
> Into a running instance of OMERO and ask for:
>
> …
> /webgateway/render_image_region_quality/43450/99/0/100?c=1|0:0&region=26,29,169,169
> (service down)
> …
> /webgateway/render_image_region_quality/43450/99/0/100?c=1|-0.000475:-0.000476&region=26,29,169,169
> (too much compute time)
>
> to the web gateway and check both issues.
>
>
> Using both of these queries (as updated in your subsequent email)
> worked fine for me on http://demo.openmicroscopy.org (5.1.2)
>
>
> Sincerely, I expected a different response, I can also check the amount of
> commits between our version and the current one with a simple git command,
> but that does not solve the issue for us. If you can not confirm that the
> problem is already solved we can not risk updating at this moment.
>
>
> Sorry for the misunderstanding, but your previous email sounded quite
> like you wanted us to reproduce under m4. Before anyone on the team
> spent their time installing it, I think it was fair to ask for
> clarification.
>
> In general, we don't and can't support pre-release builds, but
> verifying against a current release (which was much easier with your
> download -- thanks!) certainly is possible.
>
> That being said, you might consider making use of demo.o.org yourself,
> grabbing the VM, or even using our docker containers for having a
> local test setup of the latest release.
>
>
> We put a lot of effort into making the OMERO service run and takes us days
> to do it.
>
>
> Let us know if there's anything we can do to reduce that burden.
>
> All the best,
> ~Josh.
>
>
> On 23 Jun 2015, at 12:48, Josh Moore <josh at glencoesoftware.com> wrote:
>
> On Tue, Jun 23, 2015 at 1:13 PM, José Salavert Torres <josator at ebi.ac.uk>
> wrote:
>
> Dear OMERO developers,
>
>
> Hi José,
>
>
> I am on commit “c6f9a94b0a16ac352e1d7287954d08c0d881eb06”.
> When asking for images through the Webserver gateway, I have experienced a
> problem that hangs the server.
>
>
> If I send an URL like:
>
> ... /webgateway/render_image_region/35203/99/0?c=1|0:0&region=29,0,169,169”
>
> with the channel min and max values equal to zero:
>
> ?c=1|0:0
>
> the server starts an intensive computation and stops returning any other
> image requests (the service is down due the overloading until you reset it).
>
> Also, if the minimum / maximum range is smaller than the precision this has
> an impact in the performance that is greater the narrower this range is. i.e
> given :
>
> min = -10
> max = 30
>
> A call with:
>
> ?c=1|20.00:22.01
>
> Will trigger an image calculation that will take a lot of time and
> resources, slowing down the server and making it unresponsive for several
> minutes.
>
> Can you reproduce these issues?
>
>
> Before we start trying to reproduce, is there a reason you're using a
> pre-release version (5.1.0-m4)? There are some 2000+ commits between
> what you're working on and the current release tag ("v5.1.2").
>
> Cheers,
> ~Josh.
>
> José
>