[ome-users] Problem with omero downloads from web

Thu Jan 14 13:38:48 GMT 2016

Hi Jace and Dave

Sorry for a slow response.

Thanks again for reporting download issue. Failing download is not related to the size of 1GB but Gunicorn worker timeout 30 sec. Your nginx log should show error that upstream prematurely closed connection while reading...

and gunicorn log will say: [CRITICAL] WORKER TIMEOUT (pid:2701)

You can test it by starting web with custom timeout http://docs.gunicorn.org/en/stable/settings.html#timeout
bin/omero web start --wsgi-args ' -t 5 --error-logfile=/home/omero/omero_dir/var/log/g_error.log '

By default OMERO.web runs with gunicorn 5 sync workers with single thread and default timeout 30 sec. Sync workers are preferred for applications that perform complex calculations and async for one that bound (I/O). So you basically need to decide your users will use web for.

You can resolve this problem in two ways depends on download activity and scale by choosing appropriate parameters:

1. async workers:

$ pip install (gevent|eventlet)
$ bin/omero web start —-workers INT --wsgi-args ' —-worker-class (gevent|eventlet) '

2. sync workers with multiple threads (this option is available since gunicorn 19):
(this is less fabulous as request will take up CPU core once is waiting)

$ bin/omero web start —-workers INT --wsgi-args ‘ --timeout INT --threads INT '
depends on python version you may have to pip install futures

For more details how to calculate number of workers, threads, etc please refer to the documentation http://docs.gunicorn.org/en/stable/settings.html

You can also run benchmarks tool such as weighttp or Apache ab and do e.g 1000 requests with 100 parallel requests, changing between different kind of workers and see what happens performance-wise.

We will definitely update our documentation to make it clear.

Ola
Software Engineer
Open Microscopy Environment
University of Dundee

On 14 Jan 2016, at 05:29, Jake Carroll <jake.carroll at uq.edu.au<mailto:jake.carroll at uq.edu.au>> wrote:

Unfortunately, I’ve just confirmed the same bug exists with the same configuration on a different platform. I tried on a fully patched CentOS 7.2 environment and the behaviour is identical. Files truncate and fail at 1.0x GB of download/stream over http/nginx.

Thanks.

-jc

Date: Tue, 12 Jan 2016 13:51:02 +0000
From: "Mason, David [dnmason]" <D.N.Mason at liverpool.ac.uk<mailto:D.N.Mason at liverpool.ac.uk>>
To: "ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>"
<ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Problem with omero *downloads* from web
client - nginx?
Message-ID:
<F15D297020D9DC45B5F9473909B3655747AAAE6F at CHEXMBX1.livad.liv.ac.uk<mailto:F15D297020D9DC45B5F9473909B3655747AAAE6F at chexmbx1.livad.liv.ac.uk>>
Content-Type: text/plain; charset="us-ascii"

I can confirm that I also get the same issue in our OMERO.web 5.2.0-ice35-b12 running Nginx + WSGI + gunicorn on Ubuntu 14.04 LTS. Download of a large original data file stalled at 1.01GB.

We rarely download the original data so I'm not surprised to have not noticed since upgrading to 5.2.

Best,

Dave

----------------------------------------------------------------------
Message: 1
Date: Mon, 11 Jan 2016 20:26:18 +0000
From: Jake Carroll <jake.carroll at uq.edu.au<mailto:jake.carroll at uq.edu.au>>
To: "ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>"
      <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: [ome-users] Problem with omero *downloads* from web client -
      nginx?
Message-ID: <3521A17B-7735-4497-94CE-935360303C67 at uq.edu.au<mailto:3521A17B-7735-4497-94CE-935360303C67 at uq.edu.au>>
Content-Type: text/plain; charset="utf-8"

Hi OME crew.

I?m in need of a little bit of your help.

I?ve upgraded to 5.2.1 as of a while ago and things appeared fine on the surface, but now that everyone is back to work from the Christmas break, we?ve found a problem (and we believe it is nginx + gunicorn related).

1. Large downloads simply fail or are truncated to smaller sizes. We keep seeming to hit a magical 1.01 or 1.1GB limit when trying to download large things via the OMERO.web interface. Insight works fine, as it is unrelated to nginx.

2. We?re running the config from the OMERO 5.2.1 documentation, nginx-wise. Nginx + WSGI + gunicorn atop Ubuntu 14.04 LTS. All up to date.

3. I?ve read things about gzip compression having an impact. Tried to turn that off. No difference. I?ve read proxy_max_temp_file_size might have an impact. Fiddled with that - no difference. Fiddled with timeouts on upstream server. No difference.

4. I can?t see anything useful in the error.log or access.log for nginx, nor in the Blitz.log etc.

5. When your users are downloading 30GB+ .ims files out of OMERO regularly, this could present a problem for us, as you can appreciated ;).

Any help would be appreciated - as I?m stumped on this one. This is usually just normal administration 101 ?just tune this? setting sort of thing, which doesn?t present much of a challenge - but this feels like a ghost.

Regards.

-jc

------------------------------

Message: 2
Date: Tue, 12 Jan 2016 16:17:23 +0000
From: "Aleksandra Tarkowska (Staff)" <A.Tarkowska at dundee.ac.uk<mailto:A.Tarkowska at dundee.ac.uk>>
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: Re: [ome-users] Problem with omero *downloads* from web
client - nginx?
Message-ID: <C595C76B-8E9C-4A0E-884D-92859E50090F at dundee.ac.uk<mailto:C595C76B-8E9C-4A0E-884D-92859E50090F at dundee.ac.uk>>
Content-Type: text/plain; charset="us-ascii"

Hi,

Thank you for reporting download problems. I can reproduce it locally and will let you know as soon as I find work around.

Ola
Software Engineer
Open Microscopy Environment
University of Dundee

On 12 Jan 2016, at 13:51, Mason, David [dnmason] <D.N.Mason at liverpool.ac.uk<mailto:D.N.Mason at liverpool.ac.uk><mailto:D.N.Mason at liverpool.ac.uk>> wrote: