<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Jace and Dave
<div class=""><br class="">
</div>
<div class="">Sorry for a slow response.</div>
<div class=""><br class="">
</div>
<div class="">Thanks again for reporting download issue. Failing download is not related to the size of 1GB but Gunicorn worker timeout 30 sec. Your nginx log should show error that upstream prematurely closed connection while reading...</div>
<div class=""><code class=""><br class="">
</code></div>
<div class="">and gunicorn log will say: <code class="">[CRITICAL] WORKER TIMEOUT (pid:2701)</code> </div>
<div class="">
<div class=""><br class="">
</div>
<div class="">You can test it by starting web with custom timeout <a href="http://docs.gunicorn.org/en/stable/settings.html#timeout" class="">http://docs.gunicorn.org/en/stable/settings.html#timeout</a></div>
<div class="">
<div class="">
<div class=""><font face="monospace" class="">bin/omero web start --wsgi-args ' -t 5 --error-logfile=/home/omero/omero_dir/var/log/g_error.log </font><span style="font-family: monospace;" class="">'</span></div>
<div class=""></div>
</div>
<div class=""><br class="">
</div>
<p class="">By default OMERO.web runs with gunicorn 5 sync workers with single thread and default timeout 30 sec. Sync workers are preferred for applications that perform complex calculations and async for one that bound (I/O). So you basically need to decide
your users will use web for.</p>
</div>
<div class=""><br class="">
</div>
<div class="">You can resolve this problem in two ways depends on download activity and scale by choosing appropriate parameters:</div>
<div class=""><br class="">
</div>
<div class="">1. async workers:</div>
<div class=""><span style="font-family: monospace;" class=""><br class="">
</span></div>
<div class=""><span style="font-family: monospace;" class="">$ pip install (gevent|eventlet)</span></div>
<div apple-content-edited="true" class="">
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class=""><font face="monospace" class="">$ bin/omero web start —-workers INT </font><span style="font-family: monospace;" class="">--wsgi-args ' </span><span style="font-family: monospace;" class="">—-worker-class (gevent|eventlet)</span><span style="font-family: monospace;" class=""> </span><span style="font-family: monospace;" class="">'</span></div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class="">2. sync workers with multiple threads (this option is available since gunicorn 19):</div>
<div class="">(this is less fabulous as request will take up CPU core once is waiting)</div>
<div class=""><font face="monospace" class=""><br class="">
</font></div>
<div class=""><font face="monospace" class="">$ bin/omero web start —-workers INT --wsgi-args ‘ --timeout INT --threads </font><span style="font-family: monospace;" class="">INT </span><span style="font-family: monospace;" class="">'</span></div>
<div class="">depends on python version you may have to <span style="font-family: monospace;" class="">pip install futures</span></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
For more details how to calculate number of workers, threads, etc please refer to the documentation
<a href="http://docs.gunicorn.org/en/stable/settings.html" class="">http://docs.gunicorn.org/en/stable/settings.html</a></div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
You can also run benchmarks tool such as weighttp or Apache ab and do e.g 1000 requests with 100 parallel requests, changing between different kind of workers and see what happens performance-wise.</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div class=""><br class="">
</div>
<div class="">We will definitely update our documentation to make it clear.</div>
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
Ola</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Software Engineer</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Open Microscopy Environment</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
University of Dundee</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 14 Jan 2016, at 05:29, Jake Carroll <<a href="mailto:jake.carroll@uq.edu.au" class="">jake.carroll@uq.edu.au</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Unfortunately, I’ve just confirmed the same bug exists with the same configuration on a different platform. I tried on a fully patched CentOS 7.2 environment and the behaviour is identical. Files truncate and fail at 1.0x GB of download/stream
over http/nginx.<br class="">
<br class="">
Thanks.<br class="">
<br class="">
-jc<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">Date: Tue, 12 Jan 2016 13:51:02 +0000<br class="">
From: "Mason, David [dnmason]" <<a href="mailto:D.N.Mason@liverpool.ac.uk" class="">D.N.Mason@liverpool.ac.uk</a>><br class="">
To: "<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a>"<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a>><br class="">
Subject: Re: [ome-users] Problem with omero *downloads* from web<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>client - nginx?<br class="">
Message-ID:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><<a href="mailto:F15D297020D9DC45B5F9473909B3655747AAAE6F@chexmbx1.livad.liv.ac.uk" class="">F15D297020D9DC45B5F9473909B3655747AAAE6F@CHEXMBX1.livad.liv.ac.uk</a>><br class="">
Content-Type: text/plain; charset="us-ascii"<br class="">
<br class="">
I can confirm that I also get the same issue in our OMERO.web 5.2.0-ice35-b12 running Nginx + WSGI + gunicorn on Ubuntu 14.04 LTS. Download of a large original data file stalled at 1.01GB.<br class="">
<br class="">
We rarely download the original data so I'm not surprised to have not noticed since upgrading to 5.2.<br class="">
<br class="">
Best,<br class="">
<br class="">
Dave<br class="">
<br class="">
----------------------------------------------------------------------<br class="">
Message: 1<br class="">
Date: Mon, 11 Jan 2016 20:26:18 +0000<br class="">
From: Jake Carroll <<a href="mailto:jake.carroll@uq.edu.au" class="">jake.carroll@uq.edu.au</a>><br class="">
To: "<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a>"<br class="">
<<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a>><br class="">
Subject: [ome-users] Problem with omero *downloads* from web client -<br class="">
nginx?<br class="">
Message-ID: <<a href="mailto:3521A17B-7735-4497-94CE-935360303C67@uq.edu.au" class="">3521A17B-7735-4497-94CE-935360303C67@uq.edu.au</a>><br class="">
Content-Type: text/plain; charset="utf-8"<br class="">
<br class="">
Hi OME crew.<br class="">
<br class="">
I?m in need of a little bit of your help.<br class="">
<br class="">
I?ve upgraded to 5.2.1 as of a while ago and things appeared fine on the surface, but now that everyone is back to work from the Christmas break, we?ve found a problem (and we believe it is nginx + gunicorn related).<br class="">
<br class="">
1. Large downloads simply fail or are truncated to smaller sizes. We keep seeming to hit a magical 1.01 or 1.1GB limit when trying to download large things via the OMERO.web interface. Insight works fine, as it is unrelated to nginx.<br class="">
<br class="">
2. We?re running the config from the OMERO 5.2.1 documentation, nginx-wise. Nginx + WSGI + gunicorn atop Ubuntu 14.04 LTS. All up to date.<br class="">
<br class="">
3. I?ve read things about gzip compression having an impact. Tried to turn that off. No difference. I?ve read proxy_max_temp_file_size might have an impact. Fiddled with that - no difference. Fiddled with timeouts on upstream server. No difference.<br class="">
<br class="">
4. I can?t see anything useful in the error.log or access.log for nginx, nor in the Blitz.log etc.<br class="">
<br class="">
5. When your users are downloading 30GB+ .ims files out of OMERO regularly, this could present a problem for us, as you can appreciated ;).<br class="">
<br class="">
Any help would be appreciated - as I?m stumped on this one. This is usually just normal administration 101 ?just tune this? setting sort of thing, which doesn?t present much of a challenge - but this feels like a ghost.<br class="">
<br class="">
Regards.<br class="">
<br class="">
-jc<br class="">
<br class="">
------------------------------<br class="">
<br class="">
Message: 2<br class="">
Date: Tue, 12 Jan 2016 16:17:23 +0000<br class="">
From: "Aleksandra Tarkowska (Staff)" <<a href="mailto:A.Tarkowska@dundee.ac.uk" class="">A.Tarkowska@dundee.ac.uk</a>><br class="">
To: OME User Support List <<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a>><br class="">
Subject: Re: [ome-users] Problem with omero *downloads* from web<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>client - nginx?<br class="">
Message-ID: <<a href="mailto:C595C76B-8E9C-4A0E-884D-92859E50090F@dundee.ac.uk" class="">C595C76B-8E9C-4A0E-884D-92859E50090F@dundee.ac.uk</a>><br class="">
Content-Type: text/plain; charset="us-ascii"<br class="">
<br class="">
Hi,<br class="">
<br class="">
Thank you for reporting download problems. I can reproduce it locally and will let you know as soon as I find work around.<br class="">
<br class="">
Ola<br class="">
Software Engineer<br class="">
Open Microscopy Environment<br class="">
University of Dundee<br class="">
<br class="">
On 12 Jan 2016, at 13:51, Mason, David [dnmason] <<a href="mailto:D.N.Mason@liverpool.ac.uk" class="">D.N.Mason@liverpool.ac.uk</a><<a href="mailto:D.N.Mason@liverpool.ac.uk" class="">mailto:D.N.Mason@liverpool.ac.uk</a>>> wrote:<br class="">
<br class="">
I can confirm that I also get the same issue in our OMERO.web 5.2.0-ice35-b12 running Nginx + WSGI + gunicorn on Ubuntu 14.04 LTS. Download of a large original data file stalled at 1.01GB.<br class="">
<br class="">
We rarely download the original data so I'm not surprised to have not noticed since upgrading to 5.2.<br class="">
<br class="">
Best,<br class="">
<br class="">
Dave<br class="">
<br class="">
----------------------------------------------------------------------<br class="">
Message: 1<br class="">
Date: Mon, 11 Jan 2016 20:26:18 +0000<br class="">
From: Jake Carroll <<a href="mailto:jake.carroll@uq.edu.au" class="">jake.carroll@uq.edu.au</a><<a href="mailto:jake.carroll@uq.edu.au" class="">mailto:jake.carroll@uq.edu.au</a>>><br class="">
To: "<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">mailto:ome-users@lists.openmicroscopy.org.uk</a>>"<br class="">
<<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">mailto:ome-users@lists.openmicroscopy.org.uk</a>>><br class="">
Subject: [ome-users] Problem with omero *downloads* from web client -<br class="">
nginx?<br class="">
Message-ID: <<a href="mailto:3521A17B-7735-4497-94CE-935360303C67@uq.edu.au" class="">3521A17B-7735-4497-94CE-935360303C67@uq.edu.au</a><<a href="mailto:3521A17B-7735-4497-94CE-935360303C67@uq.edu.au" class="">mailto:3521A17B-7735-4497-94CE-935360303C67@uq.edu.au</a>>><br class="">
Content-Type: text/plain; charset="utf-8"<br class="">
<br class="">
Hi OME crew.<br class="">
<br class="">
I?m in need of a little bit of your help.<br class="">
<br class="">
I?ve upgraded to 5.2.1 as of a while ago and things appeared fine on the surface, but now that everyone is back to work from the Christmas break, we?ve found a problem (and we believe it is nginx + gunicorn related).<br class="">
<br class="">
1. Large downloads simply fail or are truncated to smaller sizes. We keep seeming to hit a magical 1.01 or 1.1GB limit when trying to download large things via the OMERO.web interface. Insight works fine, as it is unrelated to nginx.<br class="">
<br class="">
2. We?re running the config from the OMERO 5.2.1 documentation, nginx-wise. Nginx + WSGI + gunicorn atop Ubuntu 14.04 LTS. All up to date.<br class="">
<br class="">
3. I?ve read things about gzip compression having an impact. Tried to turn that off. No difference. I?ve read proxy_max_temp_file_size might have an impact. Fiddled with that - no difference. Fiddled with timeouts on upstream server. No difference.<br class="">
<br class="">
4. I can?t see anything useful in the error.log or access.log for nginx, nor in the Blitz.log etc.<br class="">
<br class="">
5. When your users are downloading 30GB+ .ims files out of OMERO regularly, this could present a problem for us, as you can appreciated ;).<br class="">
<br class="">
Any help would be appreciated - as I?m stumped on this one. This is usually just normal administration 101 ?just tune this? setting sort of thing, which doesn?t present much of a challenge - but this feels like a ghost.<br class="">
<br class="">
Regards.<br class="">
<br class="">
-jc<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">mailto:ome-users@lists.openmicroscopy.org.uk</a>><br class="">
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" class="">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br class="">
<br class="">
<br class="">
The University of Dundee is a registered Scottish Charity, No: SC015096<br class="">
-------------- next part --------------<br class="">
An HTML attachment was scrubbed...<br class="">
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20160112/e950ee4a/attachment-0001.html><br class="">
<br class="">
------------------------------<br class="">
<br class="">
Subject: Digest Footer<br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
ome-users@lists.openmicroscopy.org.uk<br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
<br class="">
<br class="">
------------------------------<br class="">
<br class="">
End of ome-users Digest, Vol 130, Issue 10<br class="">
******************************************<br class="">
</blockquote>
_______________________________________________<br class="">
ome-users mailing list<br class="">
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>