<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Jake,
<div class=""><br class="">
</div>
<div class="">128/256 workers seams a lot. Did you have a chance to look at <a href="http://docs.gunicorn.org/en/stable/design.html#how-many-workers" class="">
http://docs.gunicorn.org/en/stable/design.html#how-many-workers</a>.</div>
<div class="">Are you using the latest libs: gunicorn 19.4.5, gevent 1.0.2 and nginx 1.9+?</div>
<div class=""><br class="">
</div>
<div class="">
<div apple-content-edited="true" class="">In your nginx config turn buffering off:</div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class="">
<div apple-content-edited="true" class=""><i class="">location @proxy_to_app {</i></div>
<div apple-content-edited="true" class=""><i class=""> ...</i></div>
<div apple-content-edited="true" class=""><i class=""> proxy_buffering off;</i></div>
<div apple-content-edited="true" class=""><i class=""> proxy_pass <a href="http://omeroweb_omero" class="">
http://omeroweb_omero</a>;</i></div>
<div apple-content-edited="true" class=""><i class=""> }</i></div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="">Start gunicorn in debug mode:</div>
<div class=""><br class="">
</div>
<div class="">
<div apple-content-edited="true" class=""><i class="">$ omero web start --workers 5 --worker-connections 1000 --wsgi-args ' --timeout 60 --graceful-timeout 60 --worker-class gevent --log-level=DEBUG --error-logfile=/home/omero/OMERO.server/var/log/gunicorn.log
'</i></div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div apple-content-edited="true" class="">
<div apple-content-edited="true" style="orphans: auto; text-align: start; text-indent: 0px; widows: auto;" class="">
<div apple-content-edited="true" class="">Make sure</div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class=""><i class="">omero config set omero.web.application_server.max_requests 0</i></div>
<div apple-content-edited="true" class=""><br class="">
</div>
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Gunicorn log will show:</div>
<div apple-content-edited="true" style="orphans: auto; text-align: start; text-indent: 0px; widows: auto;" class="">
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31569] [INFO] Starting gunicorn 19.4.5</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31569] [DEBUG] Arbiter booted</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31569] [INFO] Listening at:
<a href="http://127.0.0.1:4080" class="">http://127.0.0.1:4080</a> (31569)</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31569] [INFO] Using worker: gevent</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31574] [INFO] Booting worker with pid: 31574</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31575] [INFO] Booting worker with pid: 31575</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31576] [INFO] Booting worker with pid: 31576</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31581] [INFO] Booting worker with pid: 31581</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31590] [INFO] Booting worker with pid: 31590</i></div>
<div apple-content-edited="true" class=""><i class="">[2016-01-31 12:48:56 +0000] [31569] [DEBUG] 5 workers</i></div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class="">I tested that on 8 Cores Xeon with 20 simultaneous downloads of 3GB tiff, like:</div>
<div apple-content-edited="true" class=""> </div>
<div apple-content-edited="true" class=""><i class="">wget --limit-rate=10M -O /data/files/30661.tiff
<a href="https://server.openmicroscopy.org/omero/webgateway/archived_files/download/30661/" class="">
https://server.openmicroscopy.org/omero/webgateway/archived_files/download/30661/</a></i></div>
<div apple-content-edited="true" class=""><i class=""><br class="">
</i></div>
<div apple-content-edited="true" class="">
<div apple-content-edited="true" class=""><i class="">100%[======================================================================>] 3,087,100,535 9.76MB/s in 9m 21s </i></div>
</div>
<div apple-content-edited="true" class="">
<div apple-content-edited="true" class=""><i class="">2016-01-31 13:21:10 (5.25 MB/s) - '/data/files/30661.tiff' saved [3087100535/3087100535]</i></div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class="">There were no timeouts. OMERO.web was responding as well.</div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class="">As Simon mentioned, we experience problems with higher speed when transfer goes above 15-20MB/s as that blocks workers. This issue is under investigation.</div>
</div>
<div apple-content-edited="true" class="">Based on your example I am guessing your average speed was about 12MB/s, it could hit the limit.</div>
<div apple-content-edited="true" class=""><br class="">
</div>
<div apple-content-edited="true" class="">Could you try it and let us know?</div>
</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="Apple-interchange-newline">
Ola</div>
<div apple-content-edited="true" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Software Engineer</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Open Microscopy Environment</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
University of Dundee</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 31 Jan 2016, at 01:31, Jake Carroll <<a href="mailto:jake.carroll@uq.edu.au" class="">jake.carroll@uq.edu.au</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi again<br class="">
<br class="">
Unfortunately, still having issues on large downloads failing via the web interface.<br class="">
<br class="">
I'm using a startup string such as this:<br class="">
<br class="">
omero web start --workers 128 --wsgi-args '--worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br class="">
<br class="">
And it doesn't seem to really matter what workers INT I use, we'll still see stalls and fails on download over the web interface.<br class="">
<br class="">
I'm trying to download a 9.5GB ims format file.<br class="">
<br class="">
The g_error.log looks interesting?<br class="">
<br class="">
root@omero-prod-gen2:~# tail -f ~omero/OMERO.server/var/log/g_error.log<br class="">
2016-01-31 09:23:53 [4781] [INFO] Booting worker with pid: 4781<br class="">
2016-01-31 09:23:53 [4794] [INFO] Booting worker with pid: 4794<br class="">
2016-01-31 09:23:53 [4798] [INFO] Booting worker with pid: 4798<br class="">
2016-01-31 09:23:53 [4814] [INFO] Booting worker with pid: 4814<br class="">
2016-01-31 09:23:53 [4808] [INFO] Booting worker with pid: 4808<br class="">
2016-01-31 09:23:53 [4823] [INFO] Booting worker with pid: 4823<br class="">
2016-01-31 09:23:53 [4827] [INFO] Booting worker with pid: 4827<br class="">
2016-01-31 09:23:53 [4838] [INFO] Booting worker with pid: 4838<br class="">
2016-01-31 09:23:53 [4858] [INFO] Booting worker with pid: 4858<br class="">
2016-01-31 09:23:53 [4874] [INFO] Booting worker with pid: 4874<br class="">
2016-01-31 09:26:00 [3852] [CRITICAL] WORKER TIMEOUT (pid:4608)<br class="">
2016-01-31 09:26:00 [3852] [CRITICAL] WORKER TIMEOUT (pid:4608)<br class="">
2016-01-31 09:26:01 [5314] [INFO] Booting worker with pid: 5314<br class="">
<br class="">
I managed to download (randomly?) more than I ever have before, with 1.7GB of the file downloaded in this configuration - but it is still failing/stalling.<br class="">
<br class="">
What could I be missing?<br class="">
<br class="">
I even tried with 256 workers:<br class="">
<br class="">
omero@omero-prod-gen2:~$ omero web start --workers 256 --wsgi-args '--worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br class="">
<br class="">
...but the workers still seem to time out at *some* random point early on:<br class="">
<br class="">
2016-01-31 09:29:24 [7360] [INFO] Booting worker with pid: 7360<br class="">
2016-01-31 09:29:24 [7371] [INFO] Booting worker with pid: 7371<br class="">
2016-01-31 09:30:14 [5433] [CRITICAL] WORKER TIMEOUT (pid:7045) <-- happened almost immediately after booting the workers.<br class="">
2016-01-31 09:30:14 [5433] [CRITICAL] WORKER TIMEOUT (pid:7045)<br class="">
2016-01-31 09:30:15 [8273] [INFO] Booting worker with pid: 8273<br class="">
<br class="">
*SO THEN* I tried booting the worker processes with a very long time out:<br class="">
<br class="">
omero web start --workers 256 --wsgi-args '-t 360 --worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br class="">
<br class="">
And, after a much much much longer download length of 4.2GB of my 9.5GB ims file it finally started to show problem signs again:<br class="">
<br class="">
<br class="">
2016-01-31 09:49:32 [8394] [CRITICAL] WORKER TIMEOUT (pid:10451)<br class="">
2016-01-31 09:49:32 [8394] [CRITICAL] WORKER TIMEOUT (pid:10451)<br class="">
2016-01-31 09:49:33 [11503] [INFO] Booting worker with pid: 11503<br class="">
<br class="">
And then it failed again, unfortunately.<br class="">
<br class="">
So made the timeout an enormous number:<br class="">
<br class="">
omero web start --workers 256 --wsgi-args '-t 1440 --worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br class="">
<br class="">
...and I can finally drag in my 9.5GB file over the OMERO web interface, without timeout failures.<br class="">
<br class="">
Something doesn't feel quite right, does it?<br class="">
<br class="">
-jc<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
ome-users mailing list<br class="">
<a href="mailto:ome-users@lists.openmicroscopy.org.uk" class="">ome-users@lists.openmicroscopy.org.uk</a><br class="">
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>