<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p dir="ltr">Hi Jake</p>
<p dir="ltr">We're still investigating the problem- our silence since your last email is because we're still unsure of the best way to fix it. We can reproduce it on our systems, and we've tested multiple alternative configurations, none of which solved the
problem. We'll continue investigating this week, as soon as we have a solution we'll let you know.</p>
<p dir="ltr">Simon<br>
</p>
<div class="gmail_quote">On 31 Jan 2016 01:31, "Jake Carroll" <<a href="mailto:jake.carroll@uq.edu.au">jake.carroll@uq.edu.au</a>> wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi again<br>
<br>
Unfortunately, still having issues on large downloads failing via the web interface.<br>
<br>
I'm using a startup string such as this:<br>
<br>
omero web start --workers 128 --wsgi-args '--worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br>
<br>
And it doesn't seem to really matter what workers INT I use, we'll still see stalls and fails on download over the web interface.<br>
<br>
I'm trying to download a 9.5GB ims format file.<br>
<br>
The g_error.log looks interesting?<br>
<br>
root@omero-prod-gen2:~# tail -f ~omero/OMERO.server/var/log/g_error.log<br>
2016-01-31 09:23:53 [4781] [INFO] Booting worker with pid: 4781<br>
2016-01-31 09:23:53 [4794] [INFO] Booting worker with pid: 4794<br>
2016-01-31 09:23:53 [4798] [INFO] Booting worker with pid: 4798<br>
2016-01-31 09:23:53 [4814] [INFO] Booting worker with pid: 4814<br>
2016-01-31 09:23:53 [4808] [INFO] Booting worker with pid: 4808<br>
2016-01-31 09:23:53 [4823] [INFO] Booting worker with pid: 4823<br>
2016-01-31 09:23:53 [4827] [INFO] Booting worker with pid: 4827<br>
2016-01-31 09:23:53 [4838] [INFO] Booting worker with pid: 4838<br>
2016-01-31 09:23:53 [4858] [INFO] Booting worker with pid: 4858<br>
2016-01-31 09:23:53 [4874] [INFO] Booting worker with pid: 4874<br>
2016-01-31 09:26:00 [3852] [CRITICAL] WORKER TIMEOUT (pid:4608)<br>
2016-01-31 09:26:00 [3852] [CRITICAL] WORKER TIMEOUT (pid:4608)<br>
2016-01-31 09:26:01 [5314] [INFO] Booting worker with pid: 5314<br>
<br>
I managed to download (randomly?) more than I ever have before, with 1.7GB of the file downloaded in this configuration - but it is still failing/stalling.<br>
<br>
What could I be missing?<br>
<br>
I even tried with 256 workers:<br>
<br>
omero@omero-prod-gen2:~$ omero web start --workers 256 --wsgi-args '--worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br>
<br>
...but the workers still seem to time out at *some* random point early on:<br>
<br>
2016-01-31 09:29:24 [7360] [INFO] Booting worker with pid: 7360<br>
2016-01-31 09:29:24 [7371] [INFO] Booting worker with pid: 7371<br>
2016-01-31 09:30:14 [5433] [CRITICAL] WORKER TIMEOUT (pid:7045) <-- happened almost immediately after booting the workers.<br>
2016-01-31 09:30:14 [5433] [CRITICAL] WORKER TIMEOUT (pid:7045)<br>
2016-01-31 09:30:15 [8273] [INFO] Booting worker with pid: 8273<br>
<br>
*SO THEN* I tried booting the worker processes with a very long time out:<br>
<br>
omero web start --workers 256 --wsgi-args '-t 360 --worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br>
<br>
And, after a much much much longer download length of 4.2GB of my 9.5GB ims file it finally started to show problem signs again:<br>
<br>
<br>
2016-01-31 09:49:32 [8394] [CRITICAL] WORKER TIMEOUT (pid:10451)<br>
2016-01-31 09:49:32 [8394] [CRITICAL] WORKER TIMEOUT (pid:10451)<br>
2016-01-31 09:49:33 [11503] [INFO] Booting worker with pid: 11503<br>
<br>
And then it failed again, unfortunately.<br>
<br>
So made the timeout an enormous number:<br>
<br>
omero web start --workers 256 --wsgi-args '-t 1440 --worker-class gevent --error-logfile=/home/omero/OMERO.server/var/log/g_error.log'<br>
<br>
...and I can finally drag in my 9.5GB file over the OMERO web interface, without timeout failures.<br>
<br>
Something doesn't feel quite right, does it?<br>
<br>
-jc<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
ome-users mailing list<br>
<a href="mailto:ome-users@lists.openmicroscopy.org.uk">ome-users@lists.openmicroscopy.org.uk</a><br>
<a href="http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users" rel="noreferrer" target="_blank">http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users</a><br>
<br>
The University of Dundee is a registered Scottish Charity, No: SC015096<br>
</blockquote>
</div>
<br>
<span style="font-size:10pt;">The University of Dundee is a registered Scottish Charity, No: SC015096</span>
</body>
</html>