[ome-users] Improving webclient performance
Paul van Schayck
paul at vanschayck.nl
Wed May 11 16:31:28 BST 2016
Hi Ola,
Thanks for the help. I probably forgot to mention the crucial bit that
I'm running OMERO 5.1.4. Could that be the biggest influence, looking
at all the changes you've made to OMERO.web organization between 5.1
and 5.2?
I've done some more testing as you proposed using `ab` and the P-TRE
test image. Also setting public access to not use bSession.
$ ab -n 10 -c 1
"http://omero.local/webclient/render_thumbnail/size/96/201/?server=1&bsession=ae766ef0-7520-4878-822c-419eb3faf6b8"
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 1 0.1 1 1
Processing: 268 319 18.6 323 338
Waiting: 268 319 18.5 323 338
Total: 269 320 18.5 324 338
$ ab -n 10 -c 1
"http://nanoscopy.unimaas.nl/webclient/render_thumbnail/size/96/201/"
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 1 0.1 1 1
Processing: 391 412 11.2 415 420
Waiting: 390 412 11.2 415 420
Total: 391 412 11.2 416 421
As you can see, actually bSession is faster than (by about 100ms) than
the public user. The render_thumbnail is about 100ms faster than the
webclient/ index call I tested with yesterday. But overall, the 320 ms
average is pretty far from the 100 to 150 you report.
I've attached the requested outputs. Note that the amount of errors in
Blitz.log is most likely due to a lot of experimenting I've done in
the past.
Cheers,
Paul
On Tue, May 10, 2016 at 9:07 PM, Aleksandra Tarkowska (Staff)
<A.Tarkowska at dundee.ac.uk> wrote:
> Sorry forgot to mention, I'd expect single render_thumbnail response takes about 100-150 ms
>
> Ola
>
>> On 10 May 2016, at 18:48, Aleksandra Tarkowska (Staff) <A.Tarkowska at dundee.ac.uk> wrote:
>>
>> Hi Paul,
>>
>> could you provide output of
>>
>> - bin/omero admin diagnostics
>> - ps aux | grep gunicorn
>> - copy of your nginx.conf and omero-web.conf
>>
>> Are you using any proxy server in front of nginx?
>>
>> It is possible that bsession is slow, could you set public user, and import one of our example P-TRE images from http://downloads.openmicroscopy.org/images/DV/ . It will be easier to compare with benchmark on http://www.openmicroscopy.org/site/support/omero5.2/sysadmins/unix/install-web/install-nginx.html
>>
>>
>> Thanks
>> Ola
>>
>>> On 10 May 2016, at 14:35, Paul van Schayck <paul at vanschayck.nl> wrote:
>>>
>>> Dear all,
>>>
>>> I've been looking into the OMERO.web requests time. We sometimes have
>>> the feeling that the interface is a bit sluggish, and could respond
>>> more snappy. So I've done some small tests.
>>>
>>> Note that this is not a detailed analysis yet. But based on viewing
>>> the logs and doing some small experiments for an hour or two.
>>>
>>> * Tests done on Ubuntu 14.04 VM with nginx and gunicorn. 4 cores, 4
>>> GiB of memory. Increased shared memory settings for postgresql. No
>>> other load on the server.
>>>
>>> * Requests to 404 pages of OMERO.web have a typical response time of
>>> 25 ms. Leading me to believe that the nginx/gunicorn combination gives
>>> relative little overhead.
>>>
>>> * Requests to real OMERO pages (like metadata details, the main index
>>> or for example a tile) give about a time anywhere between 400 and 500
>>> ms. With sometimes outliers to 300ms or 600ms.
>>>
>>> * Because I saw so little difference between metadata details or an
>>> image tile I opted to just get the main index page. I used this
>>> command in the end for most testing:
>>>
>>> curl "http://omero.local/webclient/?server=1&bsession=21145499-5e24-451c-b5de-5b95c95d7c9a"
>>> -w "time_starttransfer: %{time_starttransfer}, time_total:
>>> %{time_total}\n" -o /dev/null
>>>
>>> A typical request would give this output:
>>>
>>> time_starttransfer: 0,412 s, time_total: 0,415 s
>>>
>>>
>>> Looking at this representative Blitz.log (also attached) the following
>>> things stood out for me:
>>>
>>> * The time spent directly in OMERO is about 383 ms. Confirming again
>>> that the network, nginx and gunicorn overhead is low as the total
>>> request (as seen from the client was 415 ms)
>>>
>>> * There are 13 calls to getConfigValue. In total spending about 143 ms.
>>>
>>> * The queries executed are only a few ms.
>>>
>>> * There is a (unexplained for me) gap between two queries of 80 ms.
>>>
>>>
>>> What really struck me is the 13 calls to getConfigValue taking so much
>>> time. In general this is about 25% of the request time. Especially as
>>> those values will in general be static between requests. Is there
>>> something wrong on my side? Is there anything else I can do to speed
>>> up web requests?
>>>
>>> Thanks,
>>>
>>> Paul van Schayck
>>> <Blitz.log>
>>> _______________________________________________
>>> ome-users mailing list
>>> ome-users at lists.openmicroscopy.org.uk
>>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>>
>> The University of Dundee is a registered Scottish Charity, No: SC015096
>> _______________________________________________
>> ome-users mailing list
>> ome-users at lists.openmicroscopy.org.uk
>> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
> _______________________________________________
> ome-users mailing list
> ome-users at lists.openmicroscopy.org.uk
> http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users
-------------- next part --------------
user www-data;
worker_processes 4;
pid /run/nginx.pid;
events {
worker_connections 768;
multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip off;
gzip_disable "msie6";
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
##
# Custom buffer settings
##
##
# nginx-naxsi config
##
# Uncomment it if you installed nginx-naxsi
##
#include /etc/nginx/naxsi_core.rules;
##
# nginx-passenger config
##
# Uncomment it if you installed nginx-passenger
##
#passenger_root /usr;
#passenger_ruby /usr/bin/ruby;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
-------------- next part --------------
upstream omeroweb {
server 127.0.0.1:4080 fail_timeout=0;
}
server {
listen 80;
server_name $hostname;
sendfile on;
client_max_body_size 0;
# maintenance page serve from here
location @maintenance {
root /home/m4i-nano-omero/OMERO.server/etc/templates/error;
try_files $uri /maintainance.html =502;
}
# weblitz django apps serve media from here
location /static {
alias /home/m4i-nano-omero/OMERO.server/lib/python/omeroweb/static;
}
location /pathviewer/viewer/static {
alias /home/m4i-nano-omero/OMERO.server/lib/python/omeroweb/static;
}
location @proxy_to_app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://omeroweb;
}
location / {
error_page 502 @maintenance;
# checks for static file, if not found proxy to app
try_files $uri @proxy_to_app;
}
}
-------------- next part --------------
m4i-nan+ 11918 0.0 0.1 50784 12804 ? S 16:38 0:00 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 11923 0.1 1.0 595508 82172 ? S 16:38 0:02 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 11924 0.1 1.0 596036 82728 ? S 16:38 0:02 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 11925 0.1 1.0 595528 82224 ? S 16:38 0:02 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 11926 0.1 0.9 594752 81500 ? S 16:38 0:01 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 11927 0.2 1.0 596584 83276 ? S 16:38 0:03 /usr/bin/python /usr/bin/gunicorn -D -p /home/m4i-nano-omero/OMERO.server/var/django.pid --bind 127.0.0.1:4080 --workers 5 --worker-connections 1000 --max-requests 0 omeroweb.wsgi:application
m4i-nan+ 14653 0.0 0.0 11748 2208 pts/1 S+ 17:09 0:00 grep --color=auto gunicorn
================================================================================
OMERO Diagnostics 5.1.4-ice35-b55
================================================================================
Commands: java -version 1.8.0 (/usr/bin/java)
Commands: python -V 2.7.6 (/usr/bin/python)
Commands: icegridnode --version 3.5.1 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.5.1 (/usr/bin/icegridadmin)
Commands: psql --version 9.3.12 (/usr/bin/psql)
Server: icegridnode running
Server: Blitz-0 active (pid = 10812, enabled)
Server: FileServer active (pid = 10827, enabled)
Server: Indexer-0 active (pid = 10838, enabled)
Server: MonitorServer active (pid = 10835, enabled)
Server: OMERO.Glacier2 active (pid = 10837, enabled)
Server: OMERO.IceStorm active (pid = 10849, enabled)
Server: PixelData-0 active (pid = 10831, enabled)
Server: Processor-0 active (pid = 10840, enabled)
Server: Tables-0 active (pid = 10855, enabled)
Log dir: /home/m4i-nano-omero/OMERO.server/var/log exists
Log files: Blitz-0.log 110.0 MB errors=66 warnings=93
Log files: Blitz-0.log.1 524.0 MB errors=213 warnings=167
Log files: Blitz-0.log.2 524.0 MB errors=41 warnings=192
Log files: Blitz-0.log.3 524.0 MB errors=66 warnings=170
Log files: Blitz-0.log.4 524.0 MB errors=307 warnings=130
Log files: Blitz-0.log.5 524.0 MB errors=862 warnings=160
Log files: Blitz-0.log.6 524.0 MB errors=0 warnings=76
Log files: Blitz-0.log.7 524.0 MB errors=487 warnings=113
Log files: Blitz-0.log.8 524.0 MB errors=7584 warnings=2210
Log files: DropBox.log 79.0 KB errors=1 warnings=30
Log files: FileServer.log 23.0 KB
Log files: Indexer-0.log 3.0 MB errors=204 warnings=12
Log files: MonitorServer.log 54.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 3.0 MB errors=53 warnings=25
Log files: OMEROweb.log.1 5.0 MB
Log files: OMEROweb.log.2 5.0 MB
Log files: OMEROweb.log.3 5.0 MB errors=4 warnings=0
Log files: OMEROweb.log.4 5.0 MB errors=6 warnings=5
Log files: OMEROweb.log.5 5.0 MB errors=5 warnings=0
Log files: OMEROweb.log.6 5.0 MB errors=40 warnings=12
Log files: OMEROweb.log.7 5.0 MB errors=14 warnings=0
Log files: OMEROweb.log.8 5.0 MB
Log files: OMEROweb.log.9 5.0 MB errors=5 warnings=864
Log files: OMEROweb_brokenrequest.lock 0.0 KB
Log files: OMEROweb_brokenrequest.log 1.0 MB errors=339 warnings=295
Log files: PixelData-0.log 1.0 MB errors=176 warnings=13
Log files: Processor-0.log 2.0 MB errors=543 warnings=5
Log files: Processor-0.log.1 4.0 MB errors=1393 warnings=2
Log files: Processor-0.log.2 4.0 MB errors=1393 warnings=25
Log files: Tables-0.log 113.0 KB errors=0 warnings=33
Log files: TestDropBox.log n/a
Log files: master.err 288.0 KB errors=0 warnings=173
Log files: master.out 0.0 KB
Log files: Total size 4375.87 MB
Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)
OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/home/m4i-nano-omero/omero/tmp' Exists? True Is writable? True (Size: 0)
JVM settings: Blitz -Xmx4186m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'percent': '50'})
JVM settings: Indexer -Xmx837m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Pixeldata -Xmx1255m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository -Xmx837m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
OMERO.web status... [RUNNING] (PID 11918)
More information about the ome-users
mailing list