[ome-users] Server error (500). Why is this displayed

Simon Li s.p.li at dundee.ac.uk
Tue Jun 17 17:02:28 BST 2014


Hi John

I’m glad you’ve got it working! Your diagnostics look fine, so unless the problem re-appears I don’t think there’s anything else we can do.

Cheers

Simon


On 17 Jun 2014, at 16:35, John Webber (NBI) <John.Webber at nbi.ac.uk<mailto:John.Webber at nbi.ac.uk>> wrote:

Hi Simon,

Thanks again for your assistance with this issue.

I have made the first of the two changes that you suggested below – to fix the error handler to work with Python 2.4.

As well as making this change, I have also rebooted the server and cleared my browser cache.  I’m not sure which of these was finally resolved the issue, but I don’t seem to be getting the “Server error[500]” any longer.

I have attached the output from “omero admin diagnostics` and also my apache config files, in case there is still another issue lurking!

Thanks
John



From: Simon Li [mailto:s.p.li at dundee.ac.uk]
Sent: 13 June 2014 11:59
To: John Webber (NBI)
Cc: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] Server error (500). Why is this displayed

Hi John

I can’t see anything obviously wrong, so I can think of a couple of options:

1. Fix the error handler to work with Python 2.4:
Edit /opt/OMERO.server-4.4.8p1-ice33-b304/lib/python/omeroweb/feedback/views.py
and change the logging statement at line 159 from:

158 def handler404(request):
159     logger.warning('Not Found: %s' % request.path,
160                 extra={
161                     'status_code': 404,
162                     'request': request
163                 })
164     if request.is<http://request.is/>_ajax():
165         msg = traceback.format_exception(*sys.exc_info())[-1]
166         return HttpResponseNotFound(msg)
167
168     return page_not_found(request, "404.html”)

to:

158 def handler404(request):
159     logger.warning('Not Found: %s' % request.path)
160     if request.is<http://request.is/>_ajax():
161         msg = traceback.format_exception(*sys.exc_info())[-1]
162         return HttpResponseNotFound(msg)
163
164     return page_not_found(request, "404.html”)

If this works the server error 500 message should be more informative, and you’ll hopefully have a warning message in OMEROweb.log.

2. Go back to basics. Are you able to setup OMERO.web in development mode instead of running via Apache?

omero config set omero.web.application_server development
omero config set omero.web.debug True

Restart, and see if the problems still occur using http://server:4080/

If you need to keep the main instance running one thing you could try is running a second OMERO.web in development mode (but connecting to the same OMERO.server). This isn’t ideal, since the OMERO sessions will be shared between both OMERO.web instances:

cp -a OMERO.server-XXX OMERO.server-webdev
cd OMERO.server-webdev
rm -rf var
bin/omero config set omero.web.application_server development
bin/omero config set omero.web.debug True
# The main OMERO.web will be using port 4080 internally to communicate with Apache
bin/omero config set omero.web.application_server.port 14080
bin/omero web start

and go to http://server:14080/

Could you also send us the output from `omero admin diagnostics` and your apache config files, e;.g.
/etc/httpd/conf.d/<OMERO>.conf
/etc/httpd/conf.d/fastcgi.conf

Cheers

Simon

On 12 Jun 2014, at 15:53, John Webber (NBI) <John.Webber at nbi.ac.uk<mailto:John.Webber at nbi.ac.uk>> wrote:


Simon (et al)

Sorry for the delay in responding to your email below – it’s been a busy week.

I have done some more investigation into this issue.

I’ve found that,
•         If I log off of the Omero server correctly, I do not encounter any issues when I attempt to log on again.
•         Similarly, on different Omero server, I can close (crash) the browser window without logging off, and when I go back on to that site, I am automatically logged straight back in.  I do not encounter the Server error (500).
•         On this Omero server, if I close (crash) the browser window without logging off, when I go back on to that site I am NOT automatically logged in – instead the server displays the normal OMERO.web login (enter username and password) screen.
•         When I enter my credentials on this OMERO.web login screen I then see the “Server error (500)” error.  Once this screen has displayed, I can click “Webadmin” or “Webclient” and use the site as normal.

Please see attached two different log files, one from a successful log into Omero, and one from a login that encountered a “Server error (500)”.

The logs are the same, down to the following lines:

2014-06-12 14:02:21,978 INFO  [                 org.perf4j.TimingLogger] (l.Server-9) start[1402578141975] time[3] tag[omero.call.success.ome.logic.AdminImpl.getSecurityRoles]
2014-06-12 14:02:21,985 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Rslt:   ome.system.Roles at 67d1de8b<mailto:ome.system.Roles at 67d1de8b>
2014-06-12 14:02:22,019 INFO  [ ome.services.blitz.fire.SessionManagerI] (l.Server-3) Found session locally: 5a1dc603-b2de-4dfc-99a6-6fe724e2521d

On the session that displays the “Server error (500)” error, the last of the lines above is not shown.

I have checked the Apache logs, and am not able to find any messages in either the /var/log/httpd/access_log or the /var/log/httpd/error_log at the time I experience the issue.  The only logs being written to when these issues occur are the /var/log/httpd/ssl_request_log, which has the following entries:

[12/Jun/2014:15:49:42 +0100] 149.155.213.117 TLSv1 DHE-RSA-AES128-SHA "POST /webclient/login/?url=FwebclientF HTTP/1.1" -
[12/Jun/2014:15:49:43 +0100] 149.155.213.117 TLSv1 DHE-RSA-AES128-SHA "GET /webclient/login/FwebclientF HTTP/1.1" 6032
[12/Jun/2014:15:49:43 +0100] 149.155.213.117 TLSv1 DHE-RSA-AES128-SHA "GET /static/feedback/css/layout.css HTTP/1.1" -

and the /var/log/httpd/ssl_access_log, which has the following entries:

149.155.213.117 - - [12/Jun/2014:15:47:34 +0100] "POST /webclient/login/?url=FwebclientF HTTP/1.1" 302 -
149.155.213.117 - - [12/Jun/2014:15:47:34 +0100] "GET /webclient/login/FwebclientF HTTP/1.1" 200 6032
149.155.213.117 - - [12/Jun/2014:15:47:35 +0100] "GET /static/feedback/css/layout.css HTTP/1.1" 304 -

Does anyone have any ideas on what is going wrong with this server?

Thanks
John



From: Simon Li [mailto:s.p.li at dundee.ac.uk]
Sent: 09 June 2014 10:53
To: John Webber (NBI)
Cc: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] Server error (500). Why is this displayed

Hi John

It sounds like you stopped OMERO.web successfully, I can't think of anything else that would be running other than django. The traceback error is occurring in the OMERO.web 404 error handler- it looks like the logging module in Python 2.4 doesn't support the additional 'extra' parameter used in omeroweb/feedback/views.py:160. Unfortunately this makes it difficult to figure out what the exact error is!

Could you check your Apache logs for clues?
/var/log/httpd/access_log and particularly /var/log/httpd/error_log

One other thing you could try is enabling debug mode in OMERO.web to see if that gives any additional information:
omero config set omero.web.debug True
but remember to disable it once the problem is fixed

Cheers

Simon

On 6 Jun 2014, at 16:20, John Webber (NBI) <John.Webber at nbi.ac.uk<mailto:John.Webber at nbi.ac.uk>> wrote:



Hi Simon,

Thanks for your quick response (as normal!) ;-)

I have stopped and restarted OMERO.web, as requested, using “<omero>/bin/omero web stop” and “<omero>/bin/omero web start”.  Once I had stopped OMERO.web, all of the “django” processes did stop correctly.  Are there any other processes that I should check as well?

OMERO.web stopped and restarted successfully, but the “/opt/Omero/var/log/OMEROweb.log” log file is STILL not being written to.  There are also no other OMEROweb.log files on the system.

The “Server Error (500)” message that I am seeing includes the following error message:

Traceback (most recent call last):

  File "/opt/OMERO.server-4.4.8p1-ice33-b304/lib/python/django/core/handlers/base.py", line 150, in get_response
    response = callback(request, **param_dict)

  File "/opt/OMERO.server-4.4.8p1-ice33-b304/lib/python/omeroweb/feedback/views.py", line 160, in handler404
   extra={

  File "/usr/lib64/python2.4/logging/__init__.py", line 983, in warning
    apply(self._log, (WARNING, msg, args), kwargs)

TypeError: _log() got an unexpected keyword argument 'extra'


<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{'sessionid': '249cf076455eda32b05b4212a84d86eb'},
META:{'DOCUMENT_ROOT': '/var/www/html',
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTPS': 'on',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8',
'HTTP_CACHE_CONTROL': 'max-age=0',
'HTTP_CONNECTION': 'keep-alive',
'HTTP_COOKIE': 'sessionid=249cf076455eda32b05b4212a84d86eb',
'HTTP_HOST': 'open-omero.nbi.ac.uk<http://open-omero.nbi.ac.uk/>',
'HTTP_REFERER': 'https://open-omero.nbi.ac.uk/webclient/login/?url=FwebclientF',
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36',
'PATH': '/sbin:/usr/sbin:/bin:/usr/bin',
'PATH_INFO': u'/webclient/login/FwebclientF',
'PATH_TRANSLATED': '/opt/OMERO.server-4.4.8p1-ice33-b304/var/omero.fcgi/webclient/login/FwebclientF',
'QUERY_STRING': '',
'REMOTE_ADDR': '149.155.213.117',
'REMOTE_PORT': '61082',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/webclient/login/FwebclientF',
'SCRIPT_FILENAME': '/opt/OMERO.server-4.4.8p1-ice33-b304/var/omero.fcgi',
'SCRIPT_NAME': u'',
'SERVER_ADDR': '149.155.214.217',
'SERVER_ADMIN': 'root at localhost',
'SERVER_NAME': 'open-omero.nbi.ac.uk<http://open-omero.nbi.ac.uk/>',
'SERVER_PORT': '443',
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SIGNATURE': '<address>Apache/2.2.3 (CentOS) Server at open-omero.nbi.ac.uk<http://open-omero.nbi.ac.uk/> Port 443</address>\n',
'SERVER_SOFTWARE': 'Apache/2.2.3 (CentOS)',
'wsgi.errors': <flup.server.fcgi_base.TeeOutputStream object at 0x10548090>,
'wsgi.input': <flup.server.fcgi_base.InputStream object at 0x10548510>,
'wsgi.multiprocess': True,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'https',
'wsgi.version': (1, 0)}>”

Please can you advise how it is best to proceed with this issue?

Thanks
John



From: Simon Li [mailto:s.p.li at dundee.ac.uk<http://dundee.ac.uk/>]
Sent: 06 June 2014 15:06
To: John Webber (NBI)
Subject: Re: [ome-users] Server error (500). Why is this displayed

Hi John

You're right, I should've said OMEROweb.log. The fact that it's empty is a bit strange though. Is it possible you've got an old OMERO.web running from a previous server upgrade? Could you run `omero web stop`, then verify all relevant processes are killed (you can use something like `ps -ef | grep django` to check). If there are any still present you can try killing them manually, then restart OMERO.web?

If that doesn't work then could you copy and paste the error message from the dialog- if there's no private information it would be helpful if you could CC the ome-users list, so that people don't see a hanging conversation. If we still can't figure out the problem then server credentials could be useful.

Hope this helps,

Simon

On 6 Jun 2014, at 11:45, "John Webber (NBI)" <John.Webber at nbi.ac.uk<mailto:John.Webber at nbi.ac.uk>>
 wrote:




Hi Simon,

I’m afraid I am not able to find a log file called  OMERO_web.log!  Where should this be located?

There is a logfile in <omero>/var/log called “OMEROweb.log”, but the date on this file is September, and the file is empty!  There is also a file called “OMEROweb_request.log”, which the same date which is also empty!

I have looked through the blitz log, but this does not seem to contain any errors or warnings at the time when I see this message.  Would you like me to send you a copy of that?

The server is actually available through our firewall, so I can give you credentials on the server for you to see the error occur, if you would like!

Thanks
John



From: Simon Li [mailto:s.p.li at dundee.ac.uk<http://dundee.ac.uk/>]
Sent: 06 June 2014 10:21
To: John Webber (NBI)
Cc: ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
Subject: Re: [ome-users] Server error (500). Why is this displayed

Hi John

You should be able to login from multiple browsers/locations without errors. Could you send us your OMERO_web.log?

Thanks

Simon


On 6 Jun 2014, at 10:34, John Webber (NBI) <John.Webber at nbi.ac.uk<mailto:John.Webber at nbi.ac.uk>> wrote:





Hi,

I have built a new omero server, but during testing I sometimes see the following “Server Error (500)” screen when I log in.  I am using Omero 4.4.8 with ice 3.3.

I notice that this seems to occur when I access the same user account from different computers or browsers.  Is this because I have not logged out correctly from another sessions?  Is there a way to remove the likelihood of users seeing this screen?

Thanks
John



<image001.png>

--------------------------------------------------------------------------------------------------------------------------------------------
John Webber
Computing and Information Services
NBI Partnership Ltd
Norwich Research park
Colney Lane
Norwich
NR4 7UH

E-Mail: john.webber at nbi.ac.uk<mailto:john.webber at nbi.ac.uk>
Web:     http://www.nbi.ac.uk<http://www.nbi.ac.uk/>
Tel:         +44(0)1603 450 818

Helpdesk (Tel):  1234      HPC Support (Web):       http://hpc.nbi.ac.uk<http://hpc.nbi.ac.uk/>  (NBI  internal)
                                                HPC Support (Tel):           2003  (NBI internal)

The NBI Partnership Ltd provides non-scientific services to the Institute of Food Research, the John Innes Centre, The Genome Analysis Centre and The Sainsbury Laboratory

_______________________________________________
ome-users mailing list
ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users


The University of Dundee is a registered Scottish Charity, No: SC015096


The University of Dundee is a registered Scottish Charity, No: SC015096


The University of Dundee is a registered Scottish Charity, No: SC015096
<successful.login><500error.login>


The University of Dundee is a registered Scottish Charity, No: SC015096
<fastcgi.conf><httpd.conf><omero-admin-diagnostics.txt>


The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20140617/ce1fc72b/attachment.html>


More information about the ome-users mailing list