[ome-users] Monitoring OMERO activity and storage

[Kenneth Gillen (Staff)]

Dear Julio,

Thanks for starting this thread! My own setup monitoring an institutional OMERO server is pretty basic.

I run a Check_MK deployment installed via the “OMD” [1] distribution, included in CentOS/RH epel repository.

This I use to monitor:

  System stats: CPU % usage, RAM % usage, % disk free,  network etc
  Existence (and age) of files, primarily for PostgreSQL nightly pg_dump backups. (e.g. alert if the backup file is >25hrs old)
  OMERO (and system) log file parsing: A set of regular expressions match (or ignore), and alert accordingly. (e.g. Alert if anything appears in OMERO.server/var/log/master.err, or `ERR` appears in Blitz-0.log) [2]

I have alert thresholds configured, and it will email a distribution list and alert into a slack channel, once a threshold is reached (for a set configurable amount of time).

Check_MK has a concept of “local” checks, with which you can write a script and deploy on the host you wish to monitor that produces some output which Check_MK then parses, to be able to monitor anything you can query on the linux command line.

I previously used a python script like this to monitor the number of users connected to OMERO via parsing the output of `bin/omero sessions who`, but this was relying on the `guest` account, and the OME team (quite rightly) stopped the ability of the `guest` user accessing `sessions who` output. I’ve been told I could supply OMERO account credentials to the python script, and have it connect with that account, so if I get that deployed, I’ll pop the script up on github along with the rest of the production server config [3].

[1]  http://mathias-kettner.com/checkmk_install_with_omd.html
[2] https://github.com/openmicroscopy/prod-playbooks/blob/2018-01/ome-demoserver.yml#L266-L288
[3] https://github.com/openmicroscopy/prod-playbooks/

As for data about groups/users – I’ve not tried this. Certainly you could write a Check_MK local check to parse the OMERO.cli output, if the OMERO.cli provides that data, like I did for “users logged in to OMERO”, but I would imagine it’d be too computationally costly to run this command frequently, e.g. once per minute, like check_mk does. One could run the OMERO.cli command with cron.daily, and parse the result with a monitoring system like Check_MK.

I could imagine storage usage like that would be useful to include in an API within OMERO somewhere in the future, for parsing with tools like this.

I’m sure we’d all be interested to hear what everyone else is doing.

All the best,

Kenny

--

Kenneth Gillen
OME System Administrator

Wellcome Trust Centre for Gene Regulation & Expression
School of Life Sciences
CTIR 2
University of Dundee
Dow Street
Dundee  DD1 5EH
United Kingdom

Tel: +44 (0) 1382 388797

http://www.twitter.com/openmicroscopy

From: ome-users <ome-users-bounces at lists.openmicroscopy.org.uk<mailto:ome-users-bounces at lists.openmicroscopy.org.uk>> on behalf of Julio MATEOS_LANGERAK <julio.mateos-langerak at igh.cnrs.fr<mailto:julio.mateos-langerak at igh.cnrs.fr>>
Reply-To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Date: Monday, 22 January 2018 15:30
To: OME User Support List <ome-users at lists.openmicroscopy.org.uk<mailto:ome-users at lists.openmicroscopy.org.uk>>
Subject: [ome-users] Monitoring OMERO activity and storage

Dear all,

I’d like to collect any information / experience that would help us to monitor over time our OMERO server activity. The idea is:

- to follow activity over time to be able to trace possible problems and the causes. We’d like to monitor nr of users connected, CPU/network activity, etc
- to establish a storage management plan, we’d like to automate the quota per user/group over time,…

Any ideas, scripts, etc are welcome.

Cheers, Julio



__________________________________________
Julio Mateos Langerak, PhD.
Arnaud de Villeneuve Campus Imaging Facility
Montpellier RIO Imaging
Institut de Génétique Humaine-CNRS
141, rue de la Cardonille
F-34396 Montpellier (France)
e-mail: Julio.Mateos-Langerak at igh.cnrs.fr<mailto:Julio.Mateos-Langerak at igh.cnrs.fr>
phone: +33.4.34.35.99.90
fax: +33.4.34.35.99.01
__________________________________________







The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openmicroscopy.org.uk/pipermail/ome-users/attachments/20180125/c3c96915/attachment.html>