[ome-users] Monitoring OMERO activity and storage

[Carnë Draug]

On 22 January 2018 at 15:30, Julio MATEOS_LANGERAK
<julio.mateos-langerak at igh.cnrs.fr> wrote:
> Dear all,
>
> I’d like to collect any information / experience that would help us
> to monitor over time our OMERO server activity. The idea is:
>
> - to follow activity over time to be able to trace possible problems
>   and the causes. We’d like to monitor nr of users connected,
>   CPU/network activity, etc

For monitoring server status we have a nagios instance.  It's the same
that monitors all of our servers, not only omero.  We don't do any
omero specific checks.

> - to establish a storage management plan, we’d like to automate the
>   quota per user/group over time,…

To get storage statistics, we have a python script [1] which outputs a
json file with the following structure:

            omerogroup1:
                omero_user_id_1 : nbytes
                omero_user_id_2 : nbytes
                omero_user_id_3 : nbytes
            omerogroup2:
                omero_user_id_1 : nbytes
                omero_user_id_4 : nbytes
                omero_user_id_5 : nbytes

We have a cronjob that runs it weekly and another python script [2] to
get some basic analytics out of it.  If plots the total omero disk
usage over time, and the current usage by PI group and department.

The script may look a bit complicated because it includes a lot of
data mapping omero gids to a PI group, and because it is also used to
generate stats from individual users home directories quotas.

Carnë

[1] https://github.com/MicronOxford/scripts/blob/d9f979b39b8f2acccda7aacf245a44cb65c7ddc2/sysadmin/omero-du.py
[2] https://github.com/MicronOxford/scripts/blob/de3a2ae914dc332ac3cc4524c7bafe41a0a3ed68/sysadmin/micron-du-stats.py