ChatOps for GitLab administration and monitoring

Description

Administering GitLab can be painful and time consuming. Let's make it better with ChatOps.

Proposal

GitLab should allow me to perform a number of admin tasks from ChatOps as well as notify me of problems as they come up.

Commands

/gitlab status

Not healthy: 4 of 5 servers online

/gitlab connections

Active connections: 98

server accepts handled requests

29408060 29408060 195568812

Reading: 0 Writing: 6 Waiting: 95

(from curl 127.0.0.1:8060/nginx_status)

/gitlab top-connections

115 connections from hostname.com

10 connections from bobhostname.com

(from netstat -at | awk '{print $4":"$5":"$6}' | awk -F: '{print $1":"$2" "$3" "$5}' | sort | uniq -c | sort -nr | head )

/gitlab whos-gitting

The following repositories currently have Git operations in progress

...

/gitlab top

The following repositories are using the most CPU

/gitlab uptime

19:11:04 up 85 days, 6:41, 1 user, load average: 11.04, 11.32, 11.40

/gitlab limit-connections 5

Connections limited to 5 per hostname

/gitlab throttle 15m

Throttling connections for next 15 minutes

Alerts

CPU utilization at 95%

Disk space utilization at 90%

randomhost.com has 50 open connections. Limit them?

Gitaly server nfs-1.gitlab.com is offline!

Links / references

https://gitlab.com/gitlab-org/gitlab-ce/issues/34311