Prometheus metrics should include Kubernetes namespace

Summary

Prometheus metrics for environments should include the Kubernetes namespace as well as the environment name.

Steps to reproduce

1. Configure Kubernetes and Prometheus integration for at least two projects on the same Prometheus server.
2. Create an environment in both projects called "staging".
3. View the environment metrics (CPU / memory) for each project.

What is the current bug behavior?

Both projects will show the exactly same data on the graphs, because the Prometheus query does not filter by Kubernetes namespace, only by environment; this means "average CPU" is actually the average CPU of all environments called "production", of any project, monitored by that Prometheus instance.

What is the expected correct behavior?

The CPU and memory graphs should show the CPU and memory use of the environment being monitored, not all environments with the same name.

Relevant logs and/or screenshots

https://people.torchbox.com/~felicity/tmp/gitlab-prom-1.PNG https://people.torchbox.com/~felicity/tmp/gitlab-prom-2.PNG https://people.torchbox.com/~felicity/tmp/gitlab-prom-3.PNG https://people.torchbox.com/~felicity/tmp/gitlab-prom-4.PNG

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

root@gitlab-3930788300-1grdh:/# gitlab-rake gitlab:env:info
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   10.5.0
Redis Version:  3.2.5
Git Version:    2.13.0
Sidekiq Version:5.0.0
Go Version:     unknown
GitLab information
Version:        9.3.2
Revision:       254b489
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.torchbox.com
HTTP Clone URL: https://git.torchbox.com/some-group/some-project.git
SSH Clone URL:  git@git.torchbox.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml, github
GitLab Shell
Version:        5.0.5
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

#### Results of GitLab application Check

Expand for output related to the GitLab application check

root@gitlab-3930788300-1grdh:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...
GitLab Shell version >= 5.0.5 ? ... OK (5.0.5)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
56/1 ... repository is empty
2/2 ... ok
17/4 ... ok
57/37 ... ok
57/38 ... ok
63/40 ... ok
63/41 ... ok
63/42 ... ok
63/43 ... ok
63/44 ... ok
67/45 ... repository is empty
63/46 ... ok
63/47 ... ok
7/48 ... ok
63/49 ... ok
63/50 ... ok
2/52 ... repository is empty
69/53 ... ok
70/54 ... ok
9/56 ... ok
2/59 ... ok
9/60 ... ok
67/61 ... repository is empty
7/62 ... ok
67/63 ... repository is empty
71/64 ... ok
67/65 ... repository is empty
75/66 ... repository is empty
72/67 ... ok
75/68 ... repository is empty
75/69 ... repository is empty
75/70 ... repository is empty
63/71 ... ok
63/72 ... ok
70/73 ... ok
67/74 ... ok
67/75 ... repository is empty
111/109 ... ok
63/142 ... ok
63/143 ... ok
54/176 ... ok
11/177 ... ok
11/178 ... ok
11/179 ... ok
11/180 ... ok
11/181 ... ok
78/214 ... ok
9/247 ... ok
62/250 ... ok
70/283 ... ok
280/284 ... ok
8/317 ... ok
67/350 ... repository is empty
70/351 ... ok
71/352 ... ok
283/353 ... ok
63/354 ... ok
287/355 ... ok
287/356 ... ok
54/357 ... ok
287/358 ... ok
289/359 ... ok
55/361 ... ok
287/362 ... ok
287/365 ... ok
292/366 ... ok
292/367 ... ok
292/368 ... ok
292/369 ... ok
63/371 ... ok
63/372 ... ok
63/373 ... ok
292/374 ... ok
2/375 ... ok
2/376 ... ok
287/377 ... ok
295/410 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
56/1 ... yes
2/2 ... yes
17/4 ... yes
57/37 ... yes
57/38 ... yes
63/40 ... yes
63/41 ... yes
63/42 ... yes
63/43 ... yes
63/44 ... yes
67/45 ... yes
63/46 ... yes
63/47 ... yes
7/48 ... yes
63/49 ... yes
63/50 ... yes
2/52 ... yes
69/53 ... yes
70/54 ... yes
9/56 ... yes
2/59 ... yes
9/60 ... yes
67/61 ... yes
7/62 ... yes
67/63 ... yes
71/64 ... yes
67/65 ... yes
75/66 ... yes
72/67 ... yes
75/68 ... yes
75/69 ... yes
75/70 ... yes
63/71 ... yes
63/72 ... yes
70/73 ... yes
67/74 ... yes
67/75 ... yes
111/109 ... yes
63/142 ... yes
63/143 ... yes
54/176 ... yes
11/177 ... yes
11/178 ... yes
11/179 ... yes
11/180 ... yes
11/181 ... yes
78/214 ... yes
9/247 ... yes
62/250 ... yes
70/283 ... yes
280/284 ... yes
8/317 ... yes
67/350 ... yes
70/351 ... yes
71/352 ... yes
283/353 ... yes
63/354 ... yes
287/355 ... yes
287/356 ... yes
54/357 ... yes
287/358 ... yes
289/359 ... yes
55/361 ... yes
287/362 ... yes
287/365 ... yes
292/366 ... yes
292/367 ... yes
292/368 ... yes
292/369 ... yes
63/371 ... yes
63/372 ... yes
63/373 ... yes
292/374 ... yes
2/375 ... yes
2/376 ... yes
287/377 ... yes
295/410 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.0)
Active users: ... 50
Checking GitLab ... Finished

Possible fixes

The Prometheus queries are documented here: https://docs.gitlab.com/ce/user/project/integrations/prometheus.html#gitlab-prometheus-queries

They should be extended to include the namespace, e.g. for CPU:

sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$KUBE_NAMESPACE",environment="$CI_ENVIRONMENT_SLUG"}[2m])) / count(container_cpu_usage_seconds_total{container_name!="POD",namespace="$KUBE_NAMESPACE",environment="$CI_ENVIRONMENT_SLUG"}) * 100

Namespace is already configured in the Kubernetes integration, so it can be taken from there.