Problem with alerts not working from metric page with gitlab installed prometheus on kubernetes 1.16

Summary

Opening this issue as suggested by @rpereira2 I have a kubernetes 1.16 gitlab managed cluster in a group. Prometheus is installed from gitlab and then I have upgraded manually the kube-state-metrics to 1.9.7 to have prometheus work with kubernetes 1.16 (#213372 (closed))

I have a project with some code that deploy correctly on kubernetes. I see health metric on the kubernates cluster. I see cpu/memory/ingress metrics on the operations/metrics page. I have enabled "generic" alert management on settings/operations.

I try to setup a cpu alert on the metric page. When i setup the alert (or delete it) I see that the gitlab/kubernetes integration stop to work for a variable time (between 10 seconds and 1-2 minutes). During this time the cluster health page and project metrics page is the same of when prometheus is not installed (and ask to install it).

After some time I see the metrics back on the metrics page with the alert setup. I then continue consuming some CPU on the POD to generate the alert:

No alert is generated on the alert page and it also say to "enable alerts".

Alerts are enabled:

Not sure if the bug is only with kubernetes 1.16 but it seems something related to alerts.

Steps to reproduce

Create/connect a kubernetes (1.16) cluster. Install prometheus. Upgrade prometheus to 11.6.0 or kube-state-metrics to 1.9.7 Enable alerting. Deploy something on kubernetes. Setup an alert on the metric page. Integration stop to work for some time and no alert is generated when it should.

What is the current bug behavior?

No alert are generated.

What is the expected correct behavior?

Alert should be generated on the project operations/alert page.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         RedHatEnterpriseServer 7.8
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.27.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.2.1-ee
Revision:       447f057d946
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.7
URL:            https://itromgit01.q8int.com
HTTP Clone URL: https://itromgit01.q8int.com/some-group/some-project.git
SSH Clone URL:  git@itromgit01.q8int.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: gitlab, github, saml

GitLab Shell
Version:        13.3.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking incoming
yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 50 users of 100 limit.
Server: ldapsecondary
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 50 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/3 ... yes
3/8 ... yes
3/13 ... yes
3/14 ... yes
3/15 ... yes
3/16 ... yes
3/17 ... yes
3/21 ... yes
8/22 ... yes
15/30 ... yes
15/34 ... yes
21/41 ... yes
21/42 ... yes
21/43 ... yes
21/44 ... yes
21/45 ... yes
15/48 ... yes
6/52 ... yes
37/54 ... yes
37/55 ... yes
15/56 ... yes
15/57 ... yes
15/60 ... yes
3/65 ... yes
3/66 ... yes
3/67 ... yes
6/68 ... yes
3/70 ... yes
15/71 ... yes
3/73 ... yes
3/74 ... yes
8/75 ... yes
8/76 ... yes
8/77 ... yes
3/78 ... yes
47/79 ... yes
47/80 ... yes
47/81 ... yes
52/82 ... yes
50/84 ... yes
55/85 ... yes
56/86 ... yes
50/88 ... yes
50/89 ... yes
57/90 ... yes
57/91 ... yes
6/94 ... yes
47/95 ... yes
62/96 ... yes
59/97 ... yes
6/98 ... yes
63/99 ... yes
63/100 ... yes
63/101 ... yes
68/105 ... yes
69/106 ... yes
71/109 ... yes
71/110 ... yes
71/112 ... yes
71/115 ... yes
6/116 ... yes
47/117 ... yes
47/118 ... yes
47/119 ... yes
71/123 ... yes
71/124 ... yes
71/125 ... yes
71/126 ... yes
71/127 ... yes
71/128 ... yes
71/129 ... yes
71/130 ... yes
71/131 ... yes
71/132 ... yes
71/133 ... yes
71/134 ... yes
71/135 ... yes
71/136 ... yes
71/137 ... yes
71/138 ... yes
71/139 ... yes
71/140 ... yes
71/141 ... yes
71/142 ... yes
71/144 ... yes
71/145 ... yes
71/146 ... yes
71/147 ... yes
71/148 ... yes
71/149 ... yes
71/150 ... yes
71/151 ... yes
71/152 ... yes
71/153 ... yes
72/154 ... yes
71/155 ... yes
71/156 ... yes
74/157 ... yes
70/159 ... yes
70/160 ... yes
70/161 ... yes
70/162 ... yes
70/163 ... yes
70/164 ... yes
70/165 ... yes
70/166 ... yes
70/167 ... yes
70/168 ... yes
70/169 ... yes
70/170 ... yes
70/171 ... yes
70/172 ... yes
70/173 ... yes
50/174 ... yes
50/175 ... yes
71/177 ... yes
71/178 ... yes
71/179 ... yes
64/180 ... yes
71/181 ... yes
50/182 ... yes
71/183 ... yes
70/184 ... yes
70/185 ... yes
71/186 ... yes
70/188 ... yes
50/189 ... yes
47/190 ... yes
64/191 ... yes
70/192 ... yes
71/193 ... yes
71/194 ... yes
71/195 ... yes
71/196 ... yes
71/197 ... yes
71/198 ... yes
71/199 ... yes
71/200 ... yes
71/201 ... yes
71/202 ... yes
71/203 ... yes
71/204 ... yes
71/205 ... yes
71/206 ... yes
71/207 ... yes
71/208 ... yes
71/209 ... yes
71/210 ... yes
71/211 ... yes
71/212 ... yes
71/213 ... yes
71/214 ... yes
71/215 ... yes
71/216 ... yes
71/217 ... yes
71/218 ... yes
71/219 ... yes
71/220 ... yes
71/221 ... yes
71/222 ... yes
71/223 ... yes
54/224 ... yes
71/225 ... yes
71/226 ... yes
47/227 ... yes
82/228 ... yes
47/229 ... yes
47/231 ... yes
71/233 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.27.0)
Git user has default SSH configuration? ... yes
Active users: ... 17
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes