gitlab-ce kubernetes gitlab-managed-apps install-prometheus panic: Unable to create mmap-ed active query log

Gitlab-ce: 13.7.1

k8s: v1.19.6

(both are self-hosted and connected successful)

can't install prometheus from gitlab-managed-apps

panic: Unable to create mmap-ed active query log

NAMESPACE             NAME                                   READY   STATUS    RESTARTS   AGE
gitlab-managed-apps   install-prometheus                     0/1     Error     0          19m
gitlab-managed-apps   runner-gitlab-runner-f6ddc5bd6-4vxg5   1/1     Running   0          3h47m
kube-system           coredns-f9fd979d6-68pc2                1/1     Running   0          33h
kube-system           coredns-f9fd979d6-rhznk                1/1     Running   0          33h
kube-system           csi-nfs-controller-5bd5cb55bc-g2ddd    3/3     Running   0          52m
kube-system           csi-nfs-controller-5bd5cb55bc-z8sf9    3/3     Running   0          52m
kube-system           csi-nfs-node-9d7f4                     3/3     Running   0          51m
kube-system           etcd-k81                               1/1     Running   0          36h
kube-system           kube-apiserver-k81                     1/1     Running   0          36h
kube-system           kube-controller-manager-k81            1/1     Running   0          33h
kube-system           kube-flannel-ds-l4xkx                  1/1     Running   0          33h
kube-system           kube-flannel-ds-rdm4n                  1/1     Running   0          33h
kube-system           kube-proxy-4976l                       1/1     Running   0          33h
kube-system           kube-proxy-g2fn4                       1/1     Running   0          36h
kube-system           kube-scheduler-k81                     1/1     Running   0          36h
kube-system           tiller-deploy-f5c865db5-f946b          1/1     Running   0          86m

I have set up self-hosted gitlab-ce and connected with self-hosted kubernetes cluster in my local network. (gitlab-ce has an external url with nginx-proxy )

kubernetes has configured with flannel network driver and nfs-csi driver for dynamic NFS volumes.

k8s is managed by gitlab.

As I see, the PersistentVolumeClaim was created with PersistentVolume in my NFS storage class. But, when prometheus-server tries to start it can't write to that PV and it crashes the installation.

Logs of prometheus-server container:

(k8s_prometheus-server_prometheus-prometheus-server-59596d7f4-g9zpb_gitlab-managed-apps_7436bc2e-2700-4684-a839-0007c370524b_3):

Expand for output container logs

level=info ts=2020-12-25T03:16:58.334Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.2, branch=HEAD, revision=d9613e5c466c6e9de548c4dae1b9aabf9aaf7c57)"
level=info ts=2020-12-25T03:16:58.335Z caller=main.go:331 build_context="(go=go1.13.5, user=root@688433cf4ff7, date=20200106-14:50:51)"
level=info ts=2020-12-25T03:16:58.335Z caller=main.go:332 host_details="(Linux 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x86_64 prometheus-prometheus-server-59596d7f4-g9zpb (none))"
level=info ts=2020-12-25T03:16:58.335Z caller=main.go:333 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-12-25T03:16:58.335Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-12-25T03:16:58.337Z caller=query_logger.go:85 component=activeQueryTracker msg="Error opening query log file" file=/data/queries.active err="open /data/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fffa21b6c1b, 0x5, 0x14, 0x2c635a0, 0xc00060f380, 0x2c635a0)
/app/promql/query_logger.go:115 +0x48c
main.main()
/app/cmd/prometheus/main.go:362 +0x5229

Logs of install-prometheus POD:

Expand for output pod logs

[kube] 2020/12/25 03:20:55 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[kube] 2020/12/25 03:20:57 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[kube] 2020/12/25 03:20:59 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[kube] 2020/12/25 03:21:01 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[kube] 2020/12/25 03:21:03 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[kube] 2020/12/25 03:21:03 Deployment is not ready: gitlab-managed-apps/prometheus-prometheus-server
[tiller] 2020/12/25 03:21:03 warning: Release "prometheus" failed: timed out waiting for the condition
[storage] 2020/12/25 03:21:03 updating release "prometheus.v1"
[tiller] 2020/12/25 03:21:03 failed install perform step: release prometheus failed: timed out waiting for the condition
INSTALL FAILED
PURGING CHART
Error: release prometheus failed: timed out waiting for the condition

I think it my a problem with "accessModes: ReadWriteOnce" ... So k8s creates PV for prometheus, but the prometheus server can't to write to that volume.

Can you help me ?

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.1
Redis Version:	5.0.9
Git Version:	2.29.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.7.1
Revision:	c97c8073a0e
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		http://X.org
HTTP Clone URL:	http://X.org/some-group/some-project.git
SSH Clone URL:	git@X.org:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.14.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git