Grafana session data is not shared on "Multiple application servers" High-Availability installations

Summary

Grafana session data is not shared on "Multiple application servers" High-Availability installations Therefore Grafana cannot be used at all on this type of deployment.

Steps to reproduce

Setup multiple application servers with non-sticky ingress routing between them, try to login to grafana. (I suspect the bug will present with any shared data backend, possibly even DRBD; I exercised this issue specifically with nfs shared filesystem, and postgres shared database type deployment.) In my brief testing, I observed grafana reject (valid) login credentials repeatedly, and occasionally I would get to the dashboard which would present authentication errors on subsequent ajax requests and invalidate the session.

Example Project

N/A, Issue impacts gitlab service not project hosting.

What is the current bug behavior?

Grafana stores session data on a single application server, causing a random session invalidation loop when accessed over non-sticky load balancing.

What is the expected correct behavior?

Grafana should accept admin change-password prompt submissions and be capable of updating that credential information on all application servers via a shared data backend. The admin password reset functionality should also be capable of updating credential information for all application servers.

Lastly, Grafana should accept valid credentials and operate with a valid session access a multi-application-server deployment, due to session data and credential data shared between the application servers as aforementioned.

Relevant logs and/or screenshots

# /opt/gitlab/embedded/bin/grafana-cli admin reset-admin-password   --homepath /var/opt/gitlab/grafana admin admin
t=2019-08-12T17:29:45+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2019-08-12T17:29:45+0000 lvl=info msg="Starting DB migration" logger=migrator

Admin password changed successfully ✔

The tell-tale message dbtype=sqlite3 indicates grafana is using file based storage in the form of a sqlite database file, from observations about application behaviour it seems clear that this data is not shared between application servers in the default configuration, despite configuration documentation of how to achieve active-active deployments with nfs as a shared filesystem.

Output of checks

Results of GitLab environment info

(url censored)

Expand for output related to GitLab environment info 

gitlab-rake gitlab:env:info


System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.21.0
Sidekiq Version:5.2.7
Go Version:	go1.10.4 linux/amd64


GitLab information
Version:	12.0.3-ee
Revision:	1b1872f9d93
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.6.9
URL:		https://gitlab.<example.org>
HTTP Clone URL:	https://gitlab.<example.org>/some-group/some-project.git
SSH Clone URL:	git@gitlab.<example.org>:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml


GitLab Shell
Version:	9.3.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories
  • storage1: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 

gitlab-rake gitlab:check SANITIZE=true


Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK
storage1 ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...

Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 60
Elasticsearch version 5.6 - 6.x? ... yes (6.3.1)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Likely grafana could be configured to use a database (schema) on the existing gitlab sql database server? https://grafana.com/docs/tutorials/ha_setup/

Maybe sqlite over nfs would "work" but I have my doubts that it would work flawlessly.