Skip to content

When using database decomposition, the configuration is not idempotent between helm executions

Summary

Between executions of helm we're noticing changes associated with the database configuration when no configuration change actually exists. Note that during local testing, it appears changes are detected about 3/5's of the time. While this would normally be harmless, this is causing needless rotations of deployments when nothing actually changes. The change is the configuration order inside of the ConfigMap which causes a shasum change to the deployment.

Steps to reproduce

Enable the use of database decomposition. For .com we do something like this (simplified):

CLICK MEEEEEE 
  psql:
    host: pgbouncer.int.gprd.gitlab.net
    port: 6432
    load_balancing:
      discover:
        nameserver: consul-gl-consul-dns.consul.svc.cluster.local
        record: db-replica.service.consul.
    ci:
      host: pgbouncer-ci.int.gprd.gitlab.net
      port: 6432
      load_balancing:
        discover:
          nameserver: consul-gl-consul-dns.consul.svc.cluster.local
          record: ci-db-replica.service.consul.
    embedding:
      host: pgbouncer-embedding.int.gprd.gitlab.net
      port: 6432
      load_balancing: false
      username: gitlab-superuser
      password:
        secret: embedding-postgresql-gprd-password-v1

Source

Proceed to send a helm diff or helm upgrade and observe that, on occasion, the ConfigMap which holds the webservice and sidekiq database configs change.

We wind up seeing diffs like such:

Unfortunate diff inside 
default, a-webservice, ConfigMap (v1) has changed:
  # Source: gitlab/charts/gitlab/charts/webservice/templates/configmap.yml
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: a-webservice
    namespace: default
    labels:
      app: webservice
      chart: webservice-7.1.2
      release: a
      heritage: Helm

  data:
    installation_type: |
      gitlab-helm-chart
    database.yml.erb: |

      production:
        main:
          adapter: postgresql
          encoding: unicode
          database: gitlabhq_production
          username: gitlab
          password: <%= File.read('/etc/gitlab/postgres/psql-password-main').strip.to_json %>
          host: "pgbouncer.int.gprd.gitlab.net"
          port: 6432
          connect_timeout:
          keepalives:
          keepalives_idle:
          keepalives_interval:
          keepalives_count:
          tcp_user_timeout:
          application_name:
          prepared_statements: false
          database_tasks: true
          load_balancing:
            discover:
              nameserver: consul-gl-consul-dns.consul.svc.cluster.local
              record: "db-replica.service.consul."

-       ci:
+       embedding:
          adapter: postgresql
          encoding: unicode
          database: gitlabhq_production
-         username: gitlab
-         password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
-         host: "pgbouncer-ci.int.gprd.gitlab.net"
+         username: gitlab-superuser
+         password: <%= File.read('/etc/gitlab/postgres/psql-password-embedding').strip.to_json %>
+         host: "pgbouncer-embedding.int.gprd.gitlab.net"
          port: 6432
          connect_timeout:
          keepalives:
          keepalives_idle:
          keepalives_interval:
          keepalives_count:
          tcp_user_timeout:
          application_name:
          prepared_statements: false
          database_tasks: true
-         load_balancing:
-           discover:
-             nameserver: consul-gl-consul-dns.consul.svc.cluster.local
-             record: "ci-db-replica.service.consul."
+

-       embedding:
+       ci:
          adapter: postgresql
          encoding: unicode
          database: gitlabhq_production
-         username: gitlab-superuser
-         password: <%= File.read('/etc/gitlab/postgres/psql-password-embedding').strip.to_json %>
-         host: "pgbouncer-embedding.int.gprd.gitlab.net"
+         username: gitlab
+         password: <%= File.read('/etc/gitlab/postgres/psql-password-ci').strip.to_json %>
+         host: "pgbouncer-ci.int.gprd.gitlab.net"
          port: 6432
          connect_timeout:
          keepalives:
          keepalives_idle:
          keepalives_interval:
          keepalives_count:
          tcp_user_timeout:
          application_name:
          prepared_statements: false
          database_tasks: true
-
+         load_balancing:
+           discover:
+             nameserver: consul-gl-consul-dns.consul.svc.cluster.local
+             record: "ci-db-replica.service.consul."
<SNIP>

In the above case the ci and embedded blocks simply swap order.

Expected behavior

Helm should not attempt to change the ConfigMap at all. I suspect we need to ask our templating to encourage it to maintain order to prevent changes from sometimes showing up.

Versions

  • Chart: latest
  • Platform:
    • Cloud: GKE
    • Self-hosted: Minikube
  • Kubernetes: (kubectl version)
    • Client: 1.25
    • Server: 1.25
  • Helm: (helm version)
    • Client: 3.10