rails secrets get overwritten after update to helm release 5.8.2
Summary
We have several gitlab installations (omnibus and helm). Only on our helm chart deployments we recently saw that the gitlab-shared-secrets
job regenerates and overwrite our rails-secrets. However this only takes effect after the containers are restarted and load the new keys. The effect is that none the encrypted fields in the database can be loaded / viewed and GitLab is more or less unusable / down (500 Internal Server Error).
We reconstructed the rails secrets from a backup, but as soon as we trigger a new helm run the gitlab-shared-secrets
job will overwrite the secret values again.
EDIT: possible ways to recover the rails-secret
If you lost access to your rails-secrets due to this bug and you don´t have a backup, there are a few things you could try:
- If not all containers restarted during the roll-out (or afterwards) you might be able to recover the secrets from a container that still has the old secrets mounted. You could try the toolbox container and check the content of its
/srv/gitlab/config/secrets.yml
. - If you have a backup of the Kubernetes Etcd you might be able to recover the secret object from a backup of the etcd before the upgrade.
Steps to reproduce
Not fully clear yet. We have seen this on multiple independent helm chart deployments.
We assume that the problem can be reproduced as follows:
- Update a gitLab helm deployment to 5.8.2 (in our case it was from 5.7.3 to 5.8.2)
- Make a dummy config change to the values that will trigger a rerun of the
gitlab-shared-secrets
job. - Tail the logs of the job and watch out for the line
secret/gitlab-rails-secret configured
Configuration used
We are using flux to deploy the HelmRelease:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: gitlab
namespace: ${customer}
spec:
releaseName: gitlab
chart:
spec:
chart: gitlab
version: "5.8.2"
sourceRef:
kind: HelmRepository
name: gitlab
namespace: flux-system
interval: 10m
timeout: 30m
install:
remediation:
retries: 3
valuesFrom:
- kind: ConfigMap
name: gitlab-runner-config
valuesKey: data
targetPath: gitlab-runner.runners.config
optional: false
values:
internalMinioEnabled: false
global:
nodeSelector:
customer: ${gitlab_nodeselector}
common:
labels: {}
image: {}
operator:
enabled: false
rollout:
autoPause: true
pod:
labels: {}
edition: ee
application:
create: false
links: []
allowClusterRoles: true
hosts:
domain: ${gitlab_domain}
hostSuffix:
https: true
externalIP:
ssh: ~
gitlab: {}
minio: {}
registry: {}
tls: {}
smartcard: {}
kas: {}
pages: {}
ingress:
configureCertmanager: false
class: ${customer}
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.middlewares: ${customer}-redirect-to-https@kubernetescrd
enabled: true
tls:
secretName: gitlab-tls
path: /
pathType: Prefix
gitlab:
license: {}
initialRootPassword: {}
psql:
connectTimeout:
keepalives:
keepalivesIdle:
keepalivesInterval:
keepalivesCount:
tcpUserTimeout:
password: {}
redis:
password:
enabled: true
gitaly:
enabled: true
authToken: {}
internal:
names: ["default"]
external: []
service:
name: gitaly
type: ClusterIP
externalPort: 8075
internalPort: 8075
tls:
externalPort: 8076
internalPort: 8076
tls:
enabled: false
praefect:
enabled: false
replaceInternalGitaly: true
authToken: {}
autoMigrate: true
dbSecret: {}
virtualStorages:
- name: default
gitalyReplicas: 3
maxUnavailable: 1
psql:
sslMode: "disable"
service:
name: praefect
type: ClusterIP
externalPort: 8075
internalPort: 8075
tls:
externalPort: 8076
internalPort: 8076
tls:
enabled: false
minio:
enabled: false
credentials: {}
ingress:
tls:
secretName: minio-tls-secret
grafana:
enabled: false
appConfig:
backup:
bucket: ${gitlab_backups_bucket}
tmpBucket: ${gitlab_backups_tmp_bucket}
registry:
bucket: ${gitlab_registry_bucket}
enableUsagePing: true
enableSeatLink: true
enableImpersonation:
applicationSettingsCacheSeconds: 60
defaultCanCreateGroup: true
usernameChangingEnabled: true
issueClosingPattern:
defaultTheme:
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
webhookTimeout:
maxRequestDurationSeconds:
cron_jobs: {}
contentSecurityPolicy:
enabled: true
report_only: false
directives:
default_src: "'self'"
script_src: "'self' 'unsafe-inline' 'unsafe-eval' https://www.recaptcha.net https://apis.google.com"
frame_ancestors: "'self'"
frame_src: "'self' https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com"
img_src: "* data: blob:"
style_src: "'self' 'unsafe-inline'"
gravatar:
plainUrl:
sslUrl:
extra:
googleAnalyticsId:
matomoUrl:
matomoSiteId:
matomoDisableCookies:
object_store:
enabled: false
proxy_download: true
storage_options: {}
connection: {}
lfs:
bucket: ${gitlab_lfs_bucket}
connection: # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: storage-config
key: config
artifacts:
bucket: ${gitlab_artifacts_bucket}
enabled: true
proxy_download: true
connection: # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: storage-config
key: config
uploads:
bucket: ${gitlab_uploads_bucket}
enabled: true
proxy_download: true
connection: # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: storage-config
key: config
packages:
bucket: ${gitlab_packages_bucket}
enabled: true
proxy_download: true
connection: # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: storage-config
key: config
externalDiffs:
enabled: false
when:
proxy_download: true
bucket: gitlab-mr-diffs
connection: {}
terraformState:
enabled: false
bucket: gitlab-terraform-state
connection: {}
dependencyProxy:
enabled: false
proxy_download: true
bucket: gitlab-dependency-proxy
connection: {}
pseudonymizer:
configMap:
bucket: gitlab-pseudo
connection: {}
incomingEmail:
enabled: false
address: ""
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
user: ""
password:
secret: ""
key: password
expungeDeleted: false
logger:
logPath: "/dev/stdout"
mailbox: inbox
idleTimeout: 60
inboxMethod: "imap"
clientSecret:
key: secret
pollInterval: 60
serviceDeskEmail:
enabled: false
address: ""
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
user: ""
password:
secret: ""
key: password
expungeDeleted: false
logger:
logPath: "/dev/stdout"
mailbox: inbox
idleTimeout: 60
inboxMethod: "imap"
clientSecret:
key: secret
pollInterval: 60
ldap:
preventSignin: false
servers: {}
gitlab_kas: {}
omniauth:
enabled: true
autoSignInWithProvider:
syncProfileFromProvider: []
syncProfileAttributes: ["email"]
allowSingleSignOn: ["saml"]
blockAutoCreatedUsers: true
autoLinkLdapUser: false
autoLinkSamlUser: true
autoLinkUser: []
externalProviders: []
allowBypassTwoFactor: ["saml"]
providers:
- secret: gitlab-omniauth-providers
key: saml
sentry:
enabled: false
dsn:
clientside_dsn:
environment:
smartcard:
enabled: false
CASecret:
clientCertificateRequiredHost:
sanExtensions: false
requiredForGitAccess: false
sidekiq:
routingRules: []
initialDefaults:
signupEnabled: false
oauth:
gitlab-pages: {}
geo:
enabled: false
role: primary
nodeName:
psql:
password: {}
registry:
replication:
enabled: false
primaryApiUrl:
kas:
enabled: false
shell:
authToken: {}
hostKeys: {}
tcp:
proxyProtocol: false
railsSecrets: {}
rails:
bootsnap:
enabled: true
registry:
certificate: {}
httpSecret: {}
notificationSecret: {}
notifications: {}
pages:
enabled: true
accessControl: false
path:
host:
port:
https:
externalHttp: []
externalHttps: []
artifactsServer: true
objectStore:
bucket: ${gitlab_pages_bucket}
enabled: true
connection: # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: storage-config
key: config
apiSecret: {}
authSecret: {}
runner:
registrationToken: {}
smtp:
enabled: false
address: smtp.mailgun.org
port: 2525
user_name: ""
password:
secret: ""
key: password
authentication: "plain"
starttls_auto: false
openssl_verify_mode: "peer"
pool: false
email:
from: ""
display_name: GitLab
reply_to: ""
subject_suffix: ""
smime:
enabled: false
secretName: ""
keyName: "tls.key"
certName: "tls.crt"
time_zone: "Europe/Berlin"
service:
labels: {}
annotations: {}
deployment:
annotations: {}
antiAffinity: soft
workhorse:
serviceName: webservice-default
webservice:
workerTimeout: 60
securityContext:
runAsUser: 65534
fsGroup: 65534
busybox:
image:
repository: registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox
tag: latest
serviceAccount:
enabled: false
create: true
annotations: {}
tracing:
connection:
string: ""
urlTemplate: ""
extraEnv: {}
upgradeCheck:
enabled: true
image: {}
securityContext:
runAsUser: 65534
fsGroup: 65534
tolerations: []
resources:
requests:
cpu: 50m
certmanager:
install: false
installCRDs: false
nameOverride: certmanager
rbac:
create: false
nginx-ingress:
enabled: false
prometheus:
install: false
redis:
install: true
existingSecret: gitlab-redis-secret
existingSecretKey: redis-password
usePasswordFile: true
cluster:
enabled: false
metrics:
enabled: true
master:
resources:
limits:
memory: 128Mi
cpu: 100m
requests:
memory: 128Mi
cpu: 100m
nodeSelector:
customer: ${gitlab_nodeselector}
persistence:
storageClass: ${redis_storage_class}
postgresql:
postgresqlUsername: gitlab
#postgresqlPostgresPassword:
install: true
postgresqlDatabase: gitlabhq_production
image:
tag: 12
usePasswordFile: true
existingSecret: postgresql-postgres-password
existingSecretKey: password
#initdbScriptsConfigMap: 'bogus'
primary:
resources:
limits:
memory: 512Mi
cpu: 500m
requests:
memory: 256Mi
cpu: 250m
master:
nodeSelector:
customer: ${gitlab_nodeselector}
resources:
limits:
memory: 512Mi
cpu: 500m
requests:
memory: 256Mi
cpu: 250m
extraVolumeMounts:
- name: custom-init-scripts
mountPath: /docker-entrypoint-preinitdb.d/init_revision.sh
subPath: init_revision.sh
podAnnotations:
postgresql.gitlab/init-revision: "1"
metrics:
enabled: true
persistence:
storageClass: ${postgres_storage_class}
size: ${postgres_volume_size}
shared-secrets:
enabled: true
rbac:
create: true
selfsign:
keyAlgorithm: "rsa"
keySize: "4096"
expiry: "3650d"
caSubject: "GitLab Helm Chart"
env: production
serviceAccount:
enabled: true
create: true
name:
resources:
requests:
cpu: 50m
securityContext:
runAsUser: 65534
fsGroup: 65534
tolerations: []
podLabels: {}
annotations: {}
gitlab:
gitlab-exporter:
resources:
limits:
cpu: 100m
memory: 100M
requests:
cpu: 100m
memory: 100M
gitaly:
persistence:
storageClass: ${gitaly_storage_class}
resources:
limits:
cpu: ${gitaly_resource_limits_cpu}
memory: ${gitaly_resource_limits_memory}
requests:
cpu: ${gitaly_resource_requests_cpu}
memory: ${gitaly_resource_requests_memory}
toolbox:
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 100Mi
replicas: 1
antiAffinityLabels:
matchLabels:
app: "gitaly"
backups:
cron:
enabled: true
schedule: 0 0 * * *
objectStorage:
config:
secret: runner-s3-config
key: config
webservice:
minReplicas: ${webservice_min_replicas}
maxReplicas: ${webservice_max_replicas}
ingress:
tls:
secretName: webservice-tls-secret
resources:
limits:
cpu: ${webservice_resource_limits_cpu}
memory: ${webservice_resource_limits_memory}
requests:
cpu: ${webservice_resource_requests_cpu}
memory: ${webservice_resource_requests_memory}
extraVolumes: |
- name: gitlab-robots
configMap:
name: gitlab-robots
items:
- key: robots.txt
path: public/robots.txt
extraVolumeMounts: |
- name: gitlab-robots
mountPath: /srv/gitlab/public/robots.txt
subPath: public/robots.txt
readOnly: true
gitlab-shell:
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
minReplicas: ${gitlab_shell_min_replicas}
maxReplicas: ${gitlab_shell_max_replicas}
sidekiq:
minReplicas: ${sidekiq_min_replicas}
maxReplicas: ${sidekiq_max_replicas}
resources:
limits:
cpu: ${sidekiq_resource_limits_cpu}
memory: ${sidekiq_resource_limits_memory}
requests:
cpu: ${sidekiq_resource_requests_cpu}
memory: ${sidekiq_resource_requests_memory}
gitlab-pages:
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 100Mi
registry:
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 100Mi
storage:
secret: s3-registry-config
key: config
ingress:
tls:
secretName: registry-tls-secret
hpa:
minReplicas: ${registry_min_replicas}
maxReplicas: ${registry_max_replicas}
gitlab-runner:
install: false
nodeSelector:
customer: ${gitlab_nodeselector}
Current behavior
gitlab-shared-secrets
job overwrites the rails-secrets on every helm run rendering GitLab unusable.
Expected behavior
The rails-secrets should never be overwritten.
Versions
- Chart: 5.8.2
- Platform:
- Cloud: other
- Self-hosted: K8s
- flux: v0.26.2
- helm-controller: v0.15.0
- kustomize-controller: v0.19.1
- notification-controller: v0.20.1
- source-controller: v0.20.1
Relevant logs
(Please provide any relevate log snippets you have collected, using code blocks (```) to format) Logs of the shared-secrets job running after helm deployed a config change:
secret "gitlab-gitlab-initial-root-password" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-initial-root-password not labeled
secret/gitlab-gitlab-initial-root-password not labeled
secret "gitlab-redis-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-redis-secret not labeled
secret/gitlab-redis-secret not labeled
secret "gitlab-postgresql-password" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-postgresql-password not labeled
secret/gitlab-postgresql-password not labeled
secret "gitlab-gitlab-shell-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-shell-secret not labeled
secret/gitlab-gitlab-shell-secret not labeled
secret "gitlab-gitaly-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitaly-secret not labeled
secret/gitlab-gitaly-secret not labeled
secret "gitlab-gitlab-runner-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-runner-secret not labeled
secret/gitlab-gitlab-runner-secret not labeled
secret "gitlab-gitlab-pages-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-pages-secret not labeled
secret/gitlab-gitlab-pages-secret not labeled
Generating a RSA private key
..............................++++
.................++++
writing new private key to 'certs/registry-example-com.key'
-----
secret "gitlab-registry-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-secret not labeled
secret/gitlab-registry-secret not labeled
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Error: Parsing expression: Lexer error: could not match text starting at 1:1 failing at 1:4.
unmatched text: "rea"
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
.........................................+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus (2 primes)
...................................+++++
.........................................+++++
e is 65537 (0x010001)
secret/gitlab-rails-secret configured
label "app.kubernetes.io/name" not found.
secret/gitlab-rails-secret not labeled
secret/gitlab-rails-secret not labeled
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
secret "gitlab-gitlab-shell-host-keys" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-shell-host-keys not labeled
secret/gitlab-gitlab-shell-host-keys not labeled
secret "gitlab-gitlab-workhorse-secret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-gitlab-workhorse-secret not labeled
secret/gitlab-gitlab-workhorse-secret not labeled
secret "gitlab-registry-httpsecret" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-httpsecret not labeled
secret/gitlab-registry-httpsecret not labeled
secret "gitlab-registry-notification" already exists.
label "app.kubernetes.io/name" not found.
secret/gitlab-registry-notification not labeled
secret/gitlab-registry-notification not labeled