2025-10-02: K8s deployment failed on staging-canary

K8s deployment failed on staging-canary (Severity 3 (Medium))

The kubernetes deployment failed on staging-canary https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/20528333, preventing code changes from being deployed.

All the webservice pods are in CrashLoopBackOff

The stacktrace showed it was related to an encryption error:

{"timestamp":"2025-10-02T16:46:28.277Z","pid":38,"message":"! Unable to load application: ActiveRecord::Encryption::Errors::Decryption: ActiveRecord::Encryption::Errors::Decryption"} bundler: failed to load command: puma (/srv/gitlab/vendor/bundle/ruby/3.2.0/bin/puma) /srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/encryption/encryptor.rb:58:in `rescue in decrypt':

The culprit MR gitlab-org/gitlab!206770 (merged) was reverted gitlab-org/gitlab!207316 (merged)

---

This ticket was created to track INC-4432, by incident.io 🔥