## Summary `gitlab-ctl reconfigure` inconsistently handles secret token updates from `gitlab.rb` to `gitlab-secrets.json`. When both `gitlab_shell['secret_token']` and `gitaly['gitaly_token']` are defined in `gitlab.rb`, only the `gitlab_shell` token overwrites existing values in `gitlab-secrets.json`, while the `gitaly` token does not, causing a mismatch between the two secrets. ## Steps to reproduce 1. Set up a GitLab instance with existing values in `/etc/gitlab/gitlab-secrets.json` for both: - `gitlab_shell.secret_token` - `gitaly.gitlab_secret` 2. Edit `/etc/gitlab/gitlab.rb` and set both tokens to **new matching values**: ```ruby gitlab_shell['secret_token'] = 'new_matching_secret_token' gitaly['gitaly_token'] = 'new_matching_secret_token' ``` 3. Run: ```bash gitlab-ctl reconfigure ``` 4. Inspect `/etc/gitlab/gitlab-secrets.json` ## Expected behavior Both secrets in `gitlab-secrets.json` should be updated to match the values specified in `gitlab.rb`: - `gitlab_shell.secret_token` → updated to `new_matching_secret_token` - `gitaly.gitlab_secret` → updated to `new_matching_secret_token` ## Actual behavior Only one secret is updated: - ✅ `gitlab_shell.secret_token` → **correctly updated** to `new_matching_secret_token` - ❌ `gitaly.gitlab_secret` → **retains old value**, not updated from `gitlab.rb` This creates a mismatch where the two secrets that should be identical are now different. ## Additional context **Workaround discovered:** If both the `gitlab_shell` and `gitaly` keys are completely removed from `gitlab-secrets.json` before running reconfigure, then both values are successfully sourced from `gitlab.rb`. This confirms the inconsistency only occurs when **overwriting existing values**. **Why this matters:** When migrating Gitaly configurations or synchronizing secrets across nodes, administrators expect that values explicitly set in `gitlab.rb` will take precedence and be written to `gitlab-secrets.json` during reconfigure. The current inconsistent behavior makes it difficult to: - Rotate secrets reliably - Synchronize configurations across distributed Gitaly setups - Troubleshoot authentication issues between GitLab Shell and Gitaly ## Environment - **GitLab version:** 18.4.4 (confirmed, likely affects earlier versions) - **Installation method:** Linux package (Omnibus) - **Configuration:** Standalone or distributed Gitaly setup ## Relevant documentation - [Configure Gitaly - About the Gitaly token](https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token) - [Configure Praefect - Requirements](https://docs.gitlab.com/ee/administration/gitaly/praefect/configure.html#requirements) ## Possible causes The reconfigure logic appears to handle `gitlab_shell['secret_token']` and `gitaly['gitaly_token']` differently when writing to `gitlab-secrets.json`: - `gitlab_shell['secret_token']` → always overwrites existing `gitlab-secrets.json` values - `gitaly['gitaly_token']` → only writes if the key doesn't already exist in `gitlab-secrets.json` This suggests different code paths or precedence rules for these two related secrets.