2024-05-10: GSTG One or more Gitaly storages are accessible by multiple filesystem paths
Customer Impact
Current Status
More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.
Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.
Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.
Activity
assigned to @igorwwwwwwwwwwwwwwwwwwww
deployment failure on gstg-cny https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/13884256
There was an error running gitlab-ctl reconfigure: One or more Gitaly storages are accessible by multiple filesystem paths: /var/opt/gitlab/gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal/repositories: default, gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internalThe diff in omnibus includes the following change: https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/blob/9997c41648a6ff05cab9af853d871b705151167f/files/gitlab-cookbooks/gitaly/libraries/gitaly.rb
It's a gitaly 17.0 breaking change that is not compatible with our gstg-cny configuration
added RootCauseNeeded label
-
From @ahmadsherif
It could be considered a bug given that we're running the check on a node where gitaly is not enabled. [...cut...]
A fix for gstg is merged https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/4743. This will not fix the similar upcoming issue for gprd.
There is a proposal in the works to disable the check when Gitaly is not enabled. Although it's arguable because there's no good reason to have multiple storages with the same path anyway. Yet, the deduplication code is only active when Gitaly is enabled, so it makes sense to disable this check there.
Edited by Toon Claes-
From @nolith
the gstg-cny deployment is now running again. We currently expect it to fail on gprd-cny
mentioned in issue on-call-handovers#4935 (closed)
The package with gitlab-org/omnibus-gitlab!7600 (merged) failed to be deployed to production:
fatal: [deploy-01-sv-gprd]: FAILED! => changed=false cmd: - gitlab-ctl - reconfigure delta: '0:00:08.941010' end: '2024-05-10 19:05:19.736368' msg: non-zero return code rc: 1 start: '2024-05-10 19:05:10.795358' stderr: |- /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: already initialized constant SELinuxDistroHelper::REDHAT_RELEASE_FILE /opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:2: warning: previous definition of REDHAT_RELEASE_FILE was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: already initialized constant SELinuxDistroHelper::OS_RELEASE_FILE /opt/gitlab/embedded/cookbooks/package/libraries/helpers/selinux_distro_helper.rb:3: warning: previous definition of OS_RELEASE_FILE was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: already initialized constant SecretsHelper::SECRETS_FILE /opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:4: warning: previous definition of SECRETS_FILE was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: already initialized constant SecretsHelper::SECRETS_FILE_CHEF_ATTR /opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:5: warning: previous definition of SECRETS_FILE_CHEF_ATTR was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: already initialized constant SecretsHelper::SKIP_GENERATE_SECRETS_CHEF_ATTR /opt/gitlab/embedded/cookbooks/package/libraries/helpers/secrets_helper.rb:6: warning: previous definition of SKIP_GENERATE_SECRETS_CHEF_ATTR was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: already initialized constant GitlabCluster::CONFIG_PATH /opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:16: warning: previous definition of CONFIG_PATH was here /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: already initialized constant GitlabCluster::JSON_FILE /opt/gitlab/embedded/cookbooks/package/libraries/gitlab_cluster.rb:17: warning: previous definition of JSON_FILE was here There was an error running gitlab-ctl reconfigure: One or more Gitaly storages are accessible by multiple filesystem paths: /var/opt/gitlab/gitaly-cny-01-stor-gprd.c.gitlab-production.internal/repositories: default, gitaly-cny-01-stor-gprd.c.gitlab-production.internal stderr_lines: <omitted> stdout: |- [2024-05-10T19:05:13+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook) Cinc Client, version 18.3.0 Patents: https://www.chef.io/patents Infra Phase starting [2024-05-10T19:05:13+00:00] INFO: *** Cinc Client 18.3.0 *** [2024-05-10T19:05:13+00:00] INFO: Platform: x86_64-linux [2024-05-10T19:05:13+00:00] INFO: Cinc-client pid: 3450971 [2024-05-10T19:05:14+00:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options [2024-05-10T19:05:14+00:00] INFO: Run List is [recipe[gitlab-ee]] [2024-05-10T19:05:14+00:00] INFO: Run List expands to [gitlab-ee] [2024-05-10T19:05:14+00:00] INFO: Starting Cinc Client Run for deploy-01-sv-gprd.c.gitlab-production.internal [2024-05-10T19:05:14+00:00] INFO: Running start handlers [2024-05-10T19:05:14+00:00] INFO: Start handlers complete. Resolving cookbooks for run list: ["gitlab-ee"] [2024-05-10T19:05:15+00:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, patroni@0.1.0, pgbouncer@0.1.0, spamcheck@0.1.0, runit@5.1.7, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0] Synchronizing cookbooks: - gitlab-ee (0.0.1) - consul (0.1.0) - patroni (0.1.0) - pgbouncer (0.1.0) - runit (5.1.7) - logrotate (0.1.0) - redis (0.1.0) - postgresql (0.1.0) - gitlab (0.0.1) - spamcheck (0.1.0) - registry (0.1.0) - gitaly (0.1.0) - praefect (0.1.0) - gitlab-kas (0.1.0) - mattermost (0.1.0) - package (0.1.0) - letsencrypt (0.1.0) - nginx (0.1.0) - gitlab-pages (0.1.0) - acme (4.1.6) - crond (0.1.0) - monitoring (0.1.0) Installing cookbook gem dependencies: Compiling cookbooks... Loading Cinc Auditor profile files: Loading Cinc Auditor input files: Loading Cinc Auditor waiver files: [2024-05-10T19:05:17+00:00] INFO: Generating default secrets [2024-05-10T19:05:19+00:00] INFO: Generating /etc/gitlab/gitlab-secrets.json file ================================================================================ Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb ================================================================================ RuntimeError ------------ One or more Gitaly storages are accessible by multiple filesystem paths: /var/opt/gitlab/gitaly-cny-01-stor-gprd.c.gitlab-production.internal/repositories: default, gitaly-cny-01-stor-gprd.c.gitlab-production.internal Cookbook Trace: (most recent call first) ---------------------------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:187:in `check_duplicate_storage_paths' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:30:in `parse_variables' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:208:in `block in generate_config' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:206:in `each' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:206:in `generate_config' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:35:in `from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:27:in `from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:22:in `from_file' Relevant File Content: ---------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb: 180: return if realpath_duplicates.empty? 181: 182: output = realpath_duplicates.map do |realpath, entries| 183: names = entries.map { |s| s[:name] }.join(', ') 184: "#{realpath}: #{names}" 185: end 186: 187>> raise "One or more Gitaly storages are accessible by multiple filesystem paths:\n #{output.join('\n ')}" 188: end 189: 190: private 191: 192: def user_config 193: Gitlab['gitaly'] 194: end 195: 196: def package_default System Info: ------------ chef_version=18.3.0 platform=ubuntu platform_version=20.04 ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client Running handlers: [2024-05-10T19:05:19+00:00] ERROR: Running exception handlers Running handlers complete [2024-05-10T19:05:19+00:00] ERROR: Exception handlers complete Infra Phase failed. 0 resources updated in 06 seconds [2024-05-10T19:05:19+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2024-05-10T19:05:19+00:00] FATAL: --------------------------------------------------------------------------------------- [2024-05-10T19:05:19+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2024-05-10T19:05:19+00:00] FATAL: --------------------------------------------------------------------------------------- [2024-05-10T19:05:19+00:00] FATAL: RuntimeError: One or more Gitaly storages are accessible by multiple filesystem paths: /var/opt/gitlab/gitaly-cny-01-stor-gprd.c.gitlab-production.internal/repositories: default, gitaly-cny-01-stor-gprd.c.gitlab-production.internal stdout_lines: <omitted>-
From @skarbek on Slack:
I see we currently have the 2AM package installed.
ok, so this node needs an upgrade.
Deployer isn't smart. We first check if the node is healthy running the gitlab-ctl reconfigure. That fails. We ignore that as a failure to Ansible, but leverage that to force it to run that task again to hopefully repair the node file that omnibus relies on. We want that fixed prior to installing the new version because omnibus would fail for the same reason.
-
The job succeeded after the node was updated https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/13893896
Just a followup on my comment above. Deployer wants to ensure the installation of GitLab is healthy before we do anything. Since omnibus would fail to install if it weren't, we want to get ahead of that. So deploy does a
gitlab-ctl check-config, which in our case here, signaled the installation was not healthy. This doesn't fail Ansible, but is instead leveraged to forcibly run thegitlab-ctl reconfigurestep. This failed because the package currently installed did not contain the fix we needed.
To work around this, I moved the file that omnibus relies on out of the way, and ran an apt upgrade, specifying the target version we were aiming to install. After this, retrying the migrations job didn't need to install the package, but just run the migrations which as noted above, was successful.
We ran into this on dev yesterday: #17988 (closed)
As I mentioned in gitlab-org/omnibus-gitlab!7564 (comment 1900057382), users can't simply drop entries without remapping the
projects.repository_storageappropriately.I see https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/4743/diffs updated the
defaultentry. Fortunately ongstg, it seems there are nodefaultentries?gitlabhq_production=# SELECT repository_storage, COUNT(*) FROM projects GROUP BY repository_storage; repository_storage | count --------------------------------------------------------+-------- gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-164c.internal | 840299 gitaly-01-stor-gstg.c.gitlab-gitaly-gstg-380a.internal | 835588 gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-164c.internal | 839731 gitaly-02-stor-gstg.c.gitlab-gitaly-gstg-380a.internal | 839643 gitaly-cny-01-stor-gstg.c.gitlab-staging-1.internal | 4028 nfs-file07 | 21 nfs-file09 | 4 (7 rows)
-
The deployment to prod https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/3240391 with gitlab-org/omnibus-gitlab!7600 (merged) successfully completed.
added IncidentMitigated label and removed IncidentActive label
added Deploys-blocked-gprd14hr Deploys-blocked-gstg14hr labels
mentioned in issue gitlab-org/release/tasks#10621 (closed)
mentioned in issue reliability-reports#238 (closed)
First of all, thank you for taking part in this incident.
We're posting this message because this issue meets the following criteria:
- This incident is open
- No activity in the past 3 days (since 2024-05-13T07:01:49.889Z)
We'd like to ask you to help us out and determine how we should act on this issue.
Incident issues in IncidentMitigated state should only remain open for ongoing incidents
If there is a reason that it should remain open, please add a note explaining the lack of activity, otherwise please consider closing, and starting a separate follow-up investigation issue if needed.
Thanks for your help!
added auto updated label
-
First of all, thank you for taking part in this incident.
We're posting this message because this issue meets the following criteria:
- This incident is open
- No activity in the past 3 days (since 2024-05-17T00:01:12.701Z)
We'd like to ask you to help us out and determine how we should act on this issue.
Incident issues in IncidentMitigated state should only remain open for ongoing incidents
If there is a reason that it should remain open, please add a note explaining the lack of activity, otherwise please consider closing, and starting a separate follow-up investigation issue if needed.
Thanks for your help!
added IncidentResolved label and removed IncidentMitigated label
