Failed gitlab-ctl reconfigure broke CI for a single group
Summary
A failed gitlab-ctl reconfigure
due to syntax errors appears to have broken CI for an entire group.
Steps to reproduce
An attempt to run gitlab-ctl reconfigure
failed due to a syntax error in gitlab.rb after configuring LDAP authentication. We soon after observed Runners picking up jobs from repositories in one particular group, but hanging with no output whatsoever. Running gitlab-ctl reconfigure
on a linted gitlab.rb did not remedy the issue, nor did destroying and redeploying Omnibus.
This was the only apparent consequence of the failed reconfiguration.
What is the current bug behavior?
GitLab Runner logs showed a steady stream of nonspecific 500 errors, and jobs appeared to never begin after they had been picked up. After isolating the error to only a single group, we observed 500 errors when attempting to navigate to the CI/CD settings for that group; the same error was not apparent in other groups.
What is the expected correct behavior?
Jobs process as usual.
Relevant logs and/or screenshots
This Runner was destroyed in our effort to isolate the problem; however, we still have the GitLab group returning HTTP 500 in its CI/CD settings.
Results of GitLab environment info
Will follow up with a redacted gitlab-rake gitlab:env:info
output.
Results of GitLab application Check
Will follow up with a redacted gitlab-rake gitlab:check SANITIZE=true
output.
FYI, the "contribution guidelines" link opened in _self in the process of submitting this issue. Also, CAPTCHA submission presented a browser warning about navigating away from the page.