Omnibus should check nginx configuration before restart
When setting an invalid configuration for nginx (example https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/4329) after reconfigure it is possible for nginx to go down with an invalid nginx.conf.
After the config change nginx is issued a restart:
[2020-09-30T09:43:53+00:00] INFO: Started Chef Infra Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
One version per cookbook
[2020-09-30T09:43:53+00:00] INFO: *** Chef Infra Client 15.12.22 ***
[2020-09-30T09:43:53+00:00] INFO: Platform: x86_64-linux
[2020-09-30T09:43:53+00:00] INFO: Chef-client pid: 20119
[2020-09-30T09:43:55+00:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options
[2020-09-30T09:43:55+00:00] INFO: Run List is [recipe[gitlab-ee]]
[2020-09-30T09:43:55+00:00] INFO: Run List expands to [gitlab-ee]
[2020-09-30T09:43:55+00:00] INFO: Starting Chef Infra Client Run for web-01-sv-pre.c.gitlab-pre.internal
[2020-09-30T09:43:55+00:00] INFO: Running start handlers
[2020-09-30T09:43:55+00:00] INFO: Start handlers complete.
[2020-09-30T09:43:57+00:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, repmgr@0.1.0, patroni@0.1.0, runit@5.1.3, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.1, crond@0.1.0]
[2020-09-30T09:44:03+00:00] WARN: Selected systemd because systemctl shows .mount units
[2020-09-30T09:44:05+00:00] WARN: only_if block for file[/etc/gitlab/gitlab-registry.key] returned a string, did you mean to run a command?
[2020-09-30T09:44:06+00:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2020-09-30T09:44:07+00:00] INFO: ruby_block[Link postgresql bin files to the correct version] called
[2020-09-30T09:44:07+00:00] INFO: runit_service[puma] running why-run enable action to support before action
[2020-09-30T09:44:08+00:00] INFO: template[/var/opt/gitlab/nginx/conf/nginx-status.conf] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/nginx-status.conf.chef-20200930094408.205283
[2020-09-30T09:44:08+00:00] INFO: template[/var/opt/gitlab/nginx/conf/nginx-status.conf] updated file contents /var/opt/gitlab/nginx/conf/nginx-status.conf
[2020-09-30T09:44:08+00:00] INFO: template[/var/opt/gitlab/nginx/conf/nginx-status.conf] sending restart action to runit_service[nginx] (delayed)
[2020-09-30T09:44:33+00:00] INFO: Chef Infra Client Run complete in 37.229304752 seconds
[2020-09-30T09:44:33+00:00] INFO: Running report handlers
[2020-09-30T09:44:33+00:00] INFO: Report handlers complete
Following this we see errors in nginx:
2020/09/30 09:57:43 [emerg] 25179#0: invalid parameter "any" in /var/opt/gitlab/nginx/conf/nginx-status.conf:8
Prior to restarting nginx, we should validate the config with a
nginx -t
and then I think fail the reconfigure if it doesn't pass.