Skip to content

Investigate why deploy page was still up at the end of deployment to release.gitlab.net

Summary

The automatic deployment to release.gitlab.net started and completed after 16.11 was tagged today (17th April 2024).

However, after that, all QA jobs failed with the following error:

An error occurred in a `before(:suite)` hook.
Failure/Error: raise Validatable::PageValidationError, "#{element.name} did not appear on #{self.name} as expected"
QA::Page::Validatable::PageValidationError:
  login-page did not appear on QA::Page::Main::Login as expected

Opening release.gitlab.net showed us the following page:

Screenshot_from_2024-04-17_19-22-57

@skarbek found the following in NGINX logs:

2024/04/17 13:14:58 [crit] 1439962#0: *957615 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 108.162.237.169 <SNIP>

I logged in to the release VM, and manually ran the sudo gitlab-ctl deploy-page down command:

rpereira@release-01-sv-release.c.gitlab-release.internal:~$ sudo gitlab-ctl deploy-page status
Deploy page is up
rpereira@release-01-sv-release.c.gitlab-release.internal:~$ sudo gitlab-ctl deploy-page down
rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html

After that, opening release.gitlab.net showed the login page, and on retrying QA jobs, all passed successfully.

Important links

Edited by Reuben Pereira