release-tools error PackageCommitOlderThanLatestError is unfriendly
Problem Description
Recently in incident: production#16032 (closed)
GitLab Shell had already been deployed to some environments. Thus the check to ensure our package commits, specifically for GitLab Shell are failing. This is indeed expected. However, this is also the fix for the incident. It's not clear in the error that we've got this tucked behind a feature flag. And when navigating the file associated with the check, there's no mention of a feature flag: https://gitlab.com/gitlab-org/release-tools/-/blob/030d11e9a1affa2f779e3a46843450bc7b11b55d/lib/release_tools/tasks/auto_deploy/check_package.rb
But, stop_deployment_if_old_package
feature flag does indeed do exactly what we need it too. Upon disabling this feature flag, we were able to move forward with the deployment.
Solution
-
Document this, currently none exist: https://gitlab.com/search?search=stop_deployment_if_old_package&nav_source=navbar&project_id=5191294&group_id=2351283&search_code=true&repository_ref=master -
Improve the error message to include what to disable when we face this error. -
Consider building a mechanism to warn us if this feature flag is off. It's meant to protect us, we shouldn't leave it off by accident!