release-tools is currently failing leading to a halt in the auto-deploy process
Auto-deploy is currently stopped due to an issue with the release-tools trying to find the current release issue.
Issue: gitlab-org/release/tasks#1806 (closed)
Failure: https://sentry.gitlab.net/gitlab/release-tools/issues/2249931/
Failed job: https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/2250293
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- John Skarbek added AutoDeploy DeliveryP1 labels
added AutoDeploy DeliveryP1 labels
- Maintainer
I believe this was caused due to the recent upgrade to ruby 2.7 gitlab-org/gitlab-build-images!335 (merged), #1319 (closed), I can't reproduce the error with 2.6:
[1] pry(#<ReleaseTools::Tasks::AutoDeploy::CheckProduction>)> monthly_issue => #<ReleaseTools::MonthlyIssue version="13.6.0"> [2] pry(#<ReleaseTools::Tasks::AutoDeploy::CheckProduction>)> monthly_issue.version.milestone_name => "13.6"
But it's quite easy to reproduce with 2.7
[2] pry(#<ReleaseTools::Tasks::AutoDeploy::CheckProduction>)> monthly_issue => #<ReleaseTools::MonthlyIssue version=nil> [3] pry(#<ReleaseTools::Tasks::AutoDeploy::CheckProduction>)> monthly_issue.version => nil [4] pry(#<ReleaseTools::Tasks::AutoDeploy::CheckProduction>)> monthly_issue.version.milestone_name NoMethodError: undefined method `milestone_name' for nil:NilClass from (pry):4:in `execute'
Collapse replies - Maintainer
Turns out the version we're using for
http
(4.1.3
) is not compatible with ruby 2.7 (see https://github.com/httprb/http/issues/582), this was causing forHTTP.get(SCHEDULE_YAML).to_s
onReleaseTools::ReleaseManagers::Schedule
to timeout propagating the error up toCheckProduction
class[3] pry(#<ReleaseTools::ReleaseManagers::Schedule>)> HTTP.get(SCHEDULE_YAML).to_s /Users/mayra_cabrera/.rvm/gems/ruby-2.7.2/gems/pry-0.12.2/lib/pry/exceptions.rb:28: warning: $SAFE will become a normal global variable in Ruby 3.0 FrozenError: can't modify frozen String: "" from /Users/mayra_cabrera/.rvm/gems/ruby-2.7.2/gems/http-4.1.1/lib/http/response/body.rb:52:in `force_encoding' [4] pry(#<ReleaseTools::ReleaseManagers::Schedule>)> YAML.safe_load(download_schedule)
After the upgrade on gitlab-org/release-tools!1261 (merged), it's working accordingly
[1] pry(#<ReleaseTools::ReleaseManagers::Schedule>)> HTTP.get(SCHEDULE_YAML).to_s => "- version: '13.6'\n date: November 22nd, 2020\n manager_americas:\n - John Skarbek\n manager_apac_emea:\n - John Jarvis\n appsec:\n - Andrew Kelly\n\n- version: '13.5'\n date: October 22nd, 2020\n manager_americas:\n - Robert Speicher\n manager_apac_emea:\n - John Jarvis\n appsec:\n - Dominic Couture\n\n- version: '13.4'\n date: September 22nd, 2020\n manager_americas:\n - Robert Speicher\n # ....
Edited by Mayra Cabrera - Maintainer
cc @stanhu ^ in case this bite us again
@mayra-cabrera Thanks for finding and fixing that. Does
release-tools
need to use Webmock to test outbound requests? I checkedgitlab-org/gitlab
, and it looks like we're using http v4.4.1, so we should be okay there.- Maintainer
@stanhu we use webmock in some of them but it's not a standardized practice, I've opened an issue about it #1329 (closed)
- John Skarbek marked this issue as related to #1324 (closed)
marked this issue as related to #1324 (closed)
- Mayra Cabrera mentioned in merge request gitlab-org/release-tools!1261 (merged)
mentioned in merge request gitlab-org/release-tools!1261 (merged)
- Author Owner
Resolved via gitlab-org/release-tools!1261 (merged) Closing
- John Skarbek closed
closed
- Mayra Cabrera assigned to @mayra-cabrera
assigned to @mayra-cabrera
- Mayra Cabrera changed milestone to %13.6
changed milestone to %13.6
- Mayra Cabrera mentioned in issue #1329 (closed)
mentioned in issue #1329 (closed)