Chatops commands failing with 403 errors
Chatops Environment lock failure
/chatops run auto_deploy lock gstg-cny
- slack: https://gitlab.slack.com/archives/C0139MAV672/p1664799530169649
- Failed job: https://ops.gitlab.net/gitlab-com/chatops/-/jobs/8134521
2022-10-03 12:18:48.136615 D [{:user=>"amyphillips"}] Chatops::Gitlab::Client -- [HTTParty] [2022-10-03 12:18:48 +0000] 200 "GET https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules" -
2022-10-03 12:18:48.447415 D [{:user=>"amyphillips"}] Chatops::Gitlab::Client -- [HTTParty] [2022-10-03 12:18:48 +0000] 403 "PUT https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules/73" 27
/app/vendor/bundle/ruby/2.6.0/gems/gitlab-4.19.0/lib/gitlab/request.rb:71:in `validate': Server responded with code 403, message: 403 Forbidden. Request URI: https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules/73 (Gitlab::Error::Forbidden)
from /app/vendor/bundle/ruby/2.6.0/gems/gitlab-4.19.0/lib/gitlab/request.rb:55:in `block (2 levels) in <class:Request>'
from /app/vendor/bundle/ruby/2.6.0/gems/gitlab-4.19.0/lib/gitlab/client/pipeline_schedules.rb:65:in `edit_pipeline_schedule'
from /usr/local/lib/ruby/2.6.0/forwardable.rb:230:in `edit_pipeline_schedule'
from /app/lib/chatops/gitlab/auto_deploy.rb:29:in `pause_prepare'
from /app/lib/chatops/commands/auto_deploy.rb:170:in `lock'
from /app/lib/chatops/commands/auto_deploy.rb:73:in `public_send'
from /app/lib/chatops/commands/auto_deploy.rb:73:in `perform'
from /app/lib/chatops/command.rb:105:in `perform'
from /app/lib/chatops/chatops.rb:45:in `run'
from /app/bin/chatops:10:in `<main>'
Auto-deploy pause pipeline failure
chatops run auto_deploy pause
slack command results in a build failure due to 403 error
This is when it first failed:
- slack: https://gitlab.slack.com/archives/C0139MAV672/p1663960904988909 (this thread contains various mitigation steps we tried)
- failed job: https://ops.gitlab.net/gitlab-com/chatops/-/jobs/8074444
2022-09-23 19:21:20.660738 I [{:user=>"jennykim-gitlab"}] Chatops -- Command to be executed -- {:name=>"auto_deploy", :chat_input=>"pause", :split_chat_input=>["pause"]}
2022-09-23 19:21:41.146756 D [{:user=>"jennykim-gitlab"}] Chatops::Gitlab::Client -- [HTTParty] [2022-09-23 19:21:41 +0000] 200 "GET https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules" -
2022-09-23 19:21:41.367090 D [{:user=>"jennykim-gitlab"}] Chatops::Gitlab::Client -- [HTTParty] [2022-09-23 19:21:41 +0000] 403 "PUT https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules/118" 27
/app/vendor/bundle/ruby/2.6.0/gems/gitlab-4.19.0/lib/gitlab/request.rb:71:in `validate': Server responded with code 403, message: 403 Forbidden. Request URI: https://ops.gitlab.net/api/v4/projects/gitlab-org%2Frelease%2Ftools/pipeline_schedules/118 (Gitlab::Error::Forbidden)
It was attempted under mayra's user as well, and her attempt also errored out the same way: https://ops.gitlab.net/gitlab-com/chatops/-/jobs/8074528
We ended up manually pausing the pipelines for now.
Edited by Amy Phillips