Failed to remove the existing associated sourced_pipeline. The record failed to save after its foreign key was set to nil.
Summary
Triggering multiple pipelines using a CI JOB TOKEN succeeds for the first one, but returns 500
error for the rest:
curl --insecure -X POST -F token=$CI_JOB_TOKEN -F ref=$SOMEBRANCH https://gitlab.com/api/v4/projects/$PROJECTID/trigger/pipeline
{"message":"500 Internal Server Error"}
Fixed
The change has been reverted - #384053 (comment 1190983404)
If a workaround is needed in the mean time, try #384053 (comment 1191565282)
Backend detail
Customer reported that calls to https://gitlab.com/api/v4/projects/<projectpath>/trigger/pipeline
had started failing with 500 errors.
Tracking them down in Kibana identified the following error:
exception.class: ActiveRecord::RecordNotSaved
exception.message: Failed to remove the existing associated sourced_pipeline. The record failed to save after its foreign key was set to nil.
exception.backtrace:
app/services/ci/pipeline_trigger_service.rb:65:in `block in create_pipeline_from_job'
lib/gitlab/ci/pipeline/chain/seed_block.rb:15:in `perform!'
lib/gitlab/ci/pipeline/chain/sequence.rb:20:in `block in build!'
lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `each'
lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `build!'
app/services/ci/create_pipeline_service.rb:98:in `execute'
app/services/ci/pipeline_trigger_service.rb:64:in `create_pipeline_from_job'
app/services/ci/pipeline_trigger_service.rb:15:in `execute'
lib/api/ci/triggers.rb:49:in `block (2 levels) in <class:Triggers>'
ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
lib/api/api_guard.rb:215:in `call'
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:37:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
zooming out in Kibana identified that these errors started roughly 2022-11-29 03:00 UTC
Links for GitLab team member:
- Kibana - https://log.gprd.gitlab.net/goto/5b34d590-7005-11ed-85ed-e7557b0a598c
- Zendesk - https://gitlab.zendesk.com/agent/tickets/349755
Steps to reproduce
Example Project
What is the current bug behavior?
stacktrace and 500 is produced
What is the expected correct behavior?
previous behaviour
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)