Skip to content

Failed to remove the existing associated sourced_pipeline. The record failed to save after its foreign key was set to nil.

Summary

Triggering multiple pipelines using a CI JOB TOKEN succeeds for the first one, but returns 500 error for the rest:

curl --insecure -X POST -F token=$CI_JOB_TOKEN -F ref=$SOMEBRANCH https://gitlab.com/api/v4/projects/$PROJECTID/trigger/pipeline
{"message":"500 Internal Server Error"}

Fixed

The change has been reverted - #384053 (comment 1190983404)

If a workaround is needed in the mean time, try #384053 (comment 1191565282)

Backend detail

Customer reported that calls to https://gitlab.com/api/v4/projects/<projectpath>/trigger/pipeline had started failing with 500 errors.

Tracking them down in Kibana identified the following error:

exception.class:  ActiveRecord::RecordNotSaved
exception.message: Failed to remove the existing associated sourced_pipeline. The record failed to save after its foreign key was set to nil.
exception.backtrace: 
app/services/ci/pipeline_trigger_service.rb:65:in `block in create_pipeline_from_job'
 lib/gitlab/ci/pipeline/chain/seed_block.rb:15:in `perform!'
 lib/gitlab/ci/pipeline/chain/sequence.rb:20:in `block in build!'
 lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `each'
 lib/gitlab/ci/pipeline/chain/sequence.rb:16:in `build!'
 app/services/ci/create_pipeline_service.rb:98:in `execute'
 app/services/ci/pipeline_trigger_service.rb:64:in `create_pipeline_from_job'
 app/services/ci/pipeline_trigger_service.rb:15:in `execute'
 lib/api/ci/triggers.rb:49:in `block (2 levels) in <class:Triggers>'
 ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
 ee/lib/gitlab/ip_address_state.rb:10:in `with'
 ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
 lib/api/api_guard.rb:215:in `call'
 ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
 lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
 lib/gitlab/middleware/memory_report.rb:13:in `call'
 lib/gitlab/middleware/speedscope.rb:13:in `call'
 lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
 lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
 lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
 lib/gitlab/metrics/web_transaction.rb:46:in `run'
 lib/gitlab/metrics/rack_middleware.rb:16:in `call'
 lib/gitlab/jira/middleware.rb:19:in `call'
 lib/gitlab/middleware/go.rb:20:in `call'
 lib/gitlab/etag_caching/middleware.rb:21:in `call'
 lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
 lib/gitlab/database/query_analyzer.rb:37:in `within'
 lib/gitlab/middleware/query_analyzer.rb:11:in `call'
 lib/gitlab/middleware/multipart.rb:173:in `call'
 lib/gitlab/middleware/read_only/controller.rb:50:in `call'
 lib/gitlab/middleware/read_only.rb:18:in `call'
 lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
 lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
 lib/gitlab/middleware/basic_health_check.rb:25:in `call'
 lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
 lib/gitlab/middleware/request_context.rb:21:in `call'
 lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
 config/initializers/fix_local_cache_middleware.rb:11:in `call'
 lib/gitlab/middleware/compressed_json.rb:37:in `call'
 lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
 lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
 lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
 lib/gitlab/middleware/release_env.rb:13:in `call'

zooming out in Kibana identified that these errors started roughly 2022-11-29 03:00 UTC

Links for GitLab team member:

Steps to reproduce

Example Project

What is the current bug behavior?

stacktrace and 500 is produced

What is the expected correct behavior?

previous behaviour

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Ben Prescott (ex-GitLab)