All runners failing after gitlab-fips package updated to 17.7.0 (500 err on server with "CI job token signing key is not set")
Summary
Local install of gitlab-runners and gitlab-fips from yum.
After upgrading gitlab to 17.7.0, all of our runners are failing. With an error on the server RuntimeError (CI job token signing key is not set).
Steps to reproduce
Start any job as it seems the error is on the server, and not all of the runners. However, I'm opening the Issue here has this seems to be only related to the runners.
I also have multiple jobs that run both in docker, docker-dind, and bare metal. All of them are failing.
Actual behavior
When a job is sent to a runner, it posts to PATCH /api/:version/jobs/:id/trace which is giving a 500 Error. The error in the stack trace is RuntimeError (CI job token signing key is not set) which I've never seen before.
Expected behavior
Job runs as expected.
Relevant logs and/or screenshots
`# gitlab-ctl tail | grep ' 500 '`
# gitlab-ctl tail | grep ' 500 '
*** redacted *** - gitlab-ci-token [20/Dec/2024:13:13:44 -0500] "GET /*** redacted ***.git/info/refs?service=git-upload-pack HTTP/2.0" 500 1624 "" "gitlab-runner 17.7.0 linux/amd64" -
*** redacted *** - - [20/Dec/2024:13:13:44 -0500] "PATCH /api/v4/jobs/55390/trace?debug_trace=false HTTP/1.1" 500 39 "" "gitlab-runner 17.7.0 (17-7-stable; go1.23.2; linux/amd64)" -
On gitlab server at: `/var/log/gitlab/gitlab-rails/production.log`
RuntimeError (CI job token signing key is not set):
/opt/gitlab/embedded/service/gitlab-rails/lib/ci/job_token/jwt.rb:66:in `key'
/opt/gitlab/embedded/service/gitlab-rails/lib/ci/job_token/jwt.rb:41:in `decode'
/opt/gitlab/embedded/service/gitlab-rails/app/models/ci/build.rb:776:in `valid_token?'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/ci/helpers/runner.rb:81:in `authenticate_job!'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/ci/runner.rb:278:in `block (2 levels) in <class:Runner>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:58:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:58:in `block (2 levels) in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/notifications.rb:208:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:57:in `block in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:328:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:260:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/notifications.rb:208:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:240:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:316:in `block in build_stack'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ip_address_state.rb:11:in `with'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api_guard.rb:219:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-oauth2-2.2.1/lib/rack/oauth2/server/resource.rb:20:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-oauth2-2.2.1/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-oauth2-2.2.1/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/error.rb:39:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/error.rb:38:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/error.rb:38:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape_logging-1.8.4/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:224:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/endpoint.rb:218:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router/route.rb:58:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:120:in `process_route'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:74:in `block in identity'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:94:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:72:in `identity'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:56:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:136:in `with_optimization'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/router.rb:55:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:165:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:70:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/grape-2.0.0/lib/grape/api/instance.rb:65:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/routing/route_set.rb:852:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-experiment-0.9.1/lib/gitlab/experiment/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:470:in `call_app!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth_openid_connect-0.8.0/lib/omniauth/strategies/openid_connect.rb:163:in `other_phase'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:195:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:169:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:202:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/omniauth-2.1.0/lib/omniauth/strategy.rb:169:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/flipper-0.26.2/lib/flipper/middleware/memoizer.rb:37:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/memory_report.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/speedscope.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:21:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/web_transaction.rb:46:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:21:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/query_analyzer.rb:11:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/ci/job_token/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/batch-loader-2.0.5/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:103:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/apollo_upload_server-2.1.6/lib/apollo_upload_server/middleware.rb:19:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:173:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-attack-6.7.0/lib/rack/attack.rb:127:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-cors-2.0.2/lib/rack/cors.rb:102:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/etag.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/conditional_get.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/http/permissions_policy.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/http/content_security_policy.rb:36:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:50:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/session/abstract/id.rb:266:in `context'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/session/abstract/id.rb:260:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/cookies.rb:704:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/callbacks.rb:99:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-rails-5.21.0/lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry/hub.rb:265:in `with_session_tracking'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry-ruby.rb:412:in `with_session_tracking'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry/rack/capture_exceptions.rb:21:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry/hub.rb:59:in `with_scope'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry-ruby.rb:392:in `with_scope'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/sentry-ruby-5.21.0/lib/sentry/rack/capture_exceptions.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/lograge-0.11.2/lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.6/lib/rails/rack/logger.rb:25:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.6/lib/rails/rack/logger.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:15:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/method_override.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-timeout-0.7.0/lib/rack/timeout/core.rb:154:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-timeout-0.7.0/lib/rack/timeout/support/timeout.rb:19:in `timeout'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-timeout-0.7.0/lib/rack/timeout/core.rb:153:in `call'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:11:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/compressed_json.rb:44:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/sendfile.rb:110:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-labkit-0.37.0/lib/labkit/middleware/rack.rb:22:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-labkit-0.37.0/lib/labkit/context.rb:35:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/gitlab-labkit-0.37.0/lib/labkit/middleware/rack.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/actionpack-7.0.8.6/lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.6/lib/rails/engine.rb:530:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.6/lib/rails/railtie.rb:226:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.6/lib/rails/railtie.rb:226:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/urlmap.rb:74:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/urlmap.rb:58:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rack-2.2.10/lib/rack/urlmap.rb:58:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/configuration.rb:279:in `call'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/request.rb:99:in `block in handle_request'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/thread_pool.rb:389:in `with_force_shutdown'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/request.rb:98:in `handle_request'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/server.rb:468:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/server.rb:249:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/puma-6.5.0/lib/puma/thread_pool.rb:166:in `block in spawn_thread'
On gitlab runner, running: `journalctl --unit=gitlab-runner.service -f`
Dec 20 12:37:43 *** redacted *** gitlab-runner[150566]: Added job to processing list builds=2 job=55388 max_builds=4 project=611 repo_url=https://*** redacted ***.git time_in_queue_seconds=3
Dec 20 12:37:45 *** redacted *** gitlab-runner[150566]: WARNING: Job failed: exit code 1
Dec 20 12:37:45 *** redacted *** gitlab-runner[150566]: duration_s=2.06148499 job=55388 project=611 runner=hQyXUCvpJ
Dec 20 12:38:45 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:39:49 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:40:55 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:42:02 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:43:10 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:44:18 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:45:26 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:46:34 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:47:42 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:48:50 *** redacted *** gitlab-runner[150566]: WARNING: Appending trace to coordinator... failed code=500 job=55388 job-log= job-status= runner=hQyXUCvpJ sent-log=0-1493 status=500 Internal Server Error update-interval=0s
Dec 20 12:48:53 *** redacted *** gitlab-runner[150566]: ERROR: Job trace termination "Fail" failed duration_s=2.06148499 error=received invalid patch trace response job=55388 project=611 runner=hQyXUCvpJ
Dec 20 12:48:53 *** redacted *** gitlab-runner[150566]: Removed job from processing list builds=0 job=55388 max_builds=4 project=611 repo_url=https://*** redacted ***.git time_in_queue_seconds=3Environment description
All local runners running gitlab-runner.x86_64 17.7.0-1 @runner_gitlab-runner
config.toml contents
concurrent = 4
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "*** redacted ***"
url = "https://*** redacted ***/"
id = 35
token = "*** redacted ***"
token_obtained_at = 2024-09-03T21:04:38Z
token_expires_at = 0001-01-01T00:00:00Z
tls-ca-file = "/etc/gitlab-runner/certs/*** redacted ***.crt"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
Used GitLab Runner version
Version: 17.7.0
Git revision: 3153ccc6
Git branch: 17-7-stable
GO version: go1.23.2
Built: 2024-12-19T20:10:51+0000
OS/Arch: linux/amd64
Possible fixes
None know to me at this time.
CI job token signing key is not set
Apparently set the signing key for CI job token, though I've never seen that and not sure how to set that key.