runner_core: POST /api/v4/runners/reset_authentication_token returns HTTP 500 for unassigned project runners
Summary
When a project-type runner that is not assigned to any project attempts to rotate its authentication token via POST /api/v4/runners/reset_authentication_token, the API returns an HTTP 500 Internal Server Error. This is a server-side misclassification of a foreseeable, user-correctable client configuration state.
The error surfaces because an unhandled ActiveRecord::RecordInvalid exception propagates all the way to the Grape response layer, rather than being caught and returned as a structured 4xx client error. The result is that every token rotation attempt by a misconfigured runner is logged as a server exception — causing legitimate SLO alerts to fire against what is effectively a client-side misconfiguration.
Feature category: runner_core
Affected endpoint: POST /api/v4/runners/reset_authentication_token (lib/api/ci/runner.rb)
Affected service: app/services/ci/runners/reset_authentication_token_service.rb
Impact
On GitLab Dedicated
This issue has caused repeated WebserviceServicePumaErrorSLOViolation PagerDuty pages across multiple GitLab Dedicated tenants. In each case, on-call engineers investigated and found no real service degradation — the 500s are entirely driven by a single misconfigured runner client retrying aggressively.
Observed on tenant zeroth_brown_tiger (2026-03-05):
- 957 HTTP 500 responses in a single burst window (~3 seconds, 11:56:47–11:56:50 UTC)
- All errors from a single runner:
meta.client_id: runner/14,gitlab-runner 18.4.0 (darwin/arm64) - Same root cause previously seen on another Dedicated tenant (incident #1900, INC-4581)
Per-incident cost:
- Each occurrence triggers a PagerDuty page and requires an on-call engineer to investigate
- Investigation time: ~30–60 minutes per incident to rule out genuine service degradation
- The SLO violation alarm is a false positive — no user-facing impact beyond the runner operator receiving unhelpful 500 errors
Example log entry from production (GitLab v18.8.5):
method: POST
path: /api/v4/runners/reset_authentication_token
status: 500
severity: INFO ← request log is INFO, but exception fires a separate ERROR log
exception.class: ActiveRecord::RecordInvalid
exception.message: Validation failed: Runner needs to be assigned to at least one project
meta.feature_category: runner_core
meta.client_id: runner/14
ua: gitlab-runner 18.4.0 (18-4-stable; go1.25.1; darwin/arm64)
duration_s: 0.024The mixed severity: INFO on the request log and a separate ERROR-level exception log for the same event is what causes monitoring to miscount these as server errors.
Recommendation
The POST /api/v4/runners/reset_authentication_token endpoint should return a 4xx response (e.g. 403 Forbidden or 422 Unprocessable Entity) when the runner is in an invalid state (not assigned to any project), rather than allowing an unhandled ActiveRecord::RecordInvalid exception to produce a 500.
See the investigation comment on this issue for a more detailed theory on what may have changed and what a fix could look like.
Verification
Query OpenSearch / Kibana for the affected endpoint returning 500s.
Index: gitlab-*
{
"query": {
"bool": {
"filter": [
{ "term": { "kubernetes.container_name": "webservice" } },
{ "term": { "meta.caller_id.keyword": "POST /api/:version/runners/reset_authentication_token" } },
{ "term": { "status": 500 } }
]
}
},
"aggs": {
"over_time": {
"date_histogram": { "field": "@timestamp", "fixed_interval": "1h" }
}
}
}Interpretation: If the query returns results with exception.class: ActiveRecord::RecordInvalid and exception.message containing "Runner needs to be assigned to at least one project", the issue is still present. After a fix, this query should return zero 500 results for this endpoint and error combination.
Example
Here's a sanitised log example:
Click to expand log example
{
"_index": "gitlab-2026-03-05",
"_type": "_doc",
"_source": {
"time": "2026-03-05T12:51:14.199Z",
"docker": {
"container_id": "<REDACTED>"
},
"kubernetes": {
"container_name": "webservice",
"namespace_name": "default",
"pod_name": "gitlab-webservice-default-<REDACTED>",
"container_image": "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v18.8.5",
"container_image_id": "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee@sha256:1a5dc591c952d48bf823c7f2355aac0575bcd13384058d1232217dc73cab7f4b",
"pod_id": "<REDACTED>",
"pod_ip": "<REDACTED>",
"host": "<REDACTED>",
"labels": {
"app.kubernetes.io/name": "gitlab",
"app.kubernetes.io/version": "v18.8.5",
"chart": "webservice-9.8.5",
"gitlab.com/webservice-name": "default",
"heritage": "Helm",
"pod-template-hash": "64dd7cd7b4",
"release": "gitlab"
},
"master_url": "<REDACTED>",
"namespace_id": "<REDACTED>",
"namespace_labels": {
"kubernetes.io/metadata.name": "default"
}
},
"component": "gitlab",
"subcomponent": "exceptions_json",
"severity": "ERROR",
"correlation_id": "<REDACTED>",
"meta.caller_id": "POST /api/:version/runners/reset_authentication_token",
"meta.remote_ip": "<REDACTED>",
"meta.feature_category": "runner_core",
"meta.client_id": "runner/<ID>",
"exception.class": "ActiveRecord::RecordInvalid",
"exception.message": "Validation failed: Runner needs to be assigned to at least one project",
"exception.backtrace": [
"activerecord (7.2.3) lib/active_record/validations.rb:87:in `raise_validation_error'",
"activerecord (7.2.3) lib/active_record/validations.rb:54:in `save!'",
"activerecord (7.2.3) lib/active_record/transactions.rb:366:in `block in save!'",
"activerecord (7.2.3) lib/active_record/transactions.rb:418:in `block (2 levels) in with_transaction_returning_status'",
"activerecord (7.2.3) lib/active_record/connection_adapters/abstract/transaction.rb:616:in `block in within_new_transaction'",
"activesupport (7.2.3) lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
"activerecord (7.2.3) lib/active_record/connection_adapters/abstract/transaction.rb:613:in `within_new_transaction'",
"activerecord (7.2.3) lib/active_record/connection_adapters/abstract/database_statements.rb:361:in `transaction'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:229:in `retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:131:in `read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'",
"activerecord (7.2.3) lib/active_record/transactions.rb:414:in `block in with_transaction_returning_status'",
"lib/gitlab/database/load_balancing/setup.rb:57:in `block in setup_connection_proxy'",
"activerecord (7.2.3) lib/active_record/transactions.rb:410:in `with_transaction_returning_status'",
"activerecord (7.2.3) lib/active_record/transactions.rb:366:in `save!'",
"activerecord (7.2.3) lib/active_record/suppressor.rb:56:in `save!'",
"lib/authn/token_field/base.rb:70:in `reset_token!'",
"app/models/concerns/token_authenticatable.rb:75:in `block in add_authentication_token_field'",
"app/services/ci/runners/reset_authentication_token_service.rb:19:in `execute!'",
"lib/api/ci/runner.rb:144:in `block (2 levels) in <class:Runner>'",
"grape (2.0.0) lib/grape/endpoint.rb:58:in `call'",
"grape (2.0.0) lib/grape/endpoint.rb:58:in `block (2 levels) in generate_api_method'",
"activesupport (7.2.3) lib/active_support/notifications.rb:212:in `instrument'",
"grape (2.0.0) lib/grape/endpoint.rb:57:in `block in generate_api_method'",
"grape (2.0.0) lib/grape/endpoint.rb:328:in `execute'",
"grape (2.0.0) lib/grape/endpoint.rb:260:in `block in run'",
"activesupport (7.2.3) lib/active_support/notifications.rb:212:in `instrument'",
"grape (2.0.0) lib/grape/endpoint.rb:240:in `run'",
"grape (2.0.0) lib/grape/endpoint.rb:316:in `block in build_stack'",
"grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"lib/gitlab/middleware/ip_address.rb:14:in `block in call'",
"lib/gitlab/ip_address_state.rb:11:in `with'",
"lib/gitlab/middleware/ip_address.rb:13:in `call'",
"grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"lib/api/api_guard.rb:270:in `call'",
"grape (2.0.0) lib/grape/middleware/base.rb:36:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource.rb:20:in `_call'",
"rack-oauth2 (2.2.1) lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'",
"rack-oauth2 (2.2.1) lib/rack/oauth2/server/abstract/handler.rb:17:in `call'",
"grape (2.0.0) lib/grape/middleware/error.rb:39:in `block in call!'",
"grape (2.0.0) lib/grape/middleware/error.rb:38:in `catch'",
"grape (2.0.0) lib/grape/middleware/error.rb:38:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'",
"grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `catch'",
"grape_logging (1.8.4) lib/grape_logging/middleware/request_logger.rb:58:in `call!'",
"grape (2.0.0) lib/grape/middleware/base.rb:29:in `call'",
"rack (2.2.21) lib/rack/head.rb:12:in `call'",
"grape (2.0.0) lib/grape/endpoint.rb:224:in `call!'",
"grape (2.0.0) lib/grape/endpoint.rb:218:in `call'",
"grape (2.0.0) lib/grape/router/route.rb:58:in `exec'",
"grape (2.0.0) lib/grape/router.rb:120:in `process_route'",
"grape (2.0.0) lib/grape/router.rb:74:in `block in identity'",
"grape (2.0.0) lib/grape/router.rb:94:in `transaction'",
"grape (2.0.0) lib/grape/router.rb:72:in `identity'",
"grape (2.0.0) lib/grape/router.rb:56:in `block in call'",
"grape (2.0.0) lib/grape/router.rb:136:in `with_optimization'",
"grape (2.0.0) lib/grape/router.rb:55:in `call'",
"grape (2.0.0) lib/grape/api/instance.rb:165:in `call'",
"grape (2.0.0) lib/grape/api/instance.rb:70:in `call!'",
"grape (2.0.0) lib/grape/api/instance.rb:65:in `call'",
"actionpack (7.2.3) lib/action_dispatch/routing/mapper.rb:33:in `block in <class:Constraints>'",
"actionpack (7.2.3) lib/action_dispatch/routing/mapper.rb:62:in `serve'",
"actionpack (7.2.3) lib/action_dispatch/journey/router.rb:53:in `block in serve'",
"config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'",
"config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
"config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
"actionpack (7.2.3) lib/action_dispatch/journey/router.rb:34:in `serve'",
"actionpack (7.2.3) lib/action_dispatch/routing/route_set.rb:896:in `call'",
"gitlab-experiment (1.2.0) lib/gitlab/experiment/middleware.rb:19:in `call'",
"omniauth (2.1.4) lib/omniauth/strategy.rb:478:in `call_app!'",
"omniauth-saml (2.2.4) lib/omniauth/strategies/saml.rb:83:in `other_phase'",
"omniauth (2.1.4) lib/omniauth/strategy.rb:195:in `call!'",
"omniauth (2.1.4) lib/omniauth/strategy.rb:169:in `call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
"lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/go.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:94:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/ci/job_token/middleware.rb:11:in `call'",
"batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
"rack-attack (6.8.0) lib/rack/attack.rb:105:in `call'",
"apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
"lib/gitlab/middleware/multipart.rb:177:in `call'",
"lib/gitlab/middleware/rack_attack_headers.rb:42:in `call'",
"rack-attack (6.8.0) lib/rack/attack.rb:129:in `call'",
"warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
"warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
"warden (1.2.9) lib/warden/manager.rb:34:in `call'",
"rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
"rack (2.2.21) lib/rack/tempfile_reaper.rb:15:in `call'",
"rack (2.2.21) lib/rack/etag.rb:27:in `call'",
"rack (2.2.21) lib/rack/conditional_get.rb:40:in `call'",
"rack (2.2.21) lib/rack/head.rb:12:in `call'",
"actionpack (7.2.3) lib/action_dispatch/http/permissions_policy.rb:38:in `call'",
"actionpack (7.2.3) lib/action_dispatch/http/content_security_policy.rb:38:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:40:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"rack (2.2.21) lib/rack/session/abstract/id.rb:266:in `context'",
"rack (2.2.21) lib/rack/session/abstract/id.rb:260:in `call'",
"lib/gitlab/middleware/secure_headers.rb:11:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/cookies.rb:704:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/callbacks.rb:31:in `block in call'",
"activesupport (7.2.3) lib/active_support/callbacks.rb:101:in `run_callbacks'",
"actionpack (7.2.3) lib/action_dispatch/middleware/callbacks.rb:30:in `call'",
"sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/debug_exceptions.rb:31:in `call'",
"lib/gitlab/middleware/path_depth_check.rb:32:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:40:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:19:in `call'",
"lib/gitlab/middleware/json_validation.rb:189:in `allow_if_validated'",
"lib/gitlab/middleware/json_validation.rb:170:in `call'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/show_exceptions.rb:32:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
"railties (7.2.3) lib/rails/rack/logger.rb:29:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/remote_ip.rb:96:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'",
"rack (2.2.21) lib/rack/method_override.rb:24:in `call'",
"rack (2.2.21) lib/rack/runtime.rb:22:in `call'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
"rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/executor.rb:16:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"rack (2.2.21) lib/rack/sendfile.rb:127:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
"gitlab-labkit (1.0.1) lib/labkit/middleware/rack.rb:22:in `block in call'",
"gitlab-labkit (1.0.1) lib/labkit/context.rb:43:in `with_context'",
"gitlab-labkit (1.0.1) lib/labkit/middleware/rack.rb:21:in `call'",
"actionpack (7.2.3) lib/action_dispatch/middleware/request_id.rb:33:in `call'",
"lib/gitlab/middleware/static_assets_authorization.rb:23:in `call'",
"railties (7.2.3) lib/rails/engine.rb:535:in `call'",
"railties (7.2.3) lib/rails/railtie.rb:226:in `public_send'",
"railties (7.2.3) lib/rails/railtie.rb:226:in `method_missing'",
"lib/gitlab/middleware/release_env.rb:12:in `call'",
"rack (2.2.21) lib/rack/urlmap.rb:74:in `block in call'",
"rack (2.2.21) lib/rack/urlmap.rb:58:in `each'",
"rack (2.2.21) lib/rack/urlmap.rb:58:in `call'",
"puma (7.1.0) lib/puma/configuration.rb:300:in `call'",
"puma (7.1.0) lib/puma/request.rb:101:in `block in handle_request'",
"puma (7.1.0) lib/puma/thread_pool.rb:355:in `with_force_shutdown'",
"puma (7.1.0) lib/puma/request.rb:100:in `handle_request'",
"puma (7.1.0) lib/puma/server.rb:503:in `process_client'",
"puma (7.1.0) lib/puma/server.rb:262:in `block in run'",
"puma (7.1.0) lib/puma/thread_pool.rb:182:in `block in spawn_thread'"
],
"user.username": null,
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "runner_core",
"tags.correlation_id": "<REDACTED>",
"@source": "k8s",
"@component": "k8s",
"params": null,
"command": null,
"error": null,
"env": null,
"limit": null,
"ref": null,
"id": null,
"authorized_projects_refresh.rows_added_slice": null,
"details": null,
"job_artifact_id": null,
"project_id": null,
"meta.indexing.identifier": null,
"@hostname": "<REDACTED>",
"fluentd_tag": "<REDACTED>",
"@timestamp": "2026-03-05T12:51:14.199467222Z"
},
"fields": {
"@timestamp": [
"2026-03-05T12:51:14.199Z"
],
"time": [
"2026-03-05T12:51:14.199Z"
]
}
}
Edited by 🤖 GitLab Bot 🤖