CI_JOB_TOKEN access change on container push in v18
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
After upgrading to Gitlab 18 (we are currently using 18.2.4-ee, but this started happening when we jumped from 17.x to 18.0), we now experience a 500 error when attempting to push a container on https://gitlab.example.com/jwt/auth?scope=repository%3Aexample-group%2Fexample-subgroup%2Fexample-project%3Apush%2Cpull&service=container_registry url (replaced our domain with gitlab.example.com). This happens with a Pipeline that was created by a person with developer access and a user with Reporter access kicks off the manual job (based on protected environments rules).
We are using crane to copy the image, so it keeps the same digest.
Steps to reproduce
It is pretty complicated to reproduce, but I will outline the steps
- Create a pipeline with 2 environments (NP, PROD)
- In environment job, tag docker with environment name (so this tag isn't immutable, but gives us a quick reference on what image is where, and it helps for container cleanup policy)
- Have reporter kick off the manual job.
What is the current bug behavior?
500 error from gitlab's https://gitlab.example.com/jwt/auth?scope=repository%3Aexample-group%2Fexample-subgroup%2Fexample-project%3Apush%2Cpull&service=container_registry url.
Based on the documentation on https://docs.gitlab.com/ci/jobs/ci_job_token/ (and how it worked in gitlab 17) I assume it would still work for Reporters to kick off the manual job (see highlighted area, and in my case a person with developer access created the pipeline, but fails on the Reporter running the manual job):
Use a CI/CD job token to authenticate with certain GitLab features from running jobs. The token receives the
same access level as the user that triggered the pipeline, but has access to fewer resources than a personal access token. A user can cause a job to run with an action like pushing a commit, triggering a manual job, or being the owner of a scheduled pipeline. This user must have a role that has the required privileges to access the resources.
What is the expected correct behavior?
Issue appears to happen from the new feature checking for immutable container tags. Based on the stacktrace, the code here https://gitlab.com/gitlab-org/gitlab/-/blob/v18.2.4-ee/app/services/auth/container_registry_authentication_service.rb?ref_type=tags#L405
Either of these (or both):
- Update the docs on https://docs.gitlab.com/ci/jobs/ci_job_token/ to make it more clear on how
CI_JOB_TOKENis used for manual ran jobs in a pipeline. - Fix the issue so access still works based on the person who kicked off the pipeline.
Relevant logs and/or screenshots
Here is a snippet from the pipeline log
2025/08/25 19:11:18 Copying from containers.example.com/example-group/example-subgroup/example-project:0e14ff03 to containers.example.com/example-group/example-subgroup/example-project:prod
2025/08/25 19:11:18 retrying GET https://gitlab.example.com/jwt/auth?scope=repository%3Aexample-group%2Fexample-subgroup%2Fexample-project%3Apush%2Cpull&service=container_registry: unexpected status code 500 Internal Server Error: <!DOCTYPE html>Here is a snippet from gitlab logs
{
"component": "gitlab",
"subcomponent": "production_json",
"method": "GET",
"path": "/jwt/auth",
"format": "html",
"controller": "JwtController",
"action": "auth",
"status": 500,
"time": "2025-08-25T14:18:09.772Z",
"params": [{
"key": "scope",
"value": "repository:example-group/example-subgroup/example-project:push,pull"
}, {
"key": "service",
"value": "container_registry"
}
],
"correlation_id": "01K3GSFXWYGT55WBAEYGVPWVAS",
"meta.caller_id": "JwtController#auth",
"meta.feature_category": "container_registry",
"meta.organization_id": 1,
"meta.remote_ip": "10.0.0.240",
"meta.user": "reporteruser",
"meta.user_id": 955,
"meta.client_id": "user/955",
"remote_ip": "10.0.0.240",
"user_id": 955,
"username": "reporteruser",
"ua": "crane/0.19.1 go-containerregistry/0.19.1",
"queue_duration_s": 0.007625,
"request_urgency": "low",
"target_duration_s": 5,
"redis_calls": 7,
"redis_duration_s": 0.002124,
"redis_read_bytes": 33,
"redis_write_bytes": 422,
"redis_shared_state_calls": 7,
"redis_shared_state_duration_s": 0.002124,
"redis_shared_state_read_bytes": 33,
"redis_shared_state_write_bytes": 422,
"db_count": 26,
"db_write_count": 0,
"db_cached_count": 8,
"db_txn_count": 0,
"db_replica_txn_count": 0,
"db_primary_txn_count": 0,
"db_replica_count": 0,
"db_primary_count": 26,
"db_replica_write_count": 0,
"db_primary_write_count": 0,
"db_replica_cached_count": 0,
"db_primary_cached_count": 8,
"db_replica_wal_count": 0,
"db_primary_wal_count": 0,
"db_replica_wal_cached_count": 0,
"db_primary_wal_cached_count": 0,
"db_replica_txn_max_duration_s": 0.0,
"db_primary_txn_max_duration_s": 0.0,
"db_replica_txn_duration_s": 0.0,
"db_primary_txn_duration_s": 0.0,
"db_replica_duration_s": 0.0,
"db_primary_duration_s": 0.017,
"db_main_txn_count": 0,
"db_ci_txn_count": 0,
"db_main_replica_txn_count": 0,
"db_ci_replica_txn_count": 0,
"db_main_count": 25,
"db_ci_count": 1,
"db_main_replica_count": 0,
"db_ci_replica_count": 0,
"db_main_write_count": 0,
"db_ci_write_count": 0,
"db_main_replica_write_count": 0,
"db_ci_replica_write_count": 0,
"db_main_cached_count": 8,
"db_ci_cached_count": 0,
"db_main_replica_cached_count": 0,
"db_ci_replica_cached_count": 0,
"db_main_wal_count": 0,
"db_ci_wal_count": 0,
"db_main_replica_wal_count": 0,
"db_ci_replica_wal_count": 0,
"db_main_wal_cached_count": 0,
"db_ci_wal_cached_count": 0,
"db_main_replica_wal_cached_count": 0,
"db_ci_replica_wal_cached_count": 0,
"db_main_txn_max_duration_s": 0.0,
"db_ci_txn_max_duration_s": 0.0,
"db_main_replica_txn_max_duration_s": 0.0,
"db_ci_replica_txn_max_duration_s": 0.0,
"db_main_txn_duration_s": 0.0,
"db_ci_txn_duration_s": 0.0,
"db_main_replica_txn_duration_s": 0.0,
"db_ci_replica_txn_duration_s": 0.0,
"db_main_duration_s": 0.015,
"db_ci_duration_s": 0.002,
"db_main_replica_duration_s": 0.0,
"db_ci_replica_duration_s": 0.0,
"cpu_s": 0.059788,
"mem_objects": 23696,
"mem_bytes": 3487928,
"mem_mallocs": 5869,
"mem_total_bytes": 4435768,
"pid": 6063,
"worker_id": "puma_0",
"rate_limiting_gates": [],
"exception.class": "ArgumentError",
"exception.message": "Unauthorized",
"exception.backtrace": [
"app/services/auth/container_registry_authentication_service.rb:405:in `protection_rule_for_push_exists?'",
"app/services/auth/container_registry_authentication_service.rb:390:in `block in repository_path_push_protected?'",
"app/services/auth/container_registry_authentication_service.rb:383:in `any?'",
"app/services/auth/container_registry_authentication_service.rb:383:in `repository_path_push_protected?'",
"app/services/auth/container_registry_authentication_service.rb:30:in `execute'",
"ee/app/services/ee/auth/container_registry_authentication_service.rb:13:in `execute'",
"app/controllers/jwt_controller.rb:29:in `auth'",
"actionpack (7.1.5.1) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'",
"actionpack (7.1.5.1) lib/abstract_controller/base.rb:224:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/rendering.rb:165:in `process_action'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:259:in `block in process_action'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:121:in `block in run_callbacks'",
"app/controllers/jwt_controller.rb:128:in `bypass_admin_mode!'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"app/controllers/application_controller.rb:493:in `set_current_admin'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"lib/gitlab/i18n.rb:116:in `with_locale'",
"app/controllers/application_controller.rb:477:in `set_locale'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"app/controllers/application_controller.rb:464:in `set_current_context'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"lib/gitlab/ip_address_state.rb:11:in `with'",
"app/controllers/application_controller.rb:470:in `set_current_ip_address'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span'",
"sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:141:in `run_callbacks'",
"actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:258:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/rescue.rb:25:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `block in instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
"activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `instrument'",
"actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:73:in `process_action'",
"actionpack (7.1.5.1) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'",
"activerecord (7.1.5.1) lib/active_record/railties/controller_runtime.rb:32:in `process_action'",
"actionpack (7.1.5.1) lib/abstract_controller/base.rb:160:in `process'",
"actionview (7.1.5.1) lib/action_view/rendering.rb:40:in `process'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:227:in `dispatch'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `block in dispatch'",
"lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'",
"actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:51:in `block in serve'",
"config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'",
"config/initializers/action_dispatch_journey_router.rb:25:in `map!'",
"config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'",
"actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'",
"actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:882:in `call'",
"gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'",
"omniauth (2.1.2) lib/omniauth/strategy.rb:202:in `call!'",
"omniauth (2.1.2) lib/omniauth/strategy.rb:169:in `call'",
"omniauth (2.1.2) lib/omniauth/strategy.rb:202:in `call!'",
"omniauth (2.1.2) lib/omniauth/strategy.rb:169:in `call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'",
"flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'",
"lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'",
"lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/go.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:83:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/ci/job_token/middleware.rb:11:in `call'",
"batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'",
"apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'",
"warden (1.2.9) lib/warden/manager.rb:36:in `block in call'",
"warden (1.2.9) lib/warden/manager.rb:34:in `catch'",
"warden (1.2.9) lib/warden/manager.rb:34:in `call'",
"rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'",
"rack (2.2.13) lib/rack/tempfile_reaper.rb:15:in `call'",
"rack (2.2.13) lib/rack/etag.rb:27:in `call'",
"rack (2.2.13) lib/rack/conditional_get.rb:27:in `call'",
"rack (2.2.13) lib/rack/head.rb:12:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/permissions_policy.rb:36:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"rack (2.2.13) lib/rack/session/abstract/id.rb:266:in `context'",
"rack (2.2.13) lib/rack/session/abstract/id.rb:260:in `call'",
"lib/gitlab/middleware/secure_headers.rb:11:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/callbacks.rb:101:in `run_callbacks'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'",
"sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'",
"sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'",
"sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `block in call'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `block in tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:39:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `tagged'",
"activesupport (7.1.5.1) lib/active_support/broadcast_logger.rb:241:in `method_missing'",
"railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'",
"rack (2.2.13) lib/rack/method_override.rb:24:in `call'",
"rack (2.2.13) lib/rack/runtime.rb:22:in `call'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'",
"rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'",
"rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:22:in `block in call'",
"gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'",
"gitlab-labkit (0.37.0) lib/labkit/middleware/rack.rb:21:in `call'",
"rack (2.2.13) lib/rack/sendfile.rb:110:in `call'",
"actionpack (7.1.5.1) lib/action_dispatch/middleware/request_id.rb:28:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"railties (7.1.5.1) lib/rails/engine.rb:536:in `call'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `public_send'",
"railties (7.1.5.1) lib/rails/railtie.rb:226:in `method_missing'",
"lib/gitlab/middleware/release_env.rb:12:in `call'",
"rack (2.2.13) lib/rack/urlmap.rb:74:in `block in call'",
"rack (2.2.13) lib/rack/urlmap.rb:58:in `each'",
"rack (2.2.13) lib/rack/urlmap.rb:58:in `call'",
"puma (6.6.0) lib/puma/configuration.rb:279:in `call'",
"puma (6.6.0) lib/puma/request.rb:99:in `block in handle_request'",
"puma (6.6.0) lib/puma/thread_pool.rb:390:in `with_force_shutdown'",
"puma (6.6.0) lib/puma/request.rb:98:in `handle_request'",
"puma (6.6.0) lib/puma/server.rb:472:in `process_client'",
"puma (6.6.0) lib/puma/server.rb:254:in `block in run'",
"puma (6.6.0) lib/puma/thread_pool.rb:167:in `block in spawn_thread'"],
"db_duration_s": 0.01714,
"view_duration_s": 0.0,
"duration_s": 0.06742
}
Possible fixes
One of these?
- update docs https://docs.gitlab.com/ci/jobs/ci_job_token/
- update code so it still works with person who kicked off the pipeline. See code here: https://gitlab.com/gitlab-org/gitlab/-/blob/v18.2.4-ee/app/services/auth/container_registry_authentication_service.rb?ref_type=tags#L405
Edited by 🤖 GitLab Bot 🤖