Pushing the very first image to gitlab registry using podman/skopeo gives 500
When pushing to a gitlab registry the first time (project got created and nothing is in the registry so far) using podman/skopeo, we always get a 500.
Any future pushs are successful.
Likely this seems to be the case as multiple http calls to jwt endpoint are being made, which will create the project's registry in the db and thus only the first one wins and all other fire a duplicate key constraint.
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/nginx/gitlab_access.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - - [02/Mar/2021:19:31:14 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.9.0 (13-9-stable; go1.13.8; linux/amd64)" -
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/nginx/gitlab_access.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - - [02/Mar/2021:19:31:14 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 13.9.0 (13-9-stable; go1.13.8; linux/amd64)" -
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-workhorse/current <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"","correlation_id":"01EZT6RKKAFPM3HGMZ2JVJ85XP","duration_ms":1,"host":"code.example.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":1,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 13.9.0 (13-9-stable; go1.13.8; linux/amd64)","written_bytes":0}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: Started GET "/jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry" for 127.0.0.1 at 2021-03-02 19:31:14 +0000
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: Processing by JwtController#auth as HTML
Mar 02 20:31:14 gitlab.example.com[8802]: Parameters: {"account"=>"z", "scope"=>"repository:z/myproject-pod/myproject:pull,push", "service"=>"container_registry"}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/postgresql/current <==
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81055 ERROR: duplicate key value violates unique constraint "index_container_repositories_on_project_id_and_name"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81105 DETAIL: Key (project_id, name)=(469, myproject) already exists.
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81106 STATEMENT: INSERT INTO "container_repositories" ("project_id", "name", "created_at", "updated_at") VALUES (469, 'myproject', '2021-03-02 19:31:14.727804', '2021-03-02 19:31:14.727804') RETURNING "id"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81106 ERROR: duplicate key value violates unique constraint "index_container_repositories_on_project_id_and_name"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81107 DETAIL: Key (project_id, name)=(469, myproject) already exists.
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81107 STATEMENT: INSERT INTO "container_repositories" ("project_id", "name", "created_at", "updated_at") VALUES (469, 'myproject', '2021-03-02 19:31:14.724177', '2021-03-02 19:31:14.724177') RETURNING "id"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81107 ERROR: duplicate key value violates unique constraint "index_container_repositories_on_project_id_and_name"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81108 DETAIL: Key (project_id, name)=(469, myproject) already exists.
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81108 STATEMENT: INSERT INTO "container_repositories" ("project_id", "name", "created_at", "updated_at") VALUES (469, 'myproject', '2021-03-02 19:31:14.728108', '2021-03-02 19:31:14.728108') RETURNING "id"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81110 ERROR: duplicate key value violates unique constraint "index_container_repositories_on_project_id_and_name"
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81110 DETAIL: Key (project_id, name)=(469, myproject) already exists.
Mar 02 20:31:14 gitlab.example.com[8802]: 2021-03-02_19:31:14.81110 STATEMENT: INSERT INTO "container_repositories" ("project_id", "name", "created_at", "updated_at") VALUES (469, 'myproject', '2021-03-02 19:31:14.707427', '2021-03-02 19:31:14.707427') RETURNING "id"
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: Completed 500 Internal Server Error in 123ms (ActiveRecord: 96.2ms | Elasticsearch: 0.0ms | Allocations: 10708)
Mar 02 20:31:14 gitlab.example.com[8802]: Completed 500 Internal Server Error in 142ms (ActiveRecord: 105.2ms | Elasticsearch: 0.0ms | Allocations: 15159)
--
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/transaction.rb:56:in `run'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/request_profiler/middleware.rb:17:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/jira/middleware.rb:19:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/go.rb:20:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/etag_caching/middleware.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/multipart.rb:172:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only/controller.rb:50:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only.rb:18:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/basic_health_check.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/request_context.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: config/initializers/fix_local_cache_middleware.rb:11:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/release_env.rb:12:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production_json.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":500,"time":"2021-03-02T19:31:14.814Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKKPD0AXJ3HN9MQPX5SQ","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":3,"redis_duration_s":0.001013,"redis_read_bytes":607,"redis_write_bytes":165,"redis_cache_calls":3,"redis_cache_duration_s":0.001013,"redis_cache_read_bytes":607,"redis_cache_write_bytes":165,"db_count":10,"db_write_count":2,"db_cached_count":0,"cpu_s":0.035797,"queue_duration_s":0.008919,"exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_container_repositories_on_project_id_and_name\"\nDETAIL: Key (project_id, name)=(469, myproject) already exists.\n","exception.backtrace":["app/models/container_repository.rb:130:in `tap'","app/models/container_repository.rb:130:in `create_from_path!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'","app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'","app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'","app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'","app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'","app/services/auth/container_registry_authentication_service.rb:73:in `map'","app/services/auth/container_registry_authentication_service.rb:73:in `scopes'","lib/gitlab/metrics/instrumentation.rb:160:in `block in scopes'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `scopes'","app/services/auth/container_registry_authentication_service.rb:24:in `execute'","lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `execute'","app/controllers/jwt_controller.rb:23:in `auth'","app/controllers/application_controller.rb:482:in `set_current_admin'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:467:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:532:in `sentry_context'","app/controllers/application_controller.rb:460:in `block in set_current_context'","lib/gitlab/application_context.rb:56:in `block in use'","lib/gitlab/application_context.rb:56:in `use'","lib/gitlab/application_context.rb:22:in `with_context'","app/controllers/application_controller.rb:451:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.09625,"view_duration_s":0.0,"duration_s":0.12324}
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":500,"time":"2021-03-02T19:31:14.814Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKKAFPM3HGMZ2M2VPJBR","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":4,"redis_duration_s":0.002224,"redis_read_bytes":810,"redis_write_bytes":224,"redis_cache_calls":4,"redis_cache_duration_s":0.002224,"redis_cache_read_bytes":810,"redis_cache_write_bytes":224,"db_count":11,"db_write_count":2,"db_cached_count":0,"cpu_s":0.051319,"queue_duration_s":0.012235,"exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_container_repositories_on_project_id_and_name\"\nDETAIL: Key (project_id, name)=(469, myproject) already exists.\n","exception.backtrace":["app/models/container_repository.rb:130:in `tap'","app/models/container_repository.rb:130:in `create_from_path!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'","app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'","app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'","app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'","app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'","app/services/auth/container_registry_authentication_service.rb:73:in `map'","app/services/auth/container_registry_authentication_service.rb:73:in `scopes'","lib/gitlab/metrics/instrumentation.rb:160:in `block in scopes'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `scopes'","app/services/auth/container_registry_authentication_service.rb:24:in `execute'","lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `execute'","app/controllers/jwt_controller.rb:23:in `auth'","app/controllers/application_controller.rb:482:in `set_current_admin'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:467:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:532:in `sentry_context'","app/controllers/application_controller.rb:460:in `block in set_current_context'","lib/gitlab/application_context.rb:56:in `block in use'","lib/gitlab/application_context.rb:56:in `use'","lib/gitlab/application_context.rb:22:in `with_context'","app/controllers/application_controller.rb:451:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.09452,"view_duration_s":0.0,"duration_s":0.13167}
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":500,"time":"2021-03-02T19:31:14.814Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKK6CGWE3SCTF5PVPQYC","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":3,"redis_duration_s":0.000751,"redis_read_bytes":607,"redis_write_bytes":165,"redis_cache_calls":3,"redis_cache_duration_s":0.000751,"redis_cache_read_bytes":607,"redis_cache_write_bytes":165,"db_count":10,"db_write_count":2,"db_cached_count":0,"cpu_s":0.047058,"queue_duration_s":0.005092,"exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_container_repositories_on_project_id_and_name\"\nDETAIL: Key (project_id, name)=(469, myproject) already exists.\n","exception.backtrace":["app/models/container_repository.rb:130:in `tap'","app/models/container_repository.rb:130:in `create_from_path!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'","app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'","app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'","app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'","app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'","app/services/auth/container_registry_authentication_service.rb:73:in `map'","app/services/auth/container_registry_authentication_service.rb:73:in `scopes'","lib/gitlab/metrics/instrumentation.rb:160:in `block in scopes'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `scopes'","app/services/auth/container_registry_authentication_service.rb:24:in `execute'","lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `execute'","app/controllers/jwt_controller.rb:23:in `auth'","app/controllers/application_controller.rb:482:in `set_current_admin'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:467:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:532:in `sentry_context'","app/controllers/application_controller.rb:460:in `block in set_current_context'","lib/gitlab/application_context.rb:56:in `block in use'","lib/gitlab/application_context.rb:56:in `use'","lib/gitlab/application_context.rb:22:in `with_context'","app/controllers/application_controller.rb:451:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.10518,"view_duration_s":0.0,"duration_s":0.14248}
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":500,"time":"2021-03-02T19:31:14.814Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKJX7HXZ7XGDA28YMWKB","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":5,"redis_duration_s":0.009824,"redis_read_bytes":973,"redis_write_bytes":255,"redis_cache_calls":5,"redis_cache_duration_s":0.009824,"redis_cache_read_bytes":973,"redis_cache_write_bytes":255,"db_count":11,"db_write_count":2,"db_cached_count":0,"cpu_s":0.041593,"queue_duration_s":0.010624,"exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR: duplicate key value violates unique constraint \"index_container_repositories_on_project_id_and_name\"\nDETAIL: Key (project_id, name)=(469, myproject) already exists.\n","exception.backtrace":["app/models/container_repository.rb:130:in `tap'","app/models/container_repository.rb:130:in `create_from_path!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'","app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'","app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'","app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'","app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'","app/services/auth/container_registry_authentication_service.rb:73:in `map'","app/services/auth/container_registry_authentication_service.rb:73:in `scopes'","lib/gitlab/metrics/instrumentation.rb:160:in `block in scopes'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `scopes'","app/services/auth/container_registry_authentication_service.rb:24:in `execute'","lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `execute'","app/controllers/jwt_controller.rb:23:in `auth'","app/controllers/application_controller.rb:482:in `set_current_admin'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:467:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:532:in `sentry_context'","app/controllers/application_controller.rb:460:in `block in set_current_context'","lib/gitlab/application_context.rb:56:in `block in use'","lib/gitlab/application_context.rb:56:in `use'","lib/gitlab/application_context.rb:22:in `with_context'","app/controllers/application_controller.rb:451:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.11015,"view_duration_s":0.0,"duration_s":0.14648}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_container_repositories_on_project_id_and_name"
Mar 02 20:31:14 gitlab.example.com[8802]: DETAIL: Key (project_id, name)=(469, myproject) already exists.
Mar 02 20:31:14 gitlab.example.com[8802]: ):
Mar 02 20:31:14 gitlab.example.com[8802]: app/models/container_repository.rb:130:in `tap'
Mar 02 20:31:14 gitlab.example.com[8802]: app/models/container_repository.rb:130:in `create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
--
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/transaction.rb:56:in `run'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/request_profiler/middleware.rb:17:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/jira/middleware.rb:19:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/go.rb:20:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/etag_caching/middleware.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/multipart.rb:172:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only/controller.rb:50:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only.rb:18:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/basic_health_check.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/request_context.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: config/initializers/fix_local_cache_middleware.rb:11:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/release_env.rb:12:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: Completed 200 OK in 157ms (Views: 0.3ms | ActiveRecord: 120.4ms | Elasticsearch: 0.0ms | Allocations: 11011)
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production_json.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2021-03-02T19:31:14.819Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKJX7HXZ7XGDA37Y7SFA","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":3,"redis_duration_s":0.000646,"redis_read_bytes":607,"redis_write_bytes":165,"redis_cache_calls":3,"redis_cache_duration_s":0.000646,"redis_cache_read_bytes":607,"redis_cache_write_bytes":165,"db_count":9,"db_write_count":1,"db_cached_count":0,"cpu_s":0.045578,"queue_duration_s":0.00459,"db_duration_s":0.12035,"view_duration_s":0.00026,"duration_s":0.1569}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-workhorse/current <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"application/json; charset=utf-8","correlation_id":"01EZT6RKJX7HXZ7XGDA37Y7SFA","duration_ms":167,"host":"code.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":167,"uri":"/jwt/auth?account=z\u0026scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush\u0026service=container_registry","user_agent":"libpod/3.0.1","written_bytes":1160}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/nginx/gitlab_access.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - z [02/Mar/2021:19:31:14 +0000] "GET /jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry HTTP/1.1" 200 1160 "" "libpod/3.0.1" -
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: Completed 422 Unprocessable Entity in 151ms (ActiveRecord: 110.0ms | Elasticsearch: 0.0ms | Allocations: 16409)
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production_json.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":422,"time":"2021-03-02T19:31:14.837Z","params":[{"key":"account","value":"z"},{"key":"scope","value":"repository:z/myproject-pod/myproject:pull,push"},{"key":"service","value":"container_registry"}],"remote_ip":"127.0.0.1","user_id":61,"username":"z","ua":"libpod/3.0.1","correlation_id":"01EZT6RKKHJ9VQ2JYNW3NTFH6C","meta.user":"z","meta.caller_id":"JwtController#auth","meta.remote_ip":"127.0.0.1","meta.feature_category":"authentication_and_authorization","redis_calls":4,"redis_duration_s":0.003097,"redis_read_bytes":810,"redis_write_bytes":224,"redis_cache_calls":4,"redis_cache_duration_s":0.003097,"redis_cache_read_bytes":810,"redis_cache_write_bytes":224,"db_count":9,"db_write_count":1,"db_cached_count":0,"cpu_s":0.047797,"queue_duration_s":0.008891,"exception.class":"ActiveRecord::RecordInvalid","exception.message":"Validation failed: Name has already been taken","exception.backtrace":["app/models/container_repository.rb:130:in `tap'","app/models/container_repository.rb:130:in `create_from_path!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'","app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'","lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'","app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'","app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'","lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'","app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'","app/services/auth/container_registry_authentication_service.rb:73:in `map'","app/services/auth/container_registry_authentication_service.rb:73:in `scopes'","lib/gitlab/metrics/instrumentation.rb:160:in `block in scopes'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `scopes'","app/services/auth/container_registry_authentication_service.rb:24:in `execute'","lib/gitlab/metrics/instrumentation.rb:160:in `block in execute'","lib/gitlab/metrics/method_call.rb:27:in `measure'","lib/gitlab/metrics/instrumentation.rb:160:in `execute'","app/controllers/jwt_controller.rb:23:in `auth'","app/controllers/application_controller.rb:482:in `set_current_admin'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:467:in `set_locale'","lib/gitlab/error_tracking.rb:52:in `with_context'","app/controllers/application_controller.rb:532:in `sentry_context'","app/controllers/application_controller.rb:460:in `block in set_current_context'","lib/gitlab/application_context.rb:56:in `block in use'","lib/gitlab/application_context.rb:56:in `use'","lib/gitlab/application_context.rb:22:in `with_context'","app/controllers/application_controller.rb:451:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.11002,"view_duration_s":0.0,"duration_s":0.15139}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-rails/production.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: ActiveRecord::RecordInvalid (Validation failed: Name has already been taken):
Mar 02 20:31:14 gitlab.example.com[8802]: app/models/container_repository.rb:130:in `tap'
Mar 02 20:31:14 gitlab.example.com[8802]: app/models/container_repository.rb:130:in `create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `create_from_path!'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:130:in `ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `ensure_container_repository!'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:116:in `process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `process_repository_access'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:87:in `process_scope'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `block in process_scope'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/method_call.rb:27:in `measure'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/instrumentation.rb:160:in `process_scope'
Mar 02 20:31:14 gitlab.example.com[8802]: app/services/auth/container_registry_authentication_service.rb:74:in `block in scopes'
--
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/transaction.rb:56:in `run'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/rack_middleware.rb:16:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/request_profiler/middleware.rb:17:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/jira/middleware.rb:19:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/go.rb:20:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/etag_caching/middleware.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/multipart.rb:172:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only/controller.rb:50:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/read_only.rb:18:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/basic_health_check.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/request_context.rb:21:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: config/initializers/fix_local_cache_middleware.rb:11:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: lib/gitlab/middleware/release_env.rb:12:in `call'
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-workhorse/current <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"text/html; charset=utf-8","correlation_id":"01EZT6RKKAFPM3HGMZ2M2VPJBR","duration_ms":184,"host":"code.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":184,"uri":"/jwt/auth?account=z\u0026scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush\u0026service=container_registry","user_agent":"libpod/3.0.1","written_bytes":2926}
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"text/html; charset=utf-8","correlation_id":"01EZT6RKKPD0AXJ3HN9MQPX5SQ","duration_ms":172,"host":"code.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":172,"uri":"/jwt/auth?account=z\u0026scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush\u0026service=container_registry","user_agent":"libpod/3.0.1","written_bytes":2926}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/nginx/gitlab_access.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - z [02/Mar/2021:19:31:14 +0000] "GET /jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry HTTP/1.1" 500 2926 "" "libpod/3.0.1" -
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - z [02/Mar/2021:19:31:14 +0000] "GET /jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry HTTP/1.1" 500 2926 "" "libpod/3.0.1" -
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - z [02/Mar/2021:19:31:14 +0000] "GET /jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry HTTP/1.1" 500 2926 "" "libpod/3.0.1" -
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/gitlab-workhorse/current <==
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"text/html; charset=utf-8","correlation_id":"01EZT6RKK6CGWE3SCTF5PVPQYC","duration_ms":188,"host":"code.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":188,"uri":"/jwt/auth?account=z\u0026scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush\u0026service=container_registry","user_agent":"libpod/3.0.1","written_bytes":2926}
Mar 02 20:31:14 gitlab.example.com[8802]: {"content_type":"text/html; charset=utf-8","correlation_id":"01EZT6RKJX7HXZ7XGDA28YMWKB","duration_ms":197,"host":"code.example.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-03-02T19:31:14Z","ttfb_ms":197,"uri":"/jwt/auth?account=z\u0026scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush\u0026service=container_registry","user_agent":"libpod/3.0.1","written_bytes":2926}
Mar 02 20:31:14 gitlab.example.com[8802]: ==> /var/log/gitlab/nginx/gitlab_access.log <==
Mar 02 20:31:14 gitlab.example.com[8802]: 192.168.1.56 - z [02/Mar/2021:19:31:14 +0000] "GET /jwt/auth?account=z&scope=repository%3Az%2Fmyproject-pod%2Fmyproject%3Apull%2Cpush&service=container_registry HTTP/1.1" 500 2926 "" "libpod/3.0.1" -
Latest greatest gitlab: gitlab-ce-13.9.1-ce.0.el7.x86_64
This issues is known for a while and there for at least 1-2 years.