find 500 Internal Server Error when get groups API Endpotin: POST /groups/{id}/hook
Summary
500 Internal Server Error when get groups API Endpotin: POST /groups/{id}/hook
Steps to reproduce
-
curl --location --request POST 'http://192.168.1.123:30000/api/v4/groups?name=dcR&path=VJurK' --header 'Authorization: Bearer xxx'
-
the group_id created in step 1
curl --location --request DELETE 'http://192.168.1.123:30000/api/v4/groups/23482' --header 'Authorization: Bearer xxx
-
the group_id created in step 1
curl --location --request POST 'http://192.168.1.123:30000/api/v4/groups/23482/hooks?url=http://rpba.co.uk/tN8NTpE7-R/' --header 'Authorization: Bearer xxx' \
--form 'url="http://ngob.com/"'
Example Project
The bug is reproducible on our local deployment of enterprise edition (14.4.2-ee).
What is the current bug behavior?
HTTP Response: "500 Internal Server Error."
What is the expected correct behavior?
HTTP Response: "404 Not Found"
Relevant logs and/or screenshots
==> /var/log/gitlab/gitlab-rails/api_json.log <==
{"time":"2022-10-23T07:03:17.760Z","severity":"INFO","duration_s":0.27737,"db_duration_s":0.18602,"view_duration_s":0.09135,"status":201,"method":"POST","path":"/api/v4/groups","params":[{"key":"name","value":"dcR"},{"key":"path","value":"VJurK"}],"host":"192.168.1.123","remote_ip":"172.17.0.1, 127.0.0.1","ua":"PostmanRuntime/7.29.2","route":"/api/:version/groups","user_id":1,"username":"root","queue_duration_s":0.01425,"redis_calls":17,"redis_duration_s":0.004564,"redis_read_bytes":1936,"redis_write_bytes":3252,"redis_cache_calls":10,"redis_cache_duration_s":0.002559,"redis_cache_read_bytes":1863,"redis_cache_write_bytes":640,"redis_queues_calls":2,"redis_queues_duration_s":0.000648,"redis_queues_read_bytes":69,"redis_queues_write_bytes":2075,"redis_shared_state_calls":5,"redis_shared_state_duration_s":0.001357,"redis_shared_state_read_bytes":4,"redis_shared_state_write_bytes":537,"db_count":66,"db_write_count":11,"db_cached_count":11,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":66,"db_primary_cached_count":11,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.189,"cpu_s":0.111064,"mem_objects":55489,"mem_bytes":3167376,"mem_mallocs":10061,"mem_total_bytes":5386936,"pid":9122,"correlation_id":"01GG1TEA6WK9SA8AHPPVT00ABE","meta.user":"root","meta.caller_id":"POST /api/:version/groups","meta.remote_ip":"172.17.0.1","meta.feature_category":"subgroups","meta.client_id":"user/1","content_length":"0"}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:17.760Z","retry":3,"queue":"mailers","class":"ActionMailer::MailDeliveryJob","args":["[FILTERED]"],"jid":"179c791dbf3efae0ad697a90","created_at":"2022-10-23T07:03:17.720Z","meta.user":"root","meta.caller_id":"POST /api/:version/groups","meta.remote_ip":"172.17.0.1","meta.feature_category":"subgroups","meta.client_id":"user/1","correlation_id":"01GG1TEA6WK9SA8AHPPVT00ABE","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:mailers:c80ace474c4e4c2e67c846b59e7019a7f3563d6f110c2346e34c42e21e8f1b85","size_limiter":"validated","enqueued_at":"2022-10-23T07:03:17.721Z","job_size_bytes":388,"pid":572,"message":"ActionMailer::MailDeliveryJob JID-179c791dbf3efae0ad697a90: done: 0.037958 sec","job_status":"done","scheduling_latency_s":0.000964,"redis_calls":5,"redis_duration_s":0.001192,"redis_read_bytes":416,"redis_write_bytes":380,"redis_cache_calls":4,"redis_cache_duration_s":0.000969,"redis_cache_read_bytes":406,"redis_cache_write_bytes":195,"redis_queues_calls":1,"redis_queues_duration_s":0.000223,"redis_queues_read_bytes":10,"redis_queues_write_bytes":185,"db_count":10,"db_write_count":0,"db_cached_count":2,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":10,"db_primary_cached_count":2,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"cpu_s":0.030335,"mem_objects":27692,"mem_bytes":2840800,"mem_mallocs":14914,"mem_total_bytes":3948480,"duration_s":0.037958,"completed_at":"2022-10-23T07:03:17.760Z","load_balancing_strategy":"primary","db_duration_s":0.005174}
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json","correlation_id":"01GG1TEA6WK9SA8AHPPVT00ABE","duration_ms":293,"host":"192.168.1.123:30000","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":201,"system":"http","time":"2022-10-23T07:03:17Z","ttfb_ms":293,"uri":"/api/v4/groups?name=dcR\u0026path=VJurK","user_agent":"PostmanRuntime/7.29.2","written_bytes":832}
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [23/Oct/2022:07:03:17 +0000] "POST /api/v4/groups?name=dcR&path=VJurK HTTP/1.1" 201 832 "" "PostmanRuntime/7.29.2" -
==> /var/log/gitlab/gitlab-rails/production.log <==
Started DELETE "/api/v4/groups/23482" for 172.17.0.1 at 2022-10-23 07:03:17 +0000
==> /var/log/gitlab/gitlab-rails/application.log <==
2022-10-23T07:03:17.909Z: User 1 scheduled a deletion of group ID 23482 with job ID 886588e2343eb08b3b01d31b
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2022-10-23T07:03:17.909Z","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","message":"User 1 scheduled a deletion of group ID 23482 with job ID 886588e2343eb08b3b01d31b"}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:17.909Z","retry":3,"queue":"group_destroy","version":0,"backtrace":5,"class":"GroupDestroyWorker","args":["23482","1"],"jid":"886588e2343eb08b3b01d31b","created_at":"2022-10-23T07:03:17.906Z","meta.user":"root","meta.caller_id":"DELETE /api/:version/groups/:id","meta.remote_ip":"172.17.0.1","meta.feature_category":"subgroups","meta.client_id":"user/1","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:group_destroy:524e1e566e62f8e51254caf6ae5cd5ad6546c4e381793307b63f71621a799b85","size_limiter":"validated","enqueued_at":"2022-10-23T07:03:17.908Z","job_size_bytes":9,"pid":572,"message":"GroupDestroyWorker JID-886588e2343eb08b3b01d31b: start","job_status":"start","scheduling_latency_s":0.00095}
==> /var/log/gitlab/gitlab-rails/api_json.log <==
{"time":"2022-10-23T07:03:17.912Z","severity":"INFO","duration_s":0.03552,"db_duration_s":0.00834,"view_duration_s":0.02718,"status":202,"method":"DELETE","path":"/api/v4/groups/23482","params":[],"host":"192.168.1.123","remote_ip":"172.17.0.1, 127.0.0.1","ua":"PostmanRuntime/7.29.2","route":"/api/:version/groups/:id","user_id":1,"username":"root","queue_duration_s":0.015677,"redis_calls":5,"redis_duration_s":0.001833,"redis_read_bytes":379,"redis_write_bytes":1374,"redis_cache_calls":2,"redis_cache_duration_s":0.000944,"redis_cache_read_bytes":321,"redis_cache_write_bytes":184,"redis_queues_calls":1,"redis_queues_duration_s":0.000184,"redis_queues_read_bytes":58,"redis_queues_write_bytes":1036,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000705,"redis_shared_state_write_bytes":154,"db_count":15,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":15,"db_primary_cached_count":0,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.008,"cpu_s":0.042767,"mem_objects":28370,"mem_bytes":4823776,"mem_mallocs":17485,"mem_total_bytes":5958576,"pid":9840,"correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","meta.user":"root","meta.caller_id":"DELETE /api/:version/groups/:id","meta.remote_ip":"172.17.0.1","meta.feature_category":"subgroups","meta.client_id":"user/1"}
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","duration_ms":53,"host":"192.168.1.123:30000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":202,"system":"http","time":"2022-10-23T07:03:17Z","ttfb_ms":53,"uri":"/api/v4/groups/23482","user_agent":"PostmanRuntime/7.29.2","written_bytes":26}
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [23/Oct/2022:07:03:17 +0000] "DELETE /api/v4/groups/23482 HTTP/1.1" 202 26 "" "PostmanRuntime/7.29.2" -
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/groups/23482/hooks?url=http://rpba.co.uk/tN8NTpE7-R/" for 172.17.0.1 at 2022-10-23 07:03:18 +0000
==> /var/log/gitlab/gitlab-rails/application.log <==
2022-10-23T07:03:18.121Z: Group "dcR" was removed
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2022-10-23T07:03:18.122Z","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","message":"Group \"dcR\" was removed"}
==> /var/log/gitlab/gitlab-rails/application.log <==
2022-10-23T07:03:18.127Z: Cannot obtain an exclusive lease. There must be another instance already in execution.
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"ERROR","time":"2022-10-23T07:03:18.127Z","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","message":"Cannot obtain an exclusive lease. There must be another instance already in execution."}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:18.134Z","retry":3,"queue":"group_destroy","version":0,"backtrace":5,"class":"GroupDestroyWorker","args":["23482","1"],"jid":"886588e2343eb08b3b01d31b","created_at":"2022-10-23T07:03:17.906Z","meta.user":"root","meta.caller_id":"DELETE /api/:version/groups/:id","meta.remote_ip":"172.17.0.1","meta.feature_category":"subgroups","meta.client_id":"user/1","correlation_id":"01GG1TEAK5Y1DRDK242MKWR7P6","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:group_destroy:524e1e566e62f8e51254caf6ae5cd5ad6546c4e381793307b63f71621a799b85","size_limiter":"validated","enqueued_at":"2022-10-23T07:03:17.908Z","job_size_bytes":9,"pid":572,"message":"GroupDestroyWorker JID-886588e2343eb08b3b01d31b: done: 0.225385 sec","job_status":"done","scheduling_latency_s":0.00095,"redis_calls":3,"redis_duration_s":0.000697,"redis_read_bytes":213,"redis_write_bytes":340,"redis_cache_calls":1,"redis_cache_duration_s":0.000291,"redis_cache_read_bytes":203,"redis_cache_write_bytes":59,"redis_queues_calls":1,"redis_queues_duration_s":0.000214,"redis_queues_read_bytes":10,"redis_queues_write_bytes":191,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000192,"redis_shared_state_write_bytes":90,"db_count":44,"db_write_count":7,"db_cached_count":4,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":44,"db_primary_cached_count":4,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.181,"cpu_s":0.049263,"mem_objects":25693,"mem_bytes":1012472,"mem_mallocs":3790,"mem_total_bytes":2040192,"duration_s":0.225385,"completed_at":"2022-10-23T07:03:18.134Z","load_balancing_strategy":"primary","db_duration_s":0.180148}
{"severity":"INFO","time":"2022-10-23T07:03:18.910Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:18.910Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.00039460009429603815,"retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:18.910Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:18.910Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.00024159997701644897,"retry":0}
==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
{"type":"gitaly-ruby","grpc.start_time":"2022-10-23T07:03:23Z","grpc.time_ms":0.113,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":347,"correlation_id":"86573bf4ce1044931b6bcfe7d17cd88d","time":"2022-10-23T07:03:23.320Z"}
{"type":"gitaly-ruby","grpc.start_time":"2022-10-23T07:03:23Z","grpc.time_ms":0.127,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":346,"correlation_id":"6be492822815b5d8bae6ca428b51a670","time":"2022-10-23T07:03:23.320Z"}
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/help" for 127.0.0.1 at 2022-10-23 07:03:25 +0000
Processing by HelpController#index as */*
Rendered layout layouts/help.html.haml (Duration: 57.1ms | Allocations: 84545)
Completed 200 OK in 60ms (Views: 56.5ms | ActiveRecord: 1.0ms | Elasticsearch: 0.0ms | Allocations: 86263)
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/help","format":"*/*","controller":"HelpController","action":"index","status":200,"time":"2022-10-23T07:03:25.198Z","params":[],"remote_ip":"127.0.0.1","user_id":null,"username":null,"ua":"curl/7.79.1-DEV","correlation_id":"01GG1TEHNZQMVBFPB20SGF3H69","meta.caller_id":"HelpController#index","meta.remote_ip":"127.0.0.1","meta.feature_category":"not_owned","meta.client_id":"ip/127.0.0.1","redis_calls":9,"redis_duration_s":0.002023,"redis_read_bytes":873,"redis_write_bytes":603,"redis_cache_calls":9,"redis_cache_duration_s":0.002023,"redis_cache_read_bytes":873,"redis_cache_write_bytes":603,"db_count":8,"db_write_count":0,"db_cached_count":4,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":8,"db_primary_cached_count":4,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"cpu_s":0.074978,"mem_objects":95032,"mem_bytes":21981632,"mem_mallocs":105262,"mem_total_bytes":25782912,"pid":10322,"queue_duration_s":0.012837,"db_duration_s":0.00101,"view_duration_s":0.05651,"duration_s":0.06034}
==> /var/log/gitlab/nginx/gitlab_access.log <==
127.0.0.1 - - [23/Oct/2022:07:03:25 +0000] "GET /help HTTP/1.1" 200 75166 "" "curl/7.79.1-DEV" -
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01GG1TEHNZQMVBFPB20SGF3H69","duration_ms":81,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":200,"system":"http","time":"2022-10-23T07:03:25Z","ttfb_ms":81,"uri":"/help","user_agent":"curl/7.79.1-DEV","written_bytes":75075}
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:25.612Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:25.613Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0005096000386402011,"retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:25.613Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:25.613Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0003445999464020133,"retry":0}
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2022-10-23 07:03:25 +0000
Processing by MetricsController#index as HTML
Completed 200 OK in 60ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 2424)
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2022-10-23T07:03:26.012Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"redis_calls":1,"redis_duration_s":0.000455,"redis_read_bytes":203,"redis_write_bytes":114,"redis_cache_calls":1,"redis_cache_duration_s":0.000455,"redis_cache_read_bytes":203,"redis_cache_write_bytes":114,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":0,"db_primary_cached_count":0,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"cpu_s":0.062704,"mem_objects":3402,"mem_bytes":569064,"mem_mallocs":4104,"mem_total_bytes":705144,"pid":10443,"correlation_id":"564e09dd-2713-42ce-96c5-80c24f791e82","db_duration_s":0.0,"view_duration_s":0.00049,"duration_s":0.05996}
==> /var/log/gitlab/gitlab-exporter/current <==
2022-10-23_07:03:27.42287 127.0.0.1 - - [23/Oct/2022:07:03:27 UTC] "GET /sidekiq HTTP/1.1" 200 95854
2022-10-23_07:03:27.42293 - -> /sidekiq
2022-10-23_07:03:27.72259 127.0.0.1 - - [23/Oct/2022:07:03:27 UTC] "GET /database HTTP/1.1" 200 1090
2022-10-23_07:03:27.72265 - -> /database
2022-10-23_07:03:30.06634 127.0.0.1 - - [23/Oct/2022:07:03:30 UTC] "GET /ruby HTTP/1.1" 200 1005
2022-10-23_07:03:30.06640 - -> /ruby
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:30.518Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:30.519Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0004972000606358051,"retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:30.519Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:30.519Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.00038460001815110445,"retry":0}
==> /var/log/gitlab/postgresql/current <==
2022-10-23_07:03:30.68757 ERROR: insert or update on table "gitlab_subscriptions" violates foreign key constraint "fk_e2595d00a1"
2022-10-23_07:03:30.68762 DETAIL: Key (namespace_id)=(23482) is not present in table "namespaces".
2022-10-23_07:03:30.68763 STATEMENT: /*application:web,correlation_id:01GG1TEAQGKC1H7AJ763F9CQ2Y,endpoint_id:POST /api/:version/groups/:id/hooks,db_config_name:main*/ INSERT INTO "gitlab_subscriptions" ("created_at", "updated_at", "start_date", "namespace_id", "trial") VALUES ('2022-10-23 07:03:30.686763', '2022-10-23 07:03:30.686763', '2022-10-23', 23482, NULL) RETURNING "id"
==> /var/log/gitlab/gitlab-rails/exceptions_json.log <==
{"severity":"ERROR","time":"2022-10-23T07:03:30.697Z","correlation_id":"01GG1TEAQGKC1H7AJ763F9CQ2Y","exception.class":"ActiveRecord::InvalidForeignKey","exception.message":"PG::ForeignKeyViolation: ERROR: insert or update on table \"gitlab_subscriptions\" violates foreign key constraint \"fk_e2595d00a1\"\nDETAIL: Key (namespace_id)=(23482) is not present in table \"namespaces\".\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","ee/app/models/ee/namespace.rb:501:in `generate_subscription'","ee/app/models/ee/namespace.rb:494:in `find_or_create_subscription'","ee/app/models/ee/namespace.rb:185:in `block in actual_plan'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","ee/app/models/ee/namespace.rb:181:in `actual_plan'","app/models/namespace.rb:434:in `actual_limits'","app/models/concerns/limitable.rb:35:in `validate_scoped_plan_limit_not_exceeded'","app/models/concerns/limitable.rb:24:in `validate_plan_limit_not_exceeded'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","ee/lib/api/group_hooks.rb:69:in `block (2 levels) in \u003cclass:GroupHooks\u003e'","ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:21:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"user.username":"root","tags.program":"web","tags.locale":"en","tags.feature_category":"integrations","tags.correlation_id":"01GG1TEAQGKC1H7AJ763F9CQ2Y"}
==> /var/log/gitlab/gitlab-rails/production.log <==
ActiveRecord::InvalidForeignKey (PG::ForeignKeyViolation: ERROR: insert or update on table "gitlab_subscriptions" violates foreign key constraint "fk_e2595d00a1"
DETAIL: Key (namespace_id)=(23482) is not present in table "namespaces".
):
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/marginalia-1.10.0/lib/marginalia.rb:95:in `execute_and_clear_with_marginalia'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:53:in `exec_query'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `exec_insert'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `exec_insert'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:171:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:375:in `_insert_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:929:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/counter_cache.rb:166:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/locking/optimistic.rb:79:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/attribute_methods/dirty.rb:201:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:461:in `block in _create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_create_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:461:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/timestamp.rb:108:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:900:in `create_or_update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:457:in `block in create_or_update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:457:in `create_or_update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/timestamp.rb:126:in `create_or_update'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:474:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/validations.rb:47:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:298:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:298:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/suppressor.rb:44:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/associations/singular_association.rb:52:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/associations/has_one_association.rb:128:in `_create_record'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/associations/association.rb:205:in `create'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/associations/builder/singular_association.rb:33:in `create_gitlab_subscription'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/namespace.rb:501:in `generate_subscription'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/namespace.rb:494:in `find_or_create_subscription'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/namespace.rb:185:in `block in actual_plan'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/namespace.rb:181:in `actual_plan'
/opt/gitlab/embedded/service/gitlab-rails/app/models/namespace.rb:434:in `actual_limits'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/limitable.rb:35:in `validate_scoped_plan_limit_not_exceeded'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/limitable.rb:24:in `validate_plan_limit_not_exceeded'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:427:in `block in make_lambda'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:179:in `block (2 levels) in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:603:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:603:in `block in default_terminator'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:180:in `block in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:512:in `block in invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:512:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:512:in `invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:105:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_validate_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.1/lib/active_model/validations.rb:406:in `run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.1/lib/active_model/validations/callbacks.rb:117:in `block in run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_validation_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.1/lib/active_model/validations/callbacks.rb:117:in `run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.1/lib/active_model/validations.rb:337:in `valid?'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/validations.rb:68:in `valid?'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/validations.rb:84:in `perform_validations'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/validations.rb:47:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:298:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:298:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/suppressor.rb:44:in `save'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/api/group_hooks.rb:69:in `block (2 levels) in <class:GroupHooks>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `block (2 levels) in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:58:in `block in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:341:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:267:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:247:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:322:in `block in build_stack'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/ip_address_state.rb:10:in `with'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api_guard.rb:213:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource.rb:20:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:39:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:231:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:225:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router/route.rb:58:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:116:in `process_route'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:72:in `block in identity'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:91:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:70:in `identity'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:55:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:132:in `with_optimization'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:54:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:167:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:71:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:66:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/mapper.rb:20:in `block in <class:Constraints>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-experiment-0.6.4/lib/gitlab/experiment/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/flipper-0.21.0/lib/flipper/middleware/memoizer.rb:77:in `memoized_call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/flipper-0.21.0/lib/flipper/middleware/memoizer.rb:42:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/speedscope.rb:13:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/web_transaction.rb:21:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:16:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/jira/middleware.rb:19:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/batch-loader-2.0.1/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:97:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/apollo_upload_server-2.1.0/lib/apollo_upload_server/middleware.rb:32:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:173:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:111:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:50:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:37:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:21:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:123:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/timeout.rb:19:in `timeout'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:122:in `call'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/middleware/rack.rb:19:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/context.rb:39:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/middleware/rack.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-raven-3.1.2/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/engine.rb:539:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/railtie.rb:207:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.1/lib/rails/railtie.rb:207:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
==> /var/log/gitlab/gitlab-rails/api_json.log <==
{"time":"2022-10-23T07:03:30.699Z","severity":"INFO","duration_s":12.68973,"db_duration_s":0.0052,"view_duration_s":12.68453,"status":500,"method":"POST","path":"/api/v4/groups/23482/hooks","params":[{"key":"url","value":"http://ngob.com/"}],"host":"192.168.1.123","remote_ip":"172.17.0.1, 127.0.0.1","ua":"PostmanRuntime/7.29.2","route":"/api/:version/groups/:id/hooks","user_id":1,"username":"root","exception.class":"ActiveRecord::InvalidForeignKey","exception.message":"PG::ForeignKeyViolation: ERROR: insert or update on table \"gitlab_subscriptions\" violates foreign key constraint \"fk_e2595d00a1\"\nDETAIL: Key (namespace_id)=(23482) is not present in table \"namespaces\".\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","ee/app/models/ee/namespace.rb:501:in `generate_subscription'","ee/app/models/ee/namespace.rb:494:in `find_or_create_subscription'","ee/app/models/ee/namespace.rb:185:in `block in actual_plan'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","ee/app/models/ee/namespace.rb:181:in `actual_plan'","app/models/namespace.rb:434:in `actual_limits'","app/models/concerns/limitable.rb:35:in `validate_scoped_plan_limit_not_exceeded'","app/models/concerns/limitable.rb:24:in `validate_plan_limit_not_exceeded'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","ee/lib/api/group_hooks.rb:69:in `block (2 levels) in <class:GroupHooks>'","ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:21:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"queue_duration_s":0.007669,"redis_calls":5,"redis_duration_s":0.000977,"redis_read_bytes":203,"redis_write_bytes":257,"redis_cache_calls":1,"redis_cache_duration_s":0.000173,"redis_cache_read_bytes":203,"redis_cache_write_bytes":55,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.000804,"redis_shared_state_write_bytes":202,"db_count":15,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":15,"db_primary_cached_count":0,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"cpu_s":0.033326,"mem_objects":17805,"mem_bytes":1415704,"mem_mallocs":4915,"mem_total_bytes":2127904,"pid":9840,"correlation_id":"01GG1TEAQGKC1H7AJ763F9CQ2Y","meta.user":"root","meta.root_namespace":"VJurK","meta.caller_id":"POST /api/:version/groups/:id/hooks","meta.remote_ip":"172.17.0.1","meta.feature_category":"integrations","meta.client_id":"user/1","content_length":"174"}
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"application/json","correlation_id":"01GG1TEAQGKC1H7AJ763F9CQ2Y","duration_ms":12700,"host":"192.168.1.123:30000","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":500,"system":"http","time":"2022-10-23T07:03:30Z","ttfb_ms":12700,"uri":"/api/v4/groups/23482/hooks?url=http://rpba.co.uk/tN8NTpE7-R/","user_agent":"PostmanRuntime/7.29.2","written_bytes":39}
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.17.0.1 - - [23/Oct/2022:07:03:30 +0000] "POST /api/v4/groups/23482/hooks?url=http://rpba.co.uk/tN8NTpE7-R/ HTTP/1.1" 500 39 "" "PostmanRuntime/7.29.2" -
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2022-10-23T07:03:34.571Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:34.572Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0005955999949947,"retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:34.572Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}
{"severity":"INFO","time":"2022-10-23T07:03:34.572Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0003980000037699938,"retry":0}Output of checks
Results of GitLab application Check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ...
yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
Administrator / SI0 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.33.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished