Unable to register or view Kubernetes agents
Summary
On a fresh installation of Gitlab Omnibus, I am unable to register agents or see the Kubernetes Agents screen.
Steps to reproduce
- Navigate to a project
- Navigate to "Infrastructure -> Kubernetes clusters" in the project sidebar
- Attempt to register an agent and receive a GRPC timeout error
- See an error on the page anytime you navigate to "Infrastructure -> Kubernetes clusters"
What is the current bug behavior?
Any attempt on any project to create a Kubernetes agent or list the Kubernetes agents after attempting to create one displays an error
What is the expected correct behavior?
I should be able to create, register, and manage Kubernetes agents
Relevant logs
Relevant logs
These are the logs from tailing while trying to create an agent named "i-am-a-test-agent-3"
Despite the fact that I cannot see agents and get an error while trying to create them, if I try to create the same agent a second time, it says the name already exists.
==> /var/log/gitlab/postgres-exporter/state <== ==> /var/log/gitlab/postgres-exporter/current <== 2023-03-27_16:30:37.77429 ts=2023-03-27T16:30:37.772Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-03-27_16:30:40.04563 ts=2023-03-27T16:30:40.044Z caller=postgres_exporter.go:645 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=13.8.0 2023-03-27_19:58:23.96118 ts=2023-03-27T19:58:23.960Z caller=main.go:135 level=info msg="Listening on address" address=localhost:9187 2023-03-27_19:58:23.96607 ts=2023-03-27T19:58:23.962Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-03-27_19:58:52.74643 ts=2023-03-27T19:58:52.745Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-03-27_19:58:52.76365 ts=2023-03-27T19:58:52.763Z caller=postgres_exporter.go:645 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=13.8.0 2023-03-27_21:00:53.57206 ts=2023-03-27T21:00:53.571Z caller=main.go:135 level=info msg="Listening on address" address=localhost:9187 2023-03-27_21:00:53.57587 ts=2023-03-27T21:00:53.574Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-03-27_21:01:22.74685 ts=2023-03-27T21:01:22.746Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-03-27_21:01:22.76075 ts=2023-03-27T21:01:22.760Z caller=postgres_exporter.go:645 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=13.8.0 ==> /var/log/gitlab/prometheus/state <== ==> /var/log/gitlab/prometheus/current <== 2023-03-28_15:00:03.47429 ts=2023-03-28T15:00:03.474Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1680004800661 maxt=1680012000000 ulid=01GWMBQBBJ7VAJG2KXGZYXE9SJ duration=2.463203321s 2023-03-28_15:00:03.55304 ts=2023-03-28T15:00:03.552Z caller=head.go:844 level=info component=tsdb msg="Head GC completed" duration=68.769859ms 2023-03-28_15:00:03.56119 ts=2023-03-28T15:00:03.560Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=73 to_segment=74 mint=1680012000000 2023-03-28_15:00:06.78796 ts=2023-03-28T15:00:06.787Z caller=head.go:1013 level=info component=tsdb msg="WAL checkpoint complete" first=73 last=74 duration=3.227401526s 2023-03-28_15:00:17.82347 ts=2023-03-28T15:00:17.823Z caller=compact.go:460 level=info component=tsdb msg="compact blocks" count=5 mint=1679940000000 maxt=1680004800000 ulid=01GWMBQH07C1M121S4HMZFQH3Z sources="[01GWK2H5HBZBGP6D7KPMEXCZPW 01GWKY02TJ24FQ9BXC0PV9XDFB 01GWKQ45KA5V61C7Z51JRHF3FF 01GWKXZWVF9260Q5CY2K1K4KEZ 01GWM4VM3CJM6WX9F6JPHT5BH6]" duration=11.032384908s 2023-03-28_15:00:17.96709 ts=2023-03-28T15:00:17.966Z caller=db.go:1294 level=info component=tsdb msg="Deleting obsolete block" block=01GWK2H5HBZBGP6D7KPMEXCZPW 2023-03-28_15:00:18.08788 ts=2023-03-28T15:00:18.087Z caller=db.go:1294 level=info component=tsdb msg="Deleting obsolete block" block=01GWKY02TJ24FQ9BXC0PV9XDFB 2023-03-28_15:00:18.21093 ts=2023-03-28T15:00:18.210Z caller=db.go:1294 level=info component=tsdb msg="Deleting obsolete block" block=01GWM4VM3CJM6WX9F6JPHT5BH6 2023-03-28_15:00:18.41326 ts=2023-03-28T15:00:18.413Z caller=db.go:1294 level=info component=tsdb msg="Deleting obsolete block" block=01GWKXZWVF9260Q5CY2K1K4KEZ 2023-03-28_15:00:18.53163 ts=2023-03-28T15:00:18.531Z caller=db.go:1294 level=info component=tsdb msg="Deleting obsolete block" block=01GWKQ45KA5V61C7Z51JRHF3FF ==> /var/log/gitlab/registry/current <== 2023-03-22_00:00:04.18288 time="2023-03-22T00:00:04.180Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=false correlation_id=01GW39V62KF6NXCFG2D6C68W11 go_version=go1.18.7 method=GET path=/v2/ root_repo= router=gorilla/mux version=v3.66.0-gitlab 2023-03-22_00:00:04.18297 time="2023-03-22T00:00:04.182Z" level=info msg="authorized request" auth_user_name= auth_user_type= correlation_id=01GW39V62KF6NXCFG2D6C68W11 go_version=go1.18.7 root_repo= version=v3.66.0-gitlab 2023-03-22_00:00:04.18301 {"content_type":"application/json","correlation_id":"01GW39V62KF6NXCFG2D6C68W11","duration_ms":2,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:41280","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2023-03-22T00:00:04.182Z","ttfb_ms":2,"uri":"/v2/","user_agent":"GitLab/15.9.3-ee","written_bytes":2} 2023-03-22_00:00:04.23630 time="2023-03-22T00:00:04.236Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=false correlation_id=01GW39V64BJH5MTWCM2B87E3MH go_version=go1.18.7 method=GET path=/gitlab/v1/ root_repo= router=gorilla/mux version=v3.66.0-gitlab 2023-03-22_00:00:04.23746 time="2023-03-22T00:00:04.237Z" level=info msg="authorized request" auth_user_name= auth_user_type= correlation_id=01GW39V64BJH5MTWCM2B87E3MH go_version=go1.18.7 version=v3.66.0-gitlab 2023-03-22_00:00:04.23779 {"content_type":"","correlation_id":"01GW39V64BJH5MTWCM2B87E3MH","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:41290","remote_ip":"127.0.0.1","status":404,"system":"http","time":"2023-03-22T00:00:04.237Z","ttfb_ms":1,"uri":"/gitlab/v1/","user_agent":"GitLab/15.9.3-ee","written_bytes":0} 2023-03-22_15:27:35.60159 time="2023-03-22T15:27:35.601Z" level=info msg="attempting to stop server gracefully..." database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated 2023-03-22_15:27:35.60247 time="2023-03-22T15:27:35.601Z" level=info msg="graceful shutdown successful" database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated ==> /var/log/gitlab/puma/state <== ==> /var/log/gitlab/puma/puma_stderr.log <== ==> /var/log/gitlab/puma/puma_stdout.log <== ==> /var/log/gitlab/puma/current <== 2023-03-27_21:00:59.30714 {"timestamp":"2023-03-27T21:00:59.306Z","pid":1500,"message":"* Environment: production"} 2023-03-27_21:00:59.30714 {"timestamp":"2023-03-27T21:00:59.306Z","pid":1500,"message":"* Master PID: 1500"} 2023-03-27_21:00:59.30715 {"timestamp":"2023-03-27T21:00:59.306Z","pid":1500,"message":"* Workers: 4"} 2023-03-27_21:00:59.30716 {"timestamp":"2023-03-27T21:00:59.306Z","pid":1500,"message":"* Restarts: (✔) hot (✖) phased"} 2023-03-27_21:00:59.30716 {"timestamp":"2023-03-27T21:00:59.306Z","pid":1500,"message":"* Preloading application"} 2023-03-27_21:03:22.37689 {"timestamp":"2023-03-27T21:03:22.376Z","pid":1500,"message":"* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"} 2023-03-27_21:03:22.37752 {"timestamp":"2023-03-27T21:03:22.377Z","pid":1500,"message":"* Listening on http://127.0.0.1:8080"} 2023-03-27_21:03:22.37779 {"timestamp":"2023-03-27T21:03:22.377Z","pid":1500,"message":"! WARNING: Detected 1 Thread(s) started in app boot:"} 2023-03-27_21:03:22.37810 {"timestamp":"2023-03-27T21:03:22.377Z","pid":1500,"message":"! #\u003cThread:0x00007efc05c0c670 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"} 2023-03-27_21:03:22.37850 {"timestamp":"2023-03-27T21:03:22.378Z","pid":1500,"message":"Use Ctrl-C to stop"} ==> /var/log/gitlab/sidekiq/state <== ==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2023-03-28T16:31:11.363Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["10"],"class":"ElasticIndexInitialBulkCronWorker","jid":"a6e188a1b711b3c29c4904e5","created_at":"2023-03-28T16:31:10.974Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:8a43c915cea5dbaf1b19e6c75ce378344601b69fae652c0ec91a49e707f813b4","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:10.977Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-a6e188a1b711b3c29c4904e5: done: 0.323452 sec","job_status":"done","scheduling_latency_s":0.062521,"redis_calls":6,"redis_duration_s":0.101244,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.069653,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.031591,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.072,"db_main_duration_s":0.072,"db_main_replica_duration_s":0.0,"cpu_s":0.019825,"mem_objects":1801,"mem_bytes":281072,"mem_mallocs":370,"mem_total_bytes":353112,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":10,"duration_s":0.323452,"completed_at":"2023-03-28T16:31:11.363Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.071905} {"severity":"INFO","time":"2023-03-28T16:31:11.346Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["12"],"class":"ElasticIndexInitialBulkCronWorker","jid":"82e35c817311ce1927a47514","created_at":"2023-03-28T16:31:11.072Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:cb835d06037e108881056ced28f6f0d9e854e555c1b19a303eedb47f929a6f38","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.085Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-82e35c817311ce1927a47514: done: 0.240881 sec","job_status":"done","scheduling_latency_s":0.020604,"redis_calls":6,"redis_duration_s":0.06439399999999999,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.02743,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.036964,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.082,"db_main_duration_s":0.082,"db_main_replica_duration_s":0.0,"cpu_s":0.019717,"mem_objects":1801,"mem_bytes":281520,"mem_mallocs":372,"mem_total_bytes":353560,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":12,"duration_s":0.240881,"completed_at":"2023-03-28T16:31:11.346Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.081934} {"severity":"INFO","time":"2023-03-28T16:31:11.352Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["11"],"class":"ElasticIndexInitialBulkCronWorker","jid":"514e6f4a46ae79c725e1dd1e","created_at":"2023-03-28T16:31:11.013Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:f3d9b29e941800fc4e9f8741bf1a3b9fd9aee3c149cc020d29c7f05bdb74efba","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.041Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-514e6f4a46ae79c725e1dd1e: done: 0.271295 sec","job_status":"done","scheduling_latency_s":0.039241,"redis_calls":6,"redis_duration_s":0.190376,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.03856,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.151816,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.033,"db_main_duration_s":0.033,"db_main_replica_duration_s":0.0,"cpu_s":0.021622,"mem_objects":1801,"mem_bytes":281072,"mem_mallocs":370,"mem_total_bytes":353112,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":11,"duration_s":0.271295,"completed_at":"2023-03-28T16:31:11.352Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.032601} {"severity":"INFO","time":"2023-03-28T16:31:11.267Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["14"],"class":"ElasticIndexInitialBulkCronWorker","jid":"25fb666bbe15e6b25f340a68","created_at":"2023-03-28T16:31:11.151Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:ac40d5fddda5259f14234221a79cdc010d7db78ef29d8812667f5e49ea6b2639","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.182Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-25fb666bbe15e6b25f340a68: start","job_status":"start","scheduling_latency_s":0.084147} {"severity":"INFO","time":"2023-03-28T16:31:11.392Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["6"],"class":"ElasticIndexInitialBulkCronWorker","jid":"7d3f10860cad2a6764846063","created_at":"2023-03-28T16:31:10.642Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:a4436ee3f346e9a3b9977be8610fcd1f4e35acefdee10bd484c866a7d118e3ea","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:10.678Z","job_size_bytes":3,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-7d3f10860cad2a6764846063: done: 0.601062 sec","job_status":"done","scheduling_latency_s":0.112263,"redis_calls":6,"redis_duration_s":0.095617,"redis_read_bytes":4,"redis_write_bytes":678,"redis_queues_calls":2,"redis_queues_duration_s":0.001312,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.094305,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":460,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.032,"db_main_duration_s":0.032,"db_main_replica_duration_s":0.0,"cpu_s":0.020332,"mem_objects":1802,"mem_bytes":281088,"mem_mallocs":370,"mem_total_bytes":353168,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":6,"duration_s":0.601062,"completed_at":"2023-03-28T16:31:11.392Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.032} {"severity":"INFO","time":"2023-03-28T16:31:11.443Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["15"],"class":"ElasticIndexInitialBulkCronWorker","jid":"a34ce313720681a89eaacdb8","created_at":"2023-03-28T16:31:11.227Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:0af5f4477b54dfac13d3139066f926fb8a307f83e6a3be92109a25481214c9c5","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.230Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-a34ce313720681a89eaacdb8: done: 0.13887 sec","job_status":"done","scheduling_latency_s":0.074561,"redis_calls":6,"redis_duration_s":0.026105999999999997,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.001223,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.024883,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.035,"db_main_duration_s":0.035,"db_main_replica_duration_s":0.0,"cpu_s":0.01983,"mem_objects":1800,"mem_bytes":260592,"mem_mallocs":369,"mem_total_bytes":332592,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":15,"duration_s":0.13887,"completed_at":"2023-03-28T16:31:11.443Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.03526} {"severity":"INFO","time":"2023-03-28T16:31:11.397Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["8"],"class":"ElasticIndexInitialBulkCronWorker","jid":"c24d7f8d7ad937ffa07a9804","created_at":"2023-03-28T16:31:10.842Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:37c6d711679df274bf2f140ac4bf7445ca404afc6b50537f8a0c34ff8f488678","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:10.869Z","job_size_bytes":3,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-c24d7f8d7ad937ffa07a9804: done: 0.459084 sec","job_status":"done","scheduling_latency_s":0.068968,"redis_calls":6,"redis_duration_s":0.196289,"redis_read_bytes":4,"redis_write_bytes":678,"redis_queues_calls":2,"redis_queues_duration_s":0.061416,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.134873,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":460,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.045,"db_main_duration_s":0.045,"db_main_replica_duration_s":0.0,"cpu_s":0.019109,"mem_objects":1802,"mem_bytes":281088,"mem_mallocs":370,"mem_total_bytes":353168,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":8,"duration_s":0.459084,"completed_at":"2023-03-28T16:31:11.397Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.044809} {"severity":"INFO","time":"2023-03-28T16:31:11.462Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["13"],"class":"ElasticIndexInitialBulkCronWorker","jid":"5fb9d64b3ec0e95640bc3e63","created_at":"2023-03-28T16:31:11.101Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:3d3d6d1021dcbd22c7490b10d518b7a873933b520ebd16624f8d91a818367cad","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.122Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-5fb9d64b3ec0e95640bc3e63: done: 0.282879 sec","job_status":"done","scheduling_latency_s":0.057488,"redis_calls":6,"redis_duration_s":0.152503,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.069484,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.083019,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.03,"db_main_duration_s":0.03,"db_main_replica_duration_s":0.0,"cpu_s":0.020492,"mem_objects":1801,"mem_bytes":281072,"mem_mallocs":370,"mem_total_bytes":353112,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":13,"duration_s":0.282879,"completed_at":"2023-03-28T16:31:11.462Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.029797} {"severity":"INFO","time":"2023-03-28T16:31:11.482Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["9"],"class":"ElasticIndexInitialBulkCronWorker","jid":"299ecd8b858ac7d7c302dae7","created_at":"2023-03-28T16:31:10.913Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:8baec2a811006d88bfd4d1c775f3a0eff8d414fc0e73124d197c0655e5c6040d","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:10.948Z","job_size_bytes":3,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-299ecd8b858ac7d7c302dae7: done: 0.47435 sec","job_status":"done","scheduling_latency_s":0.059123,"redis_calls":6,"redis_duration_s":0.152009,"redis_read_bytes":4,"redis_write_bytes":678,"redis_queues_calls":2,"redis_queues_duration_s":0.058998,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.093011,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":460,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.038,"db_main_duration_s":0.038,"db_main_replica_duration_s":0.0,"cpu_s":0.022572,"mem_objects":1802,"mem_bytes":281088,"mem_mallocs":370,"mem_total_bytes":353168,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":9,"duration_s":0.47435,"completed_at":"2023-03-28T16:31:11.482Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.083325} {"severity":"INFO","time":"2023-03-28T16:31:11.507Z","retry":0,"queue":"cronjob:elastic_index_initial_bulk_cron","version":0,"queue_namespace":"cronjob","args":["14"],"class":"ElasticIndexInitialBulkCronWorker","jid":"25fb666bbe15e6b25f340a68","created_at":"2023-03-28T16:31:11.151Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"f65c467529c2278276713c976f63d47e","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:cronjob:elastic_index_initial_bulk_cron:ac40d5fddda5259f14234221a79cdc010d7db78ef29d8812667f5e49ea6b2639","size_limiter":"validated","enqueued_at":"2023-03-28T16:31:11.182Z","job_size_bytes":4,"pid":1625,"message":"ElasticIndexInitialBulkCronWorker JID-25fb666bbe15e6b25f340a68: done: 0.240942 sec","job_status":"done","scheduling_latency_s":0.084147,"redis_calls":6,"redis_duration_s":0.033337000000000006,"redis_read_bytes":4,"redis_write_bytes":681,"redis_queues_calls":2,"redis_queues_duration_s":0.02415,"redis_queues_read_bytes":2,"redis_queues_write_bytes":218,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.009187,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":463,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.012,"db_main_duration_s":0.012,"db_main_replica_duration_s":0.0,"cpu_s":0.018582,"mem_objects":1801,"mem_bytes":281072,"mem_mallocs":370,"mem_total_bytes":353112,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.elastic_index_initial_bulk_cron_worker.records_count":0,"extra.elastic_index_initial_bulk_cron_worker.shard_number":14,"duration_s":0.240942,"completed_at":"2023-03-28T16:31:11.507Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.01147} ==> /var/log/gitlab/logrotate/state <== ==> /var/log/gitlab/logrotate/current <== ==> /var/log/gitlab/node-exporter/state <== ==> /var/log/gitlab/node-exporter/current <== 2023-03-27_21:00:53.73809 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=thermal_zone 2023-03-27_21:00:53.73810 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=time 2023-03-27_21:00:53.73810 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=timex 2023-03-27_21:00:53.73811 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=udp_queues 2023-03-27_21:00:53.73812 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=uname 2023-03-27_21:00:53.73815 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=vmstat 2023-03-27_21:00:53.73818 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=xfs 2023-03-27_21:00:53.73818 ts=2023-03-27T21:00:53.736Z caller=node_exporter.go:117 level=info collector=zfs 2023-03-27_21:00:53.74972 ts=2023-03-27T21:00:53.749Z caller=tls_config.go:232 level=info msg="Listening on" address=127.0.0.1:9100 2023-03-27_21:00:53.74978 ts=2023-03-27T21:00:53.749Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9100 ==> /var/log/gitlab/postgresql/state <== ==> /var/log/gitlab/postgresql/current <== 2023-03-27_21:00:55.86098 LOG: database system is ready to accept connections 2023-03-28_16:20:26.70025 LOG: no match in usermap "gitlab" for user "gitlab" authenticated as "root" 2023-03-28_16:20:26.70043 FATAL: Peer authentication failed for user "gitlab" 2023-03-28_16:20:26.70045 DETAIL: Connection matched pg_hba.conf line 70: "local all all peer map=gitlab" 2023-03-28_16:20:27.49449 LOG: no match in usermap "gitlab" for user "gitlab" authenticated as "root" 2023-03-28_16:20:27.49461 FATAL: Peer authentication failed for user "gitlab" 2023-03-28_16:20:27.49463 DETAIL: Connection matched pg_hba.conf line 70: "local all all peer map=gitlab" 2023-03-28_16:20:27.64181 LOG: no match in usermap "gitlab" for user "gitlab" authenticated as "root" 2023-03-28_16:20:27.64187 FATAL: Peer authentication failed for user "gitlab" 2023-03-28_16:20:27.64189 DETAIL: Connection matched pg_hba.conf line 70: "local all all peer map=gitlab" ==> /var/log/gitlab/redis/state <== ==> /var/log/gitlab/redis/current <== 2023-03-28_16:24:53.03558 1529:M 28 Mar 2023 16:24:53.035 * 10 changes in 300 seconds. Saving... 2023-03-28_16:24:53.03665 1529:M 28 Mar 2023 16:24:53.036 * Background saving started by pid 26734 2023-03-28_16:24:53.14436 26734:C 28 Mar 2023 16:24:53.144 * DB saved on disk 2023-03-28_16:24:53.14583 26734:C 28 Mar 2023 16:24:53.145 * RDB: 0 MB of memory used by copy-on-write 2023-03-28_16:24:53.23851 1529:M 28 Mar 2023 16:24:53.238 * Background saving terminated with success 2023-03-28_16:29:54.03427 1529:M 28 Mar 2023 16:29:54.034 * 10 changes in 300 seconds. Saving... 2023-03-28_16:29:54.03535 1529:M 28 Mar 2023 16:29:54.035 * Background saving started by pid 26940 2023-03-28_16:29:54.13344 26940:C 28 Mar 2023 16:29:54.133 * DB saved on disk 2023-03-28_16:29:54.13483 26940:C 28 Mar 2023 16:29:54.134 * RDB: 0 MB of memory used by copy-on-write 2023-03-28_16:29:54.13616 1529:M 28 Mar 2023 16:29:54.136 * Background saving terminated with success ==> /var/log/gitlab/gitlab-workhorse/state <== ==> /var/log/gitlab/gitlab-workhorse/current <== {"content_type":"","correlation_id":"01GWMGYMN6EXK5R7Y09Q28MV8R","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:22Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYNME4C0K1X3YBBMJZVDZ","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:23Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYPKPNYH2X5GVT0ZV3TPN","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:24Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYQJZF7CGZ09TYGYYMD5R","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:25Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYRJ7CEYTVYKSDXYCY0R1","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:26Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYSHHMD3KQ7CKNWAQ3PYQ","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:27Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYTGRZAXVDD34BD0JCVMG","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:28Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYVG4G67VDETFBG12V6RN","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:29Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYWFBP6042XN8HNESR1K4","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:30Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} {"content_type":"","correlation_id":"01GWMGYXEKGJ3Q9DSVEFW8664H","duration_ms":3,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:31Z","ttfb_ms":3,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} ==> /var/log/gitlab/redis-exporter/state <== ==> /var/log/gitlab/redis-exporter/current <== 2023-03-27_16:29:52.03641 time="2023-03-27T16:29:52Z" level=info msg="Redis Metrics Exporter 1.46.0 build date: sha1: Go: go1.18.7 GOOS: linux GOARCH: amd64" 2023-03-27_16:29:52.03894 time="2023-03-27T16:29:52Z" level=info msg="Providing metrics at localhost:9121/metrics" 2023-03-27_19:58:23.88543 time="2023-03-27T19:58:23Z" level=info msg="Redis Metrics Exporter 1.46.0 build date: sha1: Go: go1.18.7 GOOS: linux GOARCH: amd64" 2023-03-27_19:58:23.89598 time="2023-03-27T19:58:23Z" level=info msg="Providing metrics at localhost:9121/metrics" 2023-03-27_21:00:53.53773 time="2023-03-27T21:00:53Z" level=info msg="Redis Metrics Exporter 1.46.0 build date: sha1: Go: go1.18.7 GOOS: linux GOARCH: amd64" 2023-03-27_21:00:53.54333 time="2023-03-27T21:00:53Z" level=info msg="Providing metrics at localhost:9121/metrics" ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== ==> /var/log/gitlab/nginx/access.log <== ==> /var/log/gitlab/nginx/state <== ==> /var/log/gitlab/nginx/gitlab_access.log <== xx.xx.xx.xxx - - [28/Mar/2023:16:31:22 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:23 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:24 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:25 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:26 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:27 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:28 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:29 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:30 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - xx.xx.xx.xxx - - [28/Mar/2023:16:31:31 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - ==> /var/log/gitlab/nginx/gitlab_registry_access.log <== ==> /var/log/gitlab/nginx/gitlab_error.log <== ==> /var/log/gitlab/nginx/gitlab_registry_error.log <== ==> /var/log/gitlab/nginx/error.log <== ==> /var/log/gitlab/nginx/current <== ==> /var/log/gitlab/alertmanager/state <== ==> /var/log/gitlab/alertmanager/current <== 2023-03-27_21:00:54.20796 ts=2023-03-27T21:00:54.207Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=master, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-03-27_21:00:54.20942 ts=2023-03-27T21:00:54.207Z caller=main.go:241 level=info build_context="(go=go1.18.7, user=GitLab-Omnibus, date=)" 2023-03-27_21:00:54.21957 ts=2023-03-27T21:00:54.218Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=xx.xx.xx.xxy port=9094 2023-03-27_21:00:54.23387 ts=2023-03-27T21:00:54.233Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-03-27_21:00:54.40268 ts=2023-03-27T21:00:54.399Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2023-03-27_21:00:54.40700 ts=2023-03-27T21:00:54.406Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2023-03-27_21:00:54.43119 ts=2023-03-27T21:00:54.431Z caller=tls_config.go:232 level=info msg="Listening on" address=127.0.0.1:9093 2023-03-27_21:00:54.43137 ts=2023-03-27T21:00:54.431Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9093 2023-03-27_21:00:56.23489 ts=2023-03-27T21:00:56.234Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000847854s 2023-03-27_21:01:04.23711 ts=2023-03-27T21:01:04.236Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003135958s ==> /var/log/gitlab/gitlab-kas/state <== ==> /var/log/gitlab/gitlab-kas/current <== 2023-03-27_19:58:24.63393 {"level":"info","time":"2023-03-27T19:58:24.630Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"} 2023-03-27_19:58:24.63726 {"level":"info","time":"2023-03-27T19:58:24.634Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"} 2023-03-27_19:58:24.63728 {"level":"info","time":"2023-03-27T19:58:24.632Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"} 2023-03-27_19:58:24.63729 {"level":"info","time":"2023-03-27T19:58:24.632Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"} 2023-03-27_19:58:24.63730 {"level":"info","time":"2023-03-27T19:58:24.636Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true} 2023-03-27_21:00:54.28542 {"level":"info","time":"2023-03-27T21:00:54.284Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"} 2023-03-27_21:00:54.28768 {"level":"info","time":"2023-03-27T21:00:54.285Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"} 2023-03-27_21:00:54.28770 {"level":"info","time":"2023-03-27T21:00:54.285Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"} 2023-03-27_21:00:54.28772 {"level":"info","time":"2023-03-27T21:00:54.286Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"} 2023-03-27_21:00:54.28802 {"level":"info","time":"2023-03-27T21:00:54.287Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true} ==> /var/log/gitlab/gitaly/state <== ==> /var/log/gitlab/gitaly/gitaly_hooks.log <== ==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <== {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:43:54Z","grpc.time_ms":0.329,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1584,"correlation_id":"a787d7cfcc0d9f783762f96e64cd489b","time":"2023-03-22T15:43:54.139Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:43:54Z","grpc.time_ms":0.329,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1583,"correlation_id":"90fab96aaec3a7acd6beb4ca6e8eb3d7","time":"2023-03-22T15:43:54.144Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:09Z","grpc.time_ms":0.329,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1584,"correlation_id":"99aba45314de0ed215646e5d5e506130","time":"2023-03-22T15:44:09.142Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:09Z","grpc.time_ms":0.471,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1583,"correlation_id":"bf8d6ec7075273f79994047288d156e5","time":"2023-03-22T15:44:09.147Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:24Z","grpc.time_ms":0.411,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1584,"correlation_id":"1984c4e3fea7064f15ebb7f623534f31","time":"2023-03-22T15:44:24.145Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:24Z","grpc.time_ms":0.329,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1583,"correlation_id":"a620a32b841161f49b2ff8d6e1cc00db","time":"2023-03-22T15:44:24.150Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:39Z","grpc.time_ms":0.491,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1584,"correlation_id":"1c42b30148211bf46cd8f379500a5da6","time":"2023-03-22T15:44:39.149Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:39Z","grpc.time_ms":0.51,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1583,"correlation_id":"6af889160dc724bb4b5ca202ec069b43","time":"2023-03-22T15:44:39.153Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:54Z","grpc.time_ms":0.328,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1584,"correlation_id":"eb8362fe81fa27240e37081db6c9c3b1","time":"2023-03-22T15:44:54.153Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-03-22T15:44:54Z","grpc.time_ms":0.295,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1583,"correlation_id":"0327b51a45ff06e02081c5f710e609b3","time":"2023-03-22T15:44:54.155Z"} ==> /var/log/gitlab/gitaly/current <== {"level":"info","msg":"finished tempdir cleaner walk","storage":"default","time":"2023-03-28T14:01:06.579Z","time_ms":0} {"level":"info","msg":"finished tempdir cleaner walk","storage":"default","time":"2023-03-28T15:01:06.579Z","time_ms":0} {"correlation_id":"01GWMDB3B4FNB5PZJ53ME5ZKW0","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"RepositoryExists","grpc.request.deadline":"2023-03-28T15:28:28.010","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":136,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":2,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2023-03-28T15:28:18.321","grpc.time_ms":0.294,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished unary call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T15:28:18.322Z","user_id":"2","username":"myuser"} {"command.count":1,"command.cpu_time_ms":5,"command.inblock":392,"command.majflt":1,"command.maxrss":308164,"command.minflt":159,"command.oublock":0,"command.real_time_ms":123,"command.spawn_token_wait_ms":0,"command.system_time_ms":5,"command.user_time_ms":0,"correlation_id":"01GWMDB3B4FNB5PZJ53ME5ZKW0","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"HasLocalBranches","grpc.request.deadline":"2023-03-28T15:28:28.003","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":136,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":2,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2023-03-28T15:28:18.334","grpc.time_ms":703.455,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished unary call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T15:28:19.038Z","user_id":"2","username":"myuser"} {"command.count":1,"command.cpu_time_ms":4,"command.inblock":0,"command.majflt":0,"command.maxrss":308164,"command.minflt":160,"command.oublock":0,"command.real_time_ms":5,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":4,"correlation_id":"01GWMDB3B4FNB5PZJ53ME5ZKW0","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"unary","grpc.method":"FindDefaultBranchName","grpc.request.deadline":"2023-03-28T15:28:29.008","grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":136,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":17,"grpc.service":"gitaly.RefService","grpc.start_time":"2023-03-28T15:28:19.080","grpc.time_ms":17.78,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished unary call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T15:28:19.098Z","user_id":"2","username":"myuser"} {"catfile.duration_ms":4,"catfile.flush_count":2,"catfile.flush_ms":0,"catfile.read_info_count":1,"catfile.read_info_ms":4,"catfile.read_object_count":1,"catfile.read_object_ms":0,"catfile.request_info_count":1,"catfile.request_info_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":1,"command.cpu_time_ms":4,"command.inblock":0,"command.majflt":0,"command.maxrss":308164,"command.minflt":158,"command.oublock":0,"command.real_time_ms":4,"command.spawn_token_wait_ms":0,"command.system_time_ms":4,"command.user_time_ms":0,"correlation_id":"01GWMDB3B4FNB5PZJ53ME5ZKW0","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"GetTreeEntries","grpc.request.deadline":"2023-03-28T15:28:49.077","grpc.request.fullMethod":"/gitaly.CommitService/GetTreeEntries","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":155,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":970,"grpc.service":"gitaly.CommitService","grpc.start_time":"2023-03-28T15:28:19.177","grpc.time_ms":14.738,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished streaming call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T15:28:19.192Z","user_id":"2","username":"myuser"} {"catfile.duration_ms":5,"catfile.flush_count":2,"catfile.flush_ms":0,"catfile.read_info_count":1,"catfile.read_info_ms":4,"catfile.read_object_count":1,"catfile.read_object_ms":1,"catfile.request_info_count":1,"catfile.request_info_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":1,"command.cpu_time_ms":4,"command.inblock":0,"command.majflt":0,"command.maxrss":308164,"command.minflt":156,"command.oublock":0,"command.real_time_ms":5,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":4,"correlation_id":"01GWMDB8HHYTGCKRCB8HRX6SJT","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"GetTreeEntries","grpc.request.deadline":"2023-03-28T15:28:55.024","grpc.request.fullMethod":"/gitaly.CommitService/GetTreeEntries","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":168,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":136,"grpc.service":"gitaly.CommitService","grpc.start_time":"2023-03-28T15:28:25.325","grpc.time_ms":15.161,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished streaming call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T15:28:25.340Z","user_id":"2","username":"myuser"} {"level":"info","msg":"finished tempdir cleaner walk","storage":"default","time":"2023-03-28T16:01:06.580Z","time_ms":0} {"catfile.duration_ms":5,"catfile.flush_count":2,"catfile.flush_ms":0,"catfile.read_info_count":1,"catfile.read_info_ms":4,"catfile.read_object_count":1,"catfile.read_object_ms":1,"catfile.request_info_count":1,"catfile.request_info_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":1,"command.cpu_time_ms":4,"command.inblock":0,"command.majflt":0,"command.maxrss":308164,"command.minflt":152,"command.oublock":0,"command.real_time_ms":5,"command.spawn_token_wait_ms":0,"command.system_time_ms":4,"command.user_time_ms":0,"correlation_id":"01GWMFWA1WSQ5ZM1KXK3Y9P74Y","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"GetTreeEntries","grpc.request.deadline":"2023-03-28T16:13:11.019","grpc.request.fullMethod":"/gitaly.CommitService/GetTreeEntries","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":168,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":136,"grpc.service":"gitaly.CommitService","grpc.start_time":"2023-03-28T16:12:41.219","grpc.time_ms":15.583,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished streaming call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T16:12:41.235Z","user_id":"2","username":"myuser"} {"catfile.duration_ms":5,"catfile.flush_count":2,"catfile.flush_ms":0,"catfile.read_info_count":1,"catfile.read_info_ms":4,"catfile.read_object_count":1,"catfile.read_object_ms":1,"catfile.request_info_count":1,"catfile.request_info_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":1,"command.cpu_time_ms":3,"command.inblock":0,"command.majflt":0,"command.maxrss":308164,"command.minflt":155,"command.oublock":0,"command.real_time_ms":3,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":3,"correlation_id":"01GWMGB4BJ4KK7QRTNXG8RMP4C","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"server_stream","grpc.method":"GetTreeEntries","grpc.request.deadline":"2023-03-28T16:21:15.001","grpc.request.fullMethod":"/gitaly.CommitService/GetTreeEntries","grpc.request.glProjectPath":"infrastructure/k8s-monorepo","grpc.request.glRepository":"project-5","grpc.request.payload_bytes":168,"grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":136,"grpc.service":"gitaly.CommitService","grpc.start_time":"2023-03-28T16:20:45.902","grpc.time_ms":12.883,"level":"info","limit.concurrency_queue_ms":0,"msg":"finished streaming call with code OK","peer.address":"@","pid":1588,"remote_ip":"10.11.2.63","span.kind":"server","system":"grpc","time":"2023-03-28T16:20:45.915Z","user_id":"2","username":"myuser"} ==> /var/log/gitlab/gitlab-rails/exceptions_json.log <== ==> /var/log/gitlab/gitlab-rails/elasticsearch.log <== ==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-03-21-18-47-00.log <== ==> /var/log/gitlab/gitlab-rails/auth_json.log <== ==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-03-22-15-41-16.log <== ==> /var/log/gitlab/gitlab-rails/graphql_json.log <== {"severity":"INFO","time":"2023-03-28T15:31:33.779Z","correlation_id":"01GWMDH3FQ5C8PFHNA2WYX60B1","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/nS4lEGu0cp_ZoEJvQCDvAGfheNVnaOdVNG5OisHnXzs=","duration_s":0.11650621399166994,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"k3s-monorepo-agent\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.009876397991320118,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:12:38.626Z","correlation_id":"01GWMFWA1WSQ5ZM1KXK3Y9P74Y","trace_type":"execute_query","query_fingerprint":"agentConfigurations/NsK33nkL6xInoaphnn74WZnvH7Y70rAdFNe4lvbfYrk=/1/QKkrzvVW-Pxj9iW4I0BpCWXXbKja8svLMfEaNczycyA=","duration_s":0.009548381989588961,"operation_name":"agentConfigurations","operation_fingerprint":"agentConfigurations/NsK33nkL6xInoaphnn74WZnvH7Y70rAdFNe4lvbfYrk=","is_mutation":false,"variables":"{\"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}","query_string":"query agentConfigurations($projectPath: ID!) {\n project(fullPath: $projectPath) {\n id\n agentConfigurations {\n nodes {\n agentName\n __typename\n }\n __typename\n }\n clusterAgents {\n nodes {\n id\n name\n __typename\n }\n __typename\n }\n __typename\n }\n}\n","meta.caller_id":"graphql:agentConfigurations","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.016062858994700946,"query_analysis.depth":7,"query_analysis.complexity":62,"query_analysis.used_fields":["Project.id","ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","ClusterAgentConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","ClusterAgentConnection.pageInfo","ClusterAgentConnection.count","ClusterAgentConnection.__typename","Project.clusterAgents","TreeEntry.id","TreeEntry.name","TreeEntry.path","TreeEntry.webPath","TreeEntry.__typename","TreeEntryConnection.nodes","TreeEntryConnection.__typename","Tree.trees","Tree.__typename","Repository.tree","Repository.__typename","Project.repository","Project.__typename","Query.project"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:12:38.631Z","correlation_id":"01GWMFWA1WSQ5ZM1KXK3Y9P74Y","trace_type":"execute_query","query_fingerprint":"getAgents/KAaYeJVrCDjNpOKCG6kKacYVwAJ2IudnrZkZ9XmOoTk=/4/RvdwoDZ2QSrlOJ2MTlOw_Pfg0F_x-Zj5X3TNp_SlDXc=","duration_s":0.0016185370041057467,"operation_name":"getAgents","operation_fingerprint":"getAgents/KAaYeJVrCDjNpOKCG6kKacYVwAJ2IudnrZkZ9XmOoTk=","is_mutation":false,"variables":"{\"defaultBranchName\"=\u003e\"main\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\", \"first\"=\u003e25, \"last\"=\u003enil}","query_string":"query getAgents($defaultBranchName: String!, $projectPath: ID!, $first: Int, $last: Int, $afterAgent: String, $beforeAgent: String) {\n project(fullPath: $projectPath) {\n id\n clusterAgents(\n first: $first\n last: $last\n before: $beforeAgent\n after: $afterAgent\n ) {\n nodes {\n ...ClusterAgentFragment\n __typename\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n count\n __typename\n }\n repository {\n tree(path: \".gitlab/agents\", ref: $defaultBranchName) {\n trees {\n nodes {\n id\n name\n path\n webPath\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:getAgents","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.016062858994700946,"query_analysis.depth":7,"query_analysis.complexity":62,"query_analysis.used_fields":["Project.id","ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","ClusterAgentConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","ClusterAgentConnection.pageInfo","ClusterAgentConnection.count","ClusterAgentConnection.__typename","Project.clusterAgents","TreeEntry.id","TreeEntry.name","TreeEntry.path","TreeEntry.webPath","TreeEntry.__typename","TreeEntryConnection.nodes","TreeEntryConnection.__typename","Tree.trees","Tree.__typename","Repository.tree","Repository.__typename","Project.repository","Project.__typename","Query.project"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:20:43.694Z","correlation_id":"01GWMGB4BJ4KK7QRTNXG8RMP4C","trace_type":"execute_query","query_fingerprint":"agentConfigurations/NsK33nkL6xInoaphnn74WZnvH7Y70rAdFNe4lvbfYrk=/1/QKkrzvVW-Pxj9iW4I0BpCWXXbKja8svLMfEaNczycyA=","duration_s":0.0016127370035974309,"operation_name":"agentConfigurations","operation_fingerprint":"agentConfigurations/NsK33nkL6xInoaphnn74WZnvH7Y70rAdFNe4lvbfYrk=","is_mutation":false,"variables":"{\"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}","query_string":"query agentConfigurations($projectPath: ID!) {\n project(fullPath: $projectPath) {\n id\n agentConfigurations {\n nodes {\n agentName\n __typename\n }\n __typename\n }\n clusterAgents {\n nodes {\n id\n name\n __typename\n }\n __typename\n }\n __typename\n }\n}\n","meta.caller_id":"graphql:agentConfigurations","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.016576346999499947,"query_analysis.depth":7,"query_analysis.complexity":62,"query_analysis.used_fields":["Project.id","ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","ClusterAgentConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","ClusterAgentConnection.pageInfo","ClusterAgentConnection.count","ClusterAgentConnection.__typename","Project.clusterAgents","TreeEntry.id","TreeEntry.name","TreeEntry.path","TreeEntry.webPath","TreeEntry.__typename","TreeEntryConnection.nodes","TreeEntryConnection.__typename","Tree.trees","Tree.__typename","Repository.tree","Repository.__typename","Project.repository","Project.__typename","Query.project"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:20:43.697Z","correlation_id":"01GWMGB4BJ4KK7QRTNXG8RMP4C","trace_type":"execute_query","query_fingerprint":"getAgents/KAaYeJVrCDjNpOKCG6kKacYVwAJ2IudnrZkZ9XmOoTk=/4/RvdwoDZ2QSrlOJ2MTlOw_Pfg0F_x-Zj5X3TNp_SlDXc=","duration_s":0.0013559950020862743,"operation_name":"getAgents","operation_fingerprint":"getAgents/KAaYeJVrCDjNpOKCG6kKacYVwAJ2IudnrZkZ9XmOoTk=","is_mutation":false,"variables":"{\"defaultBranchName\"=\u003e\"main\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\", \"first\"=\u003e25, \"last\"=\u003enil}","query_string":"query getAgents($defaultBranchName: String!, $projectPath: ID!, $first: Int, $last: Int, $afterAgent: String, $beforeAgent: String) {\n project(fullPath: $projectPath) {\n id\n clusterAgents(\n first: $first\n last: $last\n before: $beforeAgent\n after: $afterAgent\n ) {\n nodes {\n ...ClusterAgentFragment\n __typename\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n count\n __typename\n }\n repository {\n tree(path: \".gitlab/agents\", ref: $defaultBranchName) {\n trees {\n nodes {\n id\n name\n path\n webPath\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:getAgents","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.016576346999499947,"query_analysis.depth":7,"query_analysis.complexity":62,"query_analysis.used_fields":["Project.id","ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","ClusterAgentConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","ClusterAgentConnection.pageInfo","ClusterAgentConnection.count","ClusterAgentConnection.__typename","Project.clusterAgents","TreeEntry.id","TreeEntry.name","TreeEntry.path","TreeEntry.webPath","TreeEntry.__typename","TreeEntryConnection.nodes","TreeEntryConnection.__typename","Tree.trees","Tree.__typename","Repository.tree","Repository.__typename","Project.repository","Project.__typename","Query.project"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:26:52.800Z","correlation_id":"01GWMGPCQKFX128336D49T41H5","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/uy2vq6AkbYHSt-sEmzFrZ3GQdGLuLNaUnyo9KdU-Utc=","duration_s":0.09887804900063202,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"I-am-a-test-agent-name\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.007821002000127919,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:27:19.613Z","correlation_id":"01GWMGQ4S5YWW5NNR1ARHHMCKQ","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/MBF1udFABlnJxI-UmIDdssC26t4YGO2AcLWqA87lXBQ=","duration_s":2.2567087759962305,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"i-am-a-test-agent\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.016775429001427256,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:28:58.767Z","correlation_id":"01GWMGT7SQTWFHT039BS8XXX42","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/MBF1udFABlnJxI-UmIDdssC26t4YGO2AcLWqA87lXBQ=","duration_s":0.07471416899352334,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"i-am-a-test-agent\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.008053875004407018,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:29:20.859Z","correlation_id":"01GWMGTXBK8S0TEXKJ9HS9BQFS","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/MBF1udFABlnJxI-UmIDdssC26t4YGO2AcLWqA87lXBQ=","duration_s":0.09165865699469578,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"i-am-a-test-agent\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.009142016991972923,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-03-28T16:29:45.268Z","correlation_id":"01GWMGVK856R6FEX2B02EED8TR","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/-75Bixa39ggSSiLIW7nKoy9pHl3EmGdBzHM5OsvypPY=","duration_s":2.1181385529926047,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"i-am-a-test-agent-2\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.005402105991379358,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} ==> /var/log/gitlab/gitlab-rails/importer.log <== ==> /var/log/gitlab/gitlab-rails/audit_json.log <== ==> /var/log/gitlab/gitlab-rails/database_load_balancing.log <== ==> /var/log/gitlab/gitlab-rails/git_json.log <== ==> /var/log/gitlab/gitlab-rails/sidekiq_client.log <== ==> /var/log/gitlab/gitlab-rails/service_measurement.log <== ==> /var/log/gitlab/gitlab-rails/application.log <== ==> /var/log/gitlab/gitlab-rails/grpc.log <== ==> /var/log/gitlab/gitlab-rails/application_json.log <== {"severity":"INFO","time":"2023-03-28T13:20:04.501Z","correlation_id":"e911c6c9e2e103edf9cca40c0474fb22","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-03-28T13:40:05.805Z","correlation_id":"6d8a381b59d970207fce4e6d2dc2cd68","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-03-28T14:00:09.243Z","correlation_id":"b18d38692636dd6d191f02a1ef5a5236","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-03-28T14:20:12.867Z","correlation_id":"db23b697a8ab5ae475eef336c07e5a2d","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-03-28T14:40:13.772Z","correlation_id":"e83c0bf124abd2f699f94657057b9a24","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-03-28T15:00:07.452Z","correlation_id":"96881a807dc96a9c63f32acc993d2410","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-03-28T15:20:14.195Z","correlation_id":"3556265910dbf979d530fcd3216d1d7b","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-03-28T15:40:09.134Z","correlation_id":"2cb4a4fd53260ec6375f365a4e44bf5a","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-03-28T16:00:03.510Z","correlation_id":"b99bcebd57975fda1b7d2591333f525b","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-03-28T16:20:07.946Z","correlation_id":"2ca26f16a92a984955f96845de12606b","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} ==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2023-03-28T12:36:09.320Z","severity":"INFO","duration_s":0.17613,"db_duration_s":0.08224,"view_duration_s":0.09389,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"082796e3f228a6ddadde9647ed6e72f8"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.023244,"redis_calls":15,"redis_duration_s":0.026901,"redis_read_bytes":228,"redis_write_bytes":1217,"redis_cache_calls":2,"redis_cache_duration_s":0.001655,"redis_cache_read_bytes":193,"redis_cache_write_bytes":262,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.025246,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":955,"db_count":6,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.083,"db_main_duration_s":0.083,"db_main_replica_duration_s":0.0,"cpu_s":0.102535,"mem_objects":10613,"mem_bytes":1031584,"mem_mallocs":3017,"mem_total_bytes":1456104,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"01GWM3FXRQ2KADBD8607FZQQR7","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/3","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T13:36:09.199Z","severity":"INFO","duration_s":0.24687,"db_duration_s":0.12416,"view_duration_s":0.12271,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"063bb5494ae2874fa32c21147fb509fb"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.116675,"redis_calls":28,"redis_duration_s":0.020086,"redis_read_bytes":358,"redis_write_bytes":3622,"redis_cache_calls":15,"redis_cache_duration_s":0.011007,"redis_cache_read_bytes":323,"redis_cache_write_bytes":2511,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.009079,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":13,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":13,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.147,"db_main_duration_s":0.147,"db_main_replica_duration_s":0.0,"cpu_s":0.210273,"mem_objects":20441,"mem_bytes":2498775,"mem_mallocs":6073,"mem_total_bytes":3316415,"pid":1725,"worker_id":"puma_2","rate_limiting_gates":[],"correlation_id":"01GWM6XS3R24C1Z0N6GFE9WX0N","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/4","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T13:36:10.143Z","severity":"INFO","duration_s":0.18474,"db_duration_s":0.04361,"view_duration_s":0.14113,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"898c577635657e408e5114f25ee3a5e1"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.124627,"redis_calls":22,"redis_duration_s":0.061214000000000005,"redis_read_bytes":1564,"redis_write_bytes":1887,"redis_cache_calls":9,"redis_cache_duration_s":0.040056,"redis_cache_read_bytes":1529,"redis_cache_write_bytes":776,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.021158,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":7,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.078,"db_main_duration_s":0.078,"db_main_replica_duration_s":0.0,"cpu_s":0.191564,"mem_objects":16922,"mem_bytes":2256327,"mem_mallocs":5589,"mem_total_bytes":2933207,"pid":1721,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01GWM6XT317ZYN9GWFKF46TNQX","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/5","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T13:36:11.608Z","severity":"INFO","duration_s":0.6559,"db_duration_s":0.56451,"view_duration_s":0.09139,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"52846c8f133c6fd50becfd15e32a42df"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.112546,"redis_calls":26,"redis_duration_s":0.02353,"redis_read_bytes":2376,"redis_write_bytes":2202,"redis_cache_calls":13,"redis_cache_duration_s":0.014006,"redis_cache_read_bytes":2341,"redis_cache_write_bytes":1091,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.009524,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":7,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.588,"db_main_duration_s":0.588,"db_main_replica_duration_s":0.0,"cpu_s":0.184394,"mem_objects":17680,"mem_bytes":2247127,"mem_mallocs":5962,"mem_total_bytes":2954327,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"correlation_id":"01GWM6XV298XZBQ0V5XD60DSXF","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/3","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T14:36:10.803Z","severity":"INFO","duration_s":0.19423,"db_duration_s":0.07544,"view_duration_s":0.11879,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"adcf0d85f417a17f4dba4d8e618d6807"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.090563,"redis_calls":28,"redis_duration_s":0.022012,"redis_read_bytes":358,"redis_write_bytes":3619,"redis_cache_calls":15,"redis_cache_duration_s":0.01145,"redis_cache_read_bytes":323,"redis_cache_write_bytes":2508,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.010562,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":13,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":13,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.098,"db_main_duration_s":0.098,"db_main_replica_duration_s":0.0,"cpu_s":0.195377,"mem_objects":19207,"mem_bytes":2539895,"mem_mallocs":5907,"mem_total_bytes":3308175,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"correlation_id":"01GWMABPCDBN29R2DJ2GP5EYEF","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/4","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T14:36:12.312Z","severity":"INFO","duration_s":0.70462,"db_duration_s":0.58838,"view_duration_s":0.11624,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"71751ef1e5550f21fbeb5e04cd06fe50"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.08904,"redis_calls":22,"redis_duration_s":0.041337,"redis_read_bytes":1561,"redis_write_bytes":1887,"redis_cache_calls":9,"redis_cache_duration_s":0.017606,"redis_cache_read_bytes":1526,"redis_cache_write_bytes":776,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.023731,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":7,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.611,"db_main_duration_s":0.611,"db_main_replica_duration_s":0.0,"cpu_s":0.16435,"mem_objects":16949,"mem_bytes":2257495,"mem_mallocs":5613,"mem_total_bytes":2935455,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"01GWMABQBNSVBT598RA0QF2EZH","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/5","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T14:36:12.740Z","severity":"INFO","duration_s":0.19614,"db_duration_s":0.12019,"view_duration_s":0.07595,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"d3ca7b998c55f26e2ce47e57f3968eaa"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.023786,"redis_calls":15,"redis_duration_s":0.009717,"redis_read_bytes":228,"redis_write_bytes":1217,"redis_cache_calls":2,"redis_cache_duration_s":0.001573,"redis_cache_read_bytes":193,"redis_cache_write_bytes":262,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.008144,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":955,"db_count":6,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.121,"db_main_duration_s":0.121,"db_main_replica_duration_s":0.0,"cpu_s":0.10484,"mem_objects":10607,"mem_bytes":908704,"mem_mallocs":3011,"mem_total_bytes":1332984,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"01GWMABRAYZYZ8Z9M4JPGJ7RZH","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/3","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T15:36:12.848Z","severity":"INFO","duration_s":0.54516,"db_duration_s":0.42043,"view_duration_s":0.12473,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"8052a232ef98eaf202072bee76768a64"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.073337,"redis_calls":28,"redis_duration_s":0.021764,"redis_read_bytes":358,"redis_write_bytes":3619,"redis_cache_calls":15,"redis_cache_duration_s":0.010858,"redis_cache_read_bytes":323,"redis_cache_write_bytes":2508,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.010906,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":13,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":13,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.432,"db_main_duration_s":0.432,"db_main_replica_duration_s":0.0,"cpu_s":0.191506,"mem_objects":19518,"mem_bytes":2294455,"mem_mallocs":5930,"mem_total_bytes":3075175,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"01GWMDSKNW8ZTJFTJ1DTD7J8V8","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/4","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T15:36:13.571Z","severity":"INFO","duration_s":0.26158,"db_duration_s":0.20178,"view_duration_s":0.0598,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"f32d002a71e5b2da676c202da8ad02c3"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.082347,"redis_calls":22,"redis_duration_s":0.011141000000000002,"redis_read_bytes":1561,"redis_write_bytes":1887,"redis_cache_calls":9,"redis_cache_duration_s":0.004695,"redis_cache_read_bytes":1526,"redis_cache_write_bytes":776,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.006446,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":1111,"db_count":7,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.224,"db_main_duration_s":0.224,"db_main_replica_duration_s":0.0,"cpu_s":0.123969,"mem_objects":16933,"mem_bytes":1929815,"mem_mallocs":5597,"mem_total_bytes":2607135,"pid":1721,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01GWMDSMN318FR5XSWPWDVJ8NR","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/5","content_length":"773","request_urgency":"low","target_duration_s":5} {"time":"2023-03-28T15:36:14.368Z","severity":"INFO","duration_s":0.11244,"db_duration_s":0.02844,"view_duration_s":0.084,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"15.10.0","revision":"456e3482","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null,"service_variables":"[FILTERED]","service_multiple_aliases":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"},{"key":"system_id","value":"s_51a726a890af"},{"key":"last_update","value":"6275db44a0e6e7b9c7c3231d08f2823f"}],"host":"gitlab.obfuscated.domain","remote_ip":"xx.xx.xx.xxx, 127.0.0.1","ua":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.025818,"redis_calls":15,"redis_duration_s":0.011567000000000001,"redis_read_bytes":228,"redis_write_bytes":1217,"redis_cache_calls":2,"redis_cache_duration_s":0.001423,"redis_cache_read_bytes":193,"redis_cache_write_bytes":262,"redis_shared_state_calls":13,"redis_shared_state_duration_s":0.010144,"redis_shared_state_read_bytes":35,"redis_shared_state_write_bytes":955,"db_count":6,"db_write_count":1,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":6,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.029,"db_main_duration_s":0.029,"db_main_replica_duration_s":0.0,"cpu_s":0.103736,"mem_objects":10605,"mem_bytes":867744,"mem_mallocs":3009,"mem_total_bytes":1291944,"pid":1721,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01GWMDSNMCTNZ4Z7QVKVYVSYQ7","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"xx.xx.xx.xxx","meta.feature_category":"continuous_integration","meta.client_id":"runner/3","content_length":"773","request_urgency":"low","target_duration_s":5} ==> /var/log/gitlab/gitlab-rails/backup_json.log <== ==> /var/log/gitlab/gitlab-rails/production.log <== ==> /var/log/gitlab/gitlab-rails/migrations.log <== ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/api/graphql","format":"*/*","controller":"GraphqlController","action":"execute","status":200,"time":"2023-03-28T16:29:20.866Z","params":[{"key":"_json","value":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]},{"key":"graphql","value":{"_json":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]}}],"correlation_id":"01GWMGTXBK8S0TEXKJ9HS9BQFS","meta.caller_id":"GraphqlController#execute","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","graphql":[{"depth":6,"complexity":31,"used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"used_deprecated_fields":[],"variables":"{\"input\"=>{\"name\"=>\"i-am-a-test-agent\", \"projectPath\"=>\"infrastructure/k8s-monorepo\"}}","operation_name":"createClusterAgent"}],"remote_ip":"10.11.2.63","user_id":2,"username":"myuser","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15","queue_duration_s":0.062803,"request_urgency":"low","target_duration_s":5,"redis_calls":15,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.010382,"redis_read_bytes":1726,"redis_write_bytes":2563,"redis_cache_calls":10,"redis_cache_duration_s":0.006437,"redis_cache_read_bytes":1543,"redis_cache_write_bytes":826,"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.002572,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1631,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001373,"redis_shared_state_write_bytes":106,"db_count":9,"db_write_count":2,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_main_count":9,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.044,"db_main_duration_s":0.044,"db_main_replica_duration_s":0.0,"cpu_s":0.195853,"mem_objects":28082,"mem_bytes":2734743,"mem_mallocs":8026,"mem_total_bytes":3858023,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"db_duration_s":0.02944,"view_duration_s":0.00065,"duration_s":0.15025} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:29:34.436Z","params":[],"redis_calls":4,"redis_duration_s":0.00212,"redis_read_bytes":810,"redis_write_bytes":335,"redis_cache_calls":4,"redis_cache_duration_s":0.00212,"redis_cache_read_bytes":810,"redis_cache_write_bytes":335,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.177847,"mem_objects":2900,"mem_bytes":1879328,"mem_mallocs":10856,"mem_total_bytes":1995328,"pid":1725,"worker_id":"puma_2","rate_limiting_gates":[],"correlation_id":"332e2f95-8ca1-44eb-97e2-65032d953fe5","db_duration_s":0.0,"view_duration_s":0.00108,"duration_s":0.1646} {"method":"POST","path":"/api/graphql","format":"*/*","controller":"GraphqlController","action":"execute","status":200,"time":"2023-03-28T16:29:45.276Z","params":[{"key":"_json","value":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]},{"key":"graphql","value":{"_json":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]}}],"correlation_id":"01GWMGVK856R6FEX2B02EED8TR","meta.caller_id":"GraphqlController#execute","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","graphql":[{"depth":6,"complexity":31,"used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"used_deprecated_fields":[],"variables":"{\"input\"=>{\"name\"=>\"i-am-a-test-agent-2\", \"projectPath\"=>\"infrastructure/k8s-monorepo\"}}","operation_name":"createClusterAgent"}],"remote_ip":"10.11.2.63","user_id":2,"username":"myuser","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15","queue_duration_s":0.056022,"request_urgency":"low","target_duration_s":5,"redis_calls":16,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.011825,"redis_read_bytes":1844,"redis_write_bytes":2708,"redis_cache_calls":11,"redis_cache_duration_s":0.006982,"redis_cache_read_bytes":1661,"redis_cache_write_bytes":971,"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.003965,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1631,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000878,"redis_shared_state_write_bytes":106,"db_count":12,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_main_count":12,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.061,"db_main_duration_s":0.061,"db_main_replica_duration_s":0.0,"cpu_s":0.176685,"mem_objects":25391,"mem_bytes":2892655,"mem_mallocs":7741,"mem_total_bytes":3908295,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"db_duration_s":0.05124,"view_duration_s":0.00104,"duration_s":2.1524} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:29:49.538Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.277952,"mem_objects":2161,"mem_bytes":1721920,"mem_mallocs":10473,"mem_total_bytes":1808360,"pid":1725,"worker_id":"puma_2","rate_limiting_gates":[],"correlation_id":"4f6912a2-025a-4740-bcbe-bbb14795dbf1","db_duration_s":0.0,"view_duration_s":0.00158,"duration_s":0.26784} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:30:04.494Z","params":[],"redis_calls":4,"redis_duration_s":0.003628,"redis_read_bytes":810,"redis_write_bytes":335,"redis_cache_calls":4,"redis_cache_duration_s":0.003628,"redis_cache_read_bytes":810,"redis_cache_write_bytes":335,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.2356,"mem_objects":2903,"mem_bytes":1940768,"mem_mallocs":10859,"mem_total_bytes":2056888,"pid":1721,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"b29cafe9-e18b-4252-886f-56c13335c08b","db_duration_s":0.0,"view_duration_s":0.00111,"duration_s":0.21511} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:30:19.492Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.23526,"mem_objects":2161,"mem_bytes":1721920,"mem_mallocs":10473,"mem_total_bytes":1808360,"pid":1721,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"757e54e7-44e8-4e10-9268-41e68f9e64a4","db_duration_s":0.0,"view_duration_s":0.00108,"duration_s":0.22123} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:30:34.523Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.263053,"mem_objects":2211,"mem_bytes":1721920,"mem_mallocs":10473,"mem_total_bytes":1810360,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"correlation_id":"52269b9b-f220-4f64-b2ec-c55b48054a36","db_duration_s":0.0,"view_duration_s":0.00234,"duration_s":0.25348} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:30:49.527Z","params":[],"redis_calls":4,"redis_duration_s":0.003783,"redis_read_bytes":810,"redis_write_bytes":335,"redis_cache_calls":4,"redis_cache_duration_s":0.003783,"redis_cache_read_bytes":810,"redis_cache_write_bytes":335,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.271063,"mem_objects":2903,"mem_bytes":1940768,"mem_mallocs":10859,"mem_total_bytes":2056888,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"correlation_id":"75f6d0db-8e10-4334-8efb-93f2f0846c49","db_duration_s":0.0,"view_duration_s":0.00158,"duration_s":0.2504} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:31:04.508Z","params":[],"redis_calls":4,"redis_duration_s":0.003743,"redis_read_bytes":810,"redis_write_bytes":335,"redis_cache_calls":4,"redis_cache_duration_s":0.003743,"redis_cache_read_bytes":810,"redis_cache_write_bytes":335,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.251466,"mem_objects":2904,"mem_bytes":1940816,"mem_mallocs":10860,"mem_total_bytes":2056976,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"3c80ac42-5bb7-4bab-8e17-8d64a01f5133","db_duration_s":0.0,"view_duration_s":0.00119,"duration_s":0.23199} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:31:19.529Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.274401,"mem_objects":2161,"mem_bytes":1721920,"mem_mallocs":10473,"mem_total_bytes":1808360,"pid":1723,"worker_id":"puma_1","rate_limiting_gates":[],"correlation_id":"d16db252-4d1f-49ef-b6e0-094da2133674","db_duration_s":0.0,"view_duration_s":0.00143,"duration_s":0.26311} ==> /var/log/gitlab/crond/current <== 2023-03-21_18:49:24.86654 time="2023-03-21T18:49:24Z" level=info msg="start runner with 0 jobs\n" 2023-03-21_19:18:12.10027 time="2023-03-21T19:18:11Z" level=info msg="got signal: terminated" 2023-03-21_19:18:12.10034 time="2023-03-21T19:18:11Z" level=info msg="stop runner" 2023-03-21_19:18:12.10035 time="2023-03-21T19:18:11Z" level=info msg=terminated 2023-03-21_19:19:09.43717 time="2023-03-21T19:19:09Z" level=info msg="starting go-crond version 22.9.1 (5bd4127; go1.18.7) " 2023-03-21_19:19:09.44190 time="2023-03-21T19:19:09Z" level=info msg="{\"ShowVersion\":false,\"ShowOnlyVersion\":false,\"ShowHelp\":false,\"Cron\":{\"DefaultUser\":\"root\",\"IncludeCronD\":[\"/var/opt/gitlab/crond\"],\"Auto\":false,\"RunParts\":null,\"RunParts1m\":null,\"RunParts15m\":null,\"RunPartsHourly\":null,\"RunPartsDaily\":null,\"RunPartsWeekly\":null,\"RunPartsMonthly\":null,\"AllowUnprivileged\":false,\"WorkDir\":\"/\",\"EnableUserSwitching\":false},\"Log\":{\"Verbose\":false,\"Json\":false},\"Server\":{\"Bind\":\"\",\"ReadTimeout\":5000000000,\"WriteTimeout\":10000000000,\"Metrics\":false},\"Args\":{\"Crontabs\":null}}" 2023-03-21_19:19:09.44215 time="2023-03-21T19:19:09Z" level=info msg="start runner with 0 jobs\n" 2023-03-22_15:27:26.56402 time="2023-03-22T15:27:26Z" level=info msg="got signal: terminated" 2023-03-22_15:27:26.58130 time="2023-03-22T15:27:26Z" level=info msg="stop runner" 2023-03-22_15:27:26.58133 time="2023-03-22T15:27:26Z" level=info msg=terminated ==> /var/log/gitlab/gitlab-exporter/state <== ==> /var/log/gitlab/gitlab-exporter/current <== 2023-03-28_16:31:14.59031 127.0.0.1 - - [28/Mar/2023:16:31:14 UTC] "GET /sidekiq HTTP/1.1" 200 124371 2023-03-28_16:31:14.59043 - -> /sidekiq 2023-03-28_16:31:15.18698 127.0.0.1 - - [28/Mar/2023:16:31:14 UTC] "GET /database HTTP/1.1" 200 3105 2023-03-28_16:31:15.18723 - -> /database 2023-03-28_16:31:27.90614 127.0.0.1 - - [28/Mar/2023:16:31:27 UTC] "GET /ruby HTTP/1.1" 200 1080 2023-03-28_16:31:27.90632 - -> /ruby 2023-03-28_16:31:29.61437 127.0.0.1 - - [28/Mar/2023:16:31:29 UTC] "GET /sidekiq HTTP/1.1" 200 124371 2023-03-28_16:31:29.61455 - -> /sidekiq 2023-03-28_16:31:30.15430 127.0.0.1 - - [28/Mar/2023:16:31:29 UTC] "GET /database HTTP/1.1" 200 3105 2023-03-28_16:31:30.15440 - -> /database ==> /var/log/gitlab/gitlab-workhorse/current <== {"content_type":"","correlation_id":"01GWMGYYDW3VWFSCY3Y0A3NZ26","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:32Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} ==> /var/log/gitlab/nginx/gitlab_access.log <== xx.xx.xx.xxx - - [28/Mar/2023:16:31:32 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - ==> /var/log/gitlab/gitlab-rails/graphql_json.log <== {"severity":"INFO","time":"2023-03-28T16:31:33.261Z","correlation_id":"01GWMGYWQZKBW6KTKZ3ATTW3YS","trace_type":"execute_query","query_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=/1/OlZofyHd_90AUTER8aeBjp2soAi2xRXJ-c2EmWJ8zjI=","duration_s":2.091781398994499,"operation_name":"createClusterAgent","operation_fingerprint":"createClusterAgent/HNLU1l1QuzbKV8bFfaypdezQJAHNT7OY_nODAs-Jtmw=","is_mutation":true,"variables":"{\"input\"=\u003e{\"name\"=\u003e\"i-am-a-test-agent-3\", \"projectPath\"=\u003e\"infrastructure/k8s-monorepo\"}}","query_string":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n","meta.caller_id":"graphql:createClusterAgent","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","query_analysis.duration_s":0.0049185540119651705,"query_analysis.depth":6,"query_analysis.complexity":31,"query_analysis.used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"query_analysis.used_deprecated_fields":[]} ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/api/graphql","format":"*/*","controller":"GraphqlController","action":"execute","status":200,"time":"2023-03-28T16:31:33.267Z","params":[{"key":"_json","value":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]},{"key":"graphql","value":{"_json":[{"operationName":"createClusterAgent","variables":"[FILTERED]","query":"mutation createClusterAgent($input: CreateClusterAgentInput!) {\n createClusterAgent(input: $input) {\n clusterAgent {\n ...ClusterAgentFragment\n __typename\n }\n errors\n __typename\n }\n}\n\nfragment ClusterAgentFragment on ClusterAgent {\n id\n name\n webPath\n connections {\n nodes {\n metadata {\n version\n __typename\n }\n __typename\n }\n __typename\n }\n tokens {\n nodes {\n id\n lastUsedAt\n __typename\n }\n __typename\n }\n __typename\n}\n"}]}}],"correlation_id":"01GWMGYWQZKBW6KTKZ3ATTW3YS","meta.caller_id":"GraphqlController#execute","meta.remote_ip":"10.11.2.63","meta.feature_category":"kubernetes_management","meta.user":"myuser","meta.user_id":2,"meta.client_id":"user/2","graphql":[{"depth":6,"complexity":31,"used_fields":["ClusterAgent.id","ClusterAgent.name","ClusterAgent.webPath","AgentMetadata.version","AgentMetadata.__typename","ConnectedAgent.metadata","ConnectedAgent.__typename","ConnectedAgentConnection.nodes","ConnectedAgentConnection.__typename","ClusterAgent.connections","ClusterAgentToken.id","ClusterAgentToken.lastUsedAt","ClusterAgentToken.__typename","ClusterAgentTokenConnection.nodes","ClusterAgentTokenConnection.__typename","ClusterAgent.tokens","ClusterAgent.__typename","CreateClusterAgentPayload.clusterAgent","CreateClusterAgentPayload.errors","CreateClusterAgentPayload.__typename","Mutation.createClusterAgent"],"used_deprecated_fields":[],"variables":"{\"input\"=>{\"name\"=>\"i-am-a-test-agent-3\", \"projectPath\"=>\"infrastructure/k8s-monorepo\"}}","operation_name":"createClusterAgent"}],"remote_ip":"10.11.2.63","user_id":2,"username":"myuser","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15","queue_duration_s":0.048493,"request_urgency":"low","target_duration_s":5,"redis_calls":11,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.009644,"redis_read_bytes":916,"redis_write_bytes":2229,"redis_cache_calls":6,"redis_cache_duration_s":0.002613,"redis_cache_read_bytes":733,"redis_cache_write_bytes":492,"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.00608,"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1631,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000951,"redis_shared_state_write_bytes":106,"db_count":12,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_main_count":12,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.026,"db_main_duration_s":0.026,"db_main_replica_duration_s":0.0,"cpu_s":0.175883,"mem_objects":23825,"mem_bytes":2519135,"mem_mallocs":7228,"mem_total_bytes":3472135,"pid":1727,"worker_id":"puma_3","rate_limiting_gates":[],"db_duration_s":0.01647,"view_duration_s":0.00104,"duration_s":2.12746} ==> /var/log/gitlab/gitlab-workhorse/current <== {"content_type":"application/json; charset=utf-8","correlation_id":"01GWMGYWQZKBW6KTKZ3ATTW3YS","duration_ms":2207,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.obfuscated.domain/infrastructure/k8s-monorepo/-/clusters","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/graphql\\z","status":200,"system":"http","time":"2023-03-28T16:31:33Z","ttfb_ms":2207,"uri":"/api/graphql","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15","written_bytes":506} ==> /var/log/gitlab/nginx/gitlab_access.log <== 10.11.2.63 - - [28/Mar/2023:16:31:33 +0000] "POST /api/graphql HTTP/2.0" 200 276 "https://gitlab.obfuscated.domain/infrastructure/k8s-monorepo/-/clusters" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" 1.83 ==> /var/log/gitlab/gitlab-workhorse/current <== {"content_type":"","correlation_id":"01GWMGYZD6E0XZHE9CW88F3B5M","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:33Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} ==> /var/log/gitlab/nginx/gitlab_access.log <== xx.xx.xx.xxx - - [28/Mar/2023:16:31:33 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" - ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-03-28T16:31:34.495Z","params":[],"redis_calls":4,"redis_duration_s":0.008367,"redis_read_bytes":810,"redis_write_bytes":335,"redis_cache_calls":4,"redis_cache_duration_s":0.008367,"redis_cache_read_bytes":810,"redis_cache_write_bytes":335,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.23289,"mem_objects":2903,"mem_bytes":1940792,"mem_mallocs":10859,"mem_total_bytes":2056912,"pid":1725,"worker_id":"puma_2","rate_limiting_gates":[],"correlation_id":"dde6a09c-05b7-4392-951e-154bfe641a36","db_duration_s":0.0,"view_duration_s":0.00126,"duration_s":0.2134} ==> /var/log/gitlab/gitlab-workhorse/current <== {"content_type":"","correlation_id":"01GWMGZ0CEXXD6XZH625VM4PKP","duration_ms":0,"host":"gitlab.obfuscated.domain","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-03-28T16:31:34Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)","written_bytes":0} ==> /var/log/gitlab/nginx/gitlab_access.log <== xx.xx.xx.xxx - - [28/Mar/2023:16:31:34 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.10.0 (15-10-stable; go1.19.6; linux/amd64)" -
Details of package version
Provide the package version installation details
||/ Name Version Architecture Description +++-======================-====================-============-============================================================ un gitlab-ce (no description available) un gitlab-ci-multi-runner (no description available) ii gitlab-ee 15.10.0-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) un gitlab-fips (no description available) rc gitlab-runner 11.2.0+dfsg-2ubuntu1 amd64 GitLab Runner - runs continuous integration (CI) jobs
Environment details
-
Operating System:
Ubuntu 20.04.6 LTS
-
Installation Target, remove incorrect values:
- VM: Other
Proxmox VE 6.4.X
- VM: Other
-
Installation Type, remove incorrect values:
- New Installation
-
Is there any other software running on the machine: No
-
Is this a single or multiple node installation? Single
-
Resources
- CPU: 4
- Memory total: 8GB
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab.obfuscated.domain' gitlab_rails['env'] = { "no_proxy" => "0,127.0.1.1,127.0.0.1,localhost,169.254.169.254" } gitlab_rails['gitlab_kas_enabled'] = true gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.obfuscated.domain/-/kubernetes-agent/' gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153' gitlab_kas['enable'] = true gitlab_kas['gitlab_address'] = 'https://gitlab.obfuscated.domain'