Unresponsive website and high CPU usage of Ruby - Migrated to 16.1.2
Summary
After a successful migration from 14.9.0-ee to 16.1.2-ee gitlab becomes unresponsive after a while (seems to take between 1 and 3 days) only returning a 502 or nothing at all (timeout).
The issue seems to be related with several sockets throwing gateway exceptions. These exceptions cause the ruby processes to spike at 100% cpu utilization.
There is no clear indication on why exactly that happens.
Restarting the container only helps temporarily, until it deadlocks itself again.
(this is my first issue here, so sorry if I'm missing some information)
Steps to reproduce
Don't really know what steps could be taken to reproduce my issue, but I'll gladly help to be able to reproduce the issue.
What is the current bug behavior?
Gitlab starts up just fine. After a while it becomes no longer accessible and the container uses all available CPU ressources
What is the expected correct behavior?
Not that.
Relevant logs
Relevant logs
==> /var/log/gitlab/gitlab-rails/production.log <== lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call' lib/gitlab/middleware/request_context.rb:15:in `call' lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call' config/initializers/fix_local_cache_middleware.rb:11:in `call' lib/gitlab/middleware/compressed_json.rb:44:in `call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call' lib/gitlab/middleware/release_env.rb:13:in `call'==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T21:58:47.015Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.047628,"mem_objects":2130,"mem_bytes":3319952,"mem_mallocs":11380,"mem_total_bytes":3405152,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"7568d3ee-4247-4e05-af8a-69a044633b7f","db_duration_s":0.0,"view_duration_s":0.00046,"duration_s":0.04406} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T21:59:02.020Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.052304,"mem_objects":2130,"mem_bytes":3319952,"mem_mallocs":11380,"mem_total_bytes":3405152,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"4595f4ad-44bc-4b7c-9954-aac7375607aa","db_duration_s":0.0,"view_duration_s":0.00047,"duration_s":0.04848} {"method":"GET","path":"/help","format":"/","controller":"HelpController","action":"index","status":200,"time":"2023-07-11T21:59:11.656Z","params":[],"correlation_id":"01H53FEBCYRTY6R6NP16YZCF0V","meta.caller_id":"HelpController#index","meta.remote_ip":"127.0.0.1","meta.feature_category":"not_owned","meta.client_id":"ip/127.0.0.1","remote_ip":"127.0.0.1","ua":"curl/8.0.1-DEV","queue_duration_s":0.022315,"response_bytes":71557,"request_urgency":"default","target_duration_s":1,"redis_calls":18,"redis_duration_s":0.0039759999999999995,"redis_read_bytes":2445,"redis_write_bytes":2815,"redis_cache_calls":8,"redis_cache_duration_s":0.001651,"redis_cache_read_bytes":498,"redis_cache_write_bytes":2180,"redis_feature_flag_calls":10,"redis_feature_flag_duration_s":0.002325,"redis_feature_flag_read_bytes":1947,"redis_feature_flag_write_bytes":635,"db_count":10,"db_write_count":0,"db_cached_count":5,"db_replica_count":0,"db_primary_count":10,"db_main_count":10,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":5,"db_main_cached_count":5,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"db_main_duration_s":0.007,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.113579,"mem_objects":105084,"mem_bytes":18238050,"mem_mallocs":78419,"mem_total_bytes":22441410,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"db_duration_s":0.00265,"view_duration_s":0.10528,"duration_s":0.11055} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T21:59:17.022Z","params":[],"redis_calls":1,"redis_duration_s":0.000235,"redis_read_bytes":203,"redis_write_bytes":52,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000235,"redis_feature_flag_read_bytes":203,"redis_feature_flag_write_bytes":52,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.054755,"mem_objects":2355,"mem_bytes":3370608,"mem_mallocs":11460,"mem_total_bytes":3464808,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"27c2775d-1e81-44c7-b77e-16d299254466","db_duration_s":0.0,"view_duration_s":0.00041,"duration_s":0.05113} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T21:59:32.016Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.049082,"mem_objects":2130,"mem_bytes":3319952,"mem_mallocs":11380,"mem_total_bytes":3405152,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"e7183ac6-3418-4a8a-8558-5b9d4f4618b9","db_duration_s":0.0,"view_duration_s":0.00046,"duration_s":0.04524} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T21:59:47.011Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.045468,"mem_objects":2130,"mem_bytes":3319928,"mem_mallocs":11380,"mem_total_bytes":3405128,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"9ca46d13-e66a-4836-9032-6720d994833b","db_duration_s":0.0,"view_duration_s":0.00047,"duration_s":0.04216} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T22:00:02.015Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.048046,"mem_objects":2130,"mem_bytes":3319928,"mem_mallocs":11380,"mem_total_bytes":3405128,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"412bc972-687b-4468-b4a6-5e562f68d8da","db_duration_s":0.0,"view_duration_s":0.00043,"duration_s":0.04408} {"method":"GET","path":"/help","format":"/","controller":"HelpController","action":"index","status":200,"time":"2023-07-11T22:00:12.086Z","params":[],"correlation_id":"01H53FG6D1JGG5SR64ZDKDFBXJ","meta.caller_id":"HelpController#index","meta.remote_ip":"127.0.0.1","meta.feature_category":"not_owned","meta.client_id":"ip/127.0.0.1","remote_ip":"127.0.0.1","ua":"curl/8.0.1-DEV","queue_duration_s":0.021132,"response_bytes":71557,"request_urgency":"default","target_duration_s":1,"redis_calls":19,"redis_duration_s":0.0042450000000000005,"redis_read_bytes":2329,"redis_write_bytes":2991,"redis_cache_calls":9,"redis_cache_duration_s":0.001778,"redis_cache_read_bytes":382,"redis_cache_write_bytes":2356,"redis_feature_flag_calls":10,"redis_feature_flag_duration_s":0.002467,"redis_feature_flag_read_bytes":1947,"redis_feature_flag_write_bytes":635,"db_count":11,"db_write_count":0,"db_cached_count":5,"db_replica_count":0,"db_primary_count":11,"db_main_count":11,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":5,"db_main_cached_count":5,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.116962,"mem_objects":105585,"mem_bytes":18304090,"mem_mallocs":78456,"mem_total_bytes":22527490,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"db_duration_s":0.00474,"view_duration_s":0.11407,"duration_s":0.12166} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T22:00:17.013Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.047021,"mem_objects":2131,"mem_bytes":3319928,"mem_mallocs":11380,"mem_total_bytes":3405168,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"8194d6b8-3c54-4853-bb6d-fa41084a4b98","db_duration_s":0.0,"view_duration_s":0.0004,"duration_s":0.04336} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-07-11T22:00:32.013Z","params":[],"redis_calls":1,"redis_duration_s":0.000208,"redis_read_bytes":203,"redis_write_bytes":52,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000208,"redis_feature_flag_read_bytes":203,"redis_feature_flag_write_bytes":52,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.046897,"mem_objects":2355,"mem_bytes":3370584,"mem_mallocs":11460,"mem_total_bytes":3464784,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"63a6b20b-aa8c-4452-8756-385417ce2a67","db_duration_s":0.0,"view_duration_s":0.00039,"duration_s":0.04353}
==> /var/log/gitlab/gitlab-rails/application_json.log <== {"severity":"INFO","time":"2023-07-11T19:00:05.807Z","meta.caller_id":"StuckCiJobsWorker","correlation_id":"61f3a94b177bd29c22193db03e8b7843","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-07-11T19:20:06.996Z","meta.caller_id":"Ci::StuckBuilds::DropRunningWorker","correlation_id":"d6fa38fbb13293eb5393e3b672f5088c","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-07-11T19:40:06.431Z","meta.caller_id":"Ci::StuckBuilds::DropScheduledWorker","correlation_id":"2fe95c5cbff248668ea8afce94f8ea1c","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-07-11T20:00:04.946Z","meta.caller_id":"StuckCiJobsWorker","correlation_id":"02dfe3adede5a2f50a10f5a9ad75ee7b","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-07-11T20:20:10.716Z","meta.caller_id":"Ci::StuckBuilds::DropRunningWorker","correlation_id":"7615fa87bfee4146175d8c4cd6a66be5","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-07-11T20:40:05.371Z","meta.caller_id":"Ci::StuckBuilds::DropScheduledWorker","correlation_id":"ad57a5f873255c8c2d05a34f9f605852","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-07-11T21:00:05.687Z","meta.caller_id":"StuckCiJobsWorker","correlation_id":"eca35e391a60d73f03a2ac3cc408674b","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"} {"severity":"INFO","time":"2023-07-11T21:20:10.056Z","meta.caller_id":"Ci::StuckBuilds::DropRunningWorker","correlation_id":"f45fcf518bf06f9a6c4be6fd072d77ba","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropRunningService: Cleaning running, timed-out builds"} {"severity":"INFO","time":"2023-07-11T21:40:08.180Z","meta.caller_id":"Ci::StuckBuilds::DropScheduledWorker","correlation_id":"77699ba0c7108bcd0049b85f645f3607","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"} {"severity":"INFO","time":"2023-07-11T22:00:03.754Z","meta.caller_id":"StuckCiJobsWorker","correlation_id":"09aa2dd9805a86065d61f2bc9be3cc50","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","message":"Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds"}
==> /var/log/gitlab/gitlab-rails/web_exporter.log <==
==> /var/log/gitlab/gitlab-rails/llm.log <==
Logfile created on 2023-06-29 14:53:20 +0000 by logger.rb/v1.4.3
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-06-29-11-08-45.log <==
main: == 20230418164957 QueueUpdateCodeSuggestionsForNamespaceSettings: migrating === main: == 20230418164957 QueueUpdateCodeSuggestionsForNamespaceSettings: migrated (0.0256s)
main: == 20230419164438 ChangeCodeSuggestionsDefaultFalseInNamespaceSettings: migrating main: -- change_column_default(:namespace_settings, :code_suggestions, {:from=>true, :to=>false}) main: -> 0.0071s main: == 20230419164438 ChangeCodeSuggestionsDefaultFalseInNamespaceSettings: migrated (0.0322s)
main: == [advisory_lock_connection] object_id: 227240, pg_backend_pid: 483
==> /var/log/gitlab/gitlab-rails/migrations.log <== {"severity":"INFO","time":"2023-06-29T12:28:58.164Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForVulnerabilityUserMentionsForGitlabCom","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.183Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForVulnerabilityUserMentionsForGitlabCom","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.215Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForCommitUserMentionsForGitlabCom","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.233Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForCommitUserMentionsForGitlabCom","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.265Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForMergeRequestUserMentionsForGitlabCom","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.284Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForMergeRequestUserMentionsForGitlabCom","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.315Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForTimelogsForGitlabCom","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.331Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForTimelogsForGitlabCom","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.355Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForAwardEmojiForGitlabCom","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100} {"severity":"INFO","time":"2023-06-29T12:28:58.372Z","correlation_id":"f637db546db2e35c750ff24beb7a7c35","method":"with_lock_retries","class":"CleanupBigintConversionForAwardEmojiForGitlabCom","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100}
==> /var/log/gitlab/gitlab-rails/grpc.log <==
Logfile created on 2022-03-22 22:06:30 +0000 by logger.rb/v1.4.2
W, [2022-07-16T08:33:41.029643 #11052] WARN -- GRPC: In each_remote_read_then_finish: call#run_batch failed somehow W, [2022-07-16T08:33:53.054659 #10993] WARN -- GRPC: In each_remote_read_then_finish: call#run_batch failed somehow
==> /var/log/gitlab/gitlab-rails/backup_json.log <== {"severity":"INFO","time":"2023-06-29T08:53:38.672Z","correlation_id":null,"message":"Dumping pages ... "} {"severity":"INFO","time":"2023-06-29T08:53:38.708Z","correlation_id":null,"message":"done"} {"severity":"INFO","time":"2023-06-29T08:53:38.708Z","correlation_id":null,"message":"Dumping lfs objects ... "} {"severity":"INFO","time":"2023-06-29T08:53:57.897Z","correlation_id":null,"message":"done"} {"severity":"INFO","time":"2023-06-29T08:53:57.897Z","correlation_id":null,"message":"Dumping terraform states ... "} {"severity":"INFO","time":"2023-06-29T08:53:57.906Z","correlation_id":null,"message":"done"} {"severity":"INFO","time":"2023-06-29T08:53:57.907Z","correlation_id":null,"message":"Dumping container registry images ... "} {"severity":"INFO","time":"2023-06-29T08:53:57.907Z","correlation_id":null,"message":"[DISABLED]"} {"severity":"INFO","time":"2023-06-29T08:53:57.907Z","correlation_id":null,"message":"Dumping packages ... "} {"severity":"INFO","time":"2023-06-29T08:53:57.916Z","correlation_id":null,"message":"done"}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-06-29-12-26-50.log <== main: == 20230615202125 PrepareIndexIssuesOnProjectHealthStatusAscWorkItemType: migrated (0.0790s)
main: == 20230615202511 PrepareIndexIssuesOnProjectHealthStatusDescWorkItemType: migrating main: -- index_exists?(:issues, [:project_id, :health_status, :id, :state_id, :work_item_type_id], {:order=>{:health_status=>"DESC NULLS LAST", :id=>:desc}, :name=>"index_issues_on_project_health_status_desc_work_item_type", :algorithm=>:concurrently}) main: -> 0.0197s main: -- add_index_options(:issues, [:project_id, :health_status, :id, :state_id, :work_item_type_id], {:order=>{:health_status=>"DESC NULLS LAST", :id=>:desc}, :name=>"index_issues_on_project_health_status_desc_work_item_type", :algorithm=>:concurrently}) main: -> 0.0000s main: == 20230615202511 PrepareIndexIssuesOnProjectHealthStatusDescWorkItemType: migrated (0.0517s)
main: == [advisory_lock_connection] object_id: 173640, pg_backend_pid: 498
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-07-05-21-57-39.log <== Running db:migrate rake task main: == [advisory_lock_connection] object_id: 173640, pg_backend_pid: 504 main: == [advisory_lock_connection] object_id: 173640, pg_backend_pid: 504
==> /var/log/gitlab/gitlab-rails/application.log <== 2023-06-29T10:29:48.507Z: ActiveRecord connection established 2023-06-29T10:35:08.516Z: ActiveRecord connection established 2023-06-29T10:39:08.527Z: ActiveRecord connection established 2023-06-29T10:40:20.848Z: Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds 2023-06-29T10:43:08.544Z: ActiveRecord connection established 2023-06-29T10:48:48.557Z: ActiveRecord connection established 2023-06-29T10:55:08.563Z: ActiveRecord connection established 2023-06-29T10:59:08.567Z: ActiveRecord connection established 2023-06-29T11:00:13.416Z: Ci::StuckBuilds::DropPendingService: Cleaning pending timed-out builds 2023-06-29T11:04:08.632Z: ActiveRecord connection established
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-06-29-12-18-27.log <== main: -> 0.0055s main: -- execute("SET statement_timeout TO 0") main: -> 0.0005s main: -- execute("ALTER TABLE organizations VALIDATE CONSTRAINT check_d130d769e0;") main: -> 0.0357s main: -- execute("RESET statement_timeout") main: -> 0.0013s main: == 20230515111314 AddTextLimitOnOrganizationName: migrated (0.1576s) ==========
main: == [advisory_lock_connection] object_id: 221740, pg_backend_pid: 504
==> /var/log/gitlab/gitlab-rails/service_measurement.log <==
Logfile created on 2022-03-22 22:06:30 +0000 by logger.rb/v1.4.2
==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2023-07-11T19:34:19.995Z","severity":"INFO","duration_s":0.01618,"db_duration_s":0.00405,"view_duration_s":0.01213,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.008621,"redis_calls":5,"redis_duration_s":0.001706,"redis_read_bytes":608,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000952,"redis_feature_flag_read_bytes":608,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000754,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.004,"db_main_duration_s":0.0,"db_ci_duration_s":0.004,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.02171,"mem_objects":6786,"mem_bytes":597616,"mem_mallocs":1804,"mem_total_bytes":869056,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53753HYCGP5W0RY3RK761AJ","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T19:35:20.018Z","severity":"INFO","duration_s":0.00757,"db_duration_s":0.00163,"view_duration_s":0.00594,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.005294,"redis_calls":5,"redis_duration_s":0.0006479999999999999,"redis_read_bytes":608,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000388,"redis_feature_flag_read_bytes":608,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.00026,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.0,"db_ci_duration_s":0.002,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.01233,"mem_objects":6966,"mem_bytes":577392,"mem_mallocs":1809,"mem_total_bytes":856032,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H5376Y63K69123155723RGXV","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T20:32:20.955Z","severity":"INFO","duration_s":0.10506,"db_duration_s":0.07265,"view_duration_s":0.03241,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.005004,"redis_calls":8,"redis_duration_s":0.0015440000000000002,"redis_read_bytes":6,"redis_write_bytes":1320,"redis_feature_flag_calls":6,"redis_feature_flag_duration_s":0.000989,"redis_feature_flag_read_bytes":6,"redis_feature_flag_write_bytes":1104,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000555,"redis_shared_state_write_bytes":216,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":3,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.056,"db_main_duration_s":0.001,"db_ci_duration_s":0.054,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.01887,"mem_objects":10097,"mem_bytes":848632,"mem_mallocs":2440,"mem_total_bytes":1252512,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53AFAVBMTJSJTY8TDWEHRJR","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T20:33:20.985Z","severity":"INFO","duration_s":0.00878,"db_duration_s":0.00272,"view_duration_s":0.00606,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.00535,"redis_calls":5,"redis_duration_s":0.000807,"redis_read_bytes":609,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000495,"redis_feature_flag_read_bytes":609,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000312,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"db_main_duration_s":0.0,"db_ci_duration_s":0.003,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.012401,"mem_objects":6848,"mem_bytes":598016,"mem_mallocs":1807,"mem_total_bytes":871936,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53AH5J8VG9KK21P0PH7WM1T","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T20:34:21.018Z","severity":"INFO","duration_s":0.00813,"db_duration_s":0.00223,"view_duration_s":0.0059,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.004895,"redis_calls":5,"redis_duration_s":0.000763,"redis_read_bytes":609,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000495,"redis_feature_flag_read_bytes":609,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000268,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.0,"db_ci_duration_s":0.002,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.011724,"mem_objects":6786,"mem_bytes":597616,"mem_mallocs":1804,"mem_total_bytes":869056,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53AK06BJRKMHYCBYY4KRB11","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T20:35:21.049Z","severity":"INFO","duration_s":0.00834,"db_duration_s":0.00266,"view_duration_s":0.00568,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.005146,"redis_calls":5,"redis_duration_s":0.000799,"redis_read_bytes":609,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000529,"redis_feature_flag_read_bytes":609,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.00027,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"db_main_duration_s":0.0,"db_ci_duration_s":0.003,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.011881,"mem_objects":6786,"mem_bytes":597616,"mem_mallocs":1804,"mem_total_bytes":869056,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53AMTT9WXSHC7M1RJP80N3Y","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T21:32:21.768Z","severity":"INFO","duration_s":0.06062,"db_duration_s":0.04943,"view_duration_s":0.01119,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.005181,"redis_calls":8,"redis_duration_s":0.0018410000000000002,"redis_read_bytes":6,"redis_write_bytes":1318,"redis_feature_flag_calls":6,"redis_feature_flag_duration_s":0.001198,"redis_feature_flag_read_bytes":6,"redis_feature_flag_write_bytes":1102,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000643,"redis_shared_state_write_bytes":216,"db_count":6,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":6,"db_main_count":3,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.05,"db_main_duration_s":0.002,"db_ci_duration_s":0.048,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.017842,"mem_objects":8245,"mem_bytes":752632,"mem_mallocs":2014,"mem_total_bytes":1082432,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53DX7A3H140TBXPB1S4KHX8","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T21:33:21.794Z","severity":"INFO","duration_s":0.01034,"db_duration_s":0.0031,"view_duration_s":0.00724,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.005159,"redis_calls":5,"redis_duration_s":0.000983,"redis_read_bytes":607,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000581,"redis_feature_flag_read_bytes":607,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000402,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"db_main_duration_s":0.0,"db_ci_duration_s":0.003,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.013492,"mem_objects":6786,"mem_bytes":597616,"mem_mallocs":1804,"mem_total_bytes":869056,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53DZ1ZF4JXXT7KPERE7G4HF","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T21:34:21.830Z","severity":"INFO","duration_s":0.016,"db_duration_s":0.00416,"view_duration_s":0.01184,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.009916,"redis_calls":5,"redis_duration_s":0.001475,"redis_read_bytes":607,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000937,"redis_feature_flag_read_bytes":607,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000538,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.004,"db_main_duration_s":0.0,"db_ci_duration_s":0.004,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.023594,"mem_objects":6813,"mem_bytes":598016,"mem_mallocs":1807,"mem_total_bytes":870536,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53E0WK89GGAV2RV4EN564P3","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27} {"time":"2023-07-11T21:35:21.865Z","severity":"INFO","duration_s":0.00911,"db_duration_s":0.0019,"view_duration_s":0.00721,"status":403,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"14.0.1","revision":"c1edb478","platform":"windows","architecture":"amd64","executor":"shell","shell":"pwsh","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null,"return_exit_code":null},"config":{"gpus":""}}},{"key":"token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"172.18.0.1, 127.0.0.1","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; windows/amd64)","route":"/api/:version/jobs/request","queue_duration_s":0.004832,"redis_calls":5,"redis_duration_s":0.000962,"redis_read_bytes":607,"redis_write_bytes":378,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000683,"redis_feature_flag_read_bytes":607,"redis_feature_flag_write_bytes":234,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000279,"redis_shared_state_write_bytes":144,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_main_count":0,"db_ci_count":3,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"db_main_duration_s":0.0,"db_ci_duration_s":0.002,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.012984,"mem_objects":6786,"mem_bytes":597616,"mem_mallocs":1804,"mem_total_bytes":869056,"pid":778,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01H53E2Q7SEEQNVVEV7EPVH97C","meta.caller_id":"POST /api/:version/jobs/request","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.client_id":"ip/172.18.0.1","content_length":"641","request_urgency":"low","target_duration_s":5,"response_bytes":27}
==> /var/log/gitlab/gitlab-rails/database_load_balancing.log <== {"severity":"DEBUG","time":"2023-07-11T18:30:07.569Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":false} {"severity":"DEBUG","time":"2023-07-11T18:30:07.570Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":true} {"severity":"DEBUG","time":"2023-07-11T18:30:26.697Z","correlation_id":null,"event":"setup","model":"ActiveRecord::Base","start_service_discovery":false} {"severity":"DEBUG","time":"2023-07-11T18:30:26.698Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":false} {"severity":"DEBUG","time":"2023-07-11T18:30:46.993Z","correlation_id":null,"event":"setup","model":"ActiveRecord::Base","start_service_discovery":false} {"severity":"DEBUG","time":"2023-07-11T18:30:46.994Z","correlation_id":null,"event":"setup","model":"ActiveRecord::Base","start_service_discovery":true} {"severity":"DEBUG","time":"2023-07-11T18:30:46.995Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":false} {"severity":"DEBUG","time":"2023-07-11T18:30:46.995Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":true} {"severity":"DEBUG","time":"2023-07-11T18:31:22.542Z","correlation_id":null,"event":"setup","model":"ActiveRecord::Base","start_service_discovery":true} {"severity":"DEBUG","time":"2023-07-11T18:31:22.543Z","correlation_id":null,"event":"setup","model":"Ci::ApplicationRecord","start_service_discovery":true}
==> /var/log/gitlab/gitlab-rails/git_json.log <==
Logfile created on 2022-03-23 01:30:25 +0000 by logger.rb/v1.4.2
==> /var/log/gitlab/gitlab-rails/graphql_json.log <== {"severity":"INFO","time":"2023-07-08T09:33:09.904Z","correlation_id":"01H4TDJ5D26KFXVM0RZ6Y0J0YE","meta.caller_id":"graphql:getAllRunnersEE","meta.remote_ip":"172.18.0.1","meta.feature_category":"runner","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getAllRunnersEE/X1oRoXqMpQMgAL-ZyeejBIV938rFqRAsOdhHCMSW19o=/3/DOMxa8P8jhNolIWUFSfJh5TEg2iBSmebcNcHWene8sA=","duration_s":0.45417041600012453,"operation_name":"getAllRunnersEE","operation_fingerprint":"getAllRunnersEE/X1oRoXqMpQMgAL-ZyeejBIV938rFqRAsOdhHCMSW19o=","is_mutation":false,"variables":"{"membership"=\u003e"DESCENDANTS", "sort"=\u003e"CREATED_DESC", "first"=\u003e20}","query_string":"query getAllRunnersEE($before: String, $after: String, $first: Int, $last: Int, $paused: Boolean, $status: CiRunnerStatus, $upgradeStatus: CiRunnerUpgradeStatus, $type: CiRunnerType, $tagList: [String!], $search: String, $sort: CiRunnerSort) {\n runners(\n before: $before\n after: $after\n first: $first\n last: $last\n paused: $paused\n status: $status\n upgradeStatus: $upgradeStatus\n type: $type\n tagList: $tagList\n search: $search\n sort: $sort\n ) {\n ...AllRunnersConnection\n __typename\n }\n}\n\nfragment AllRunnersConnection on CiRunnerConnection {\n nodes {\n ...ListItem\n adminUrl\n editAdminUrl\n __typename\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n __typename\n}\n\nfragment ListItem on CiRunner {\n ...ListItemShared\n upgradeStatus\n __typename\n}\n\nfragment ListItemShared on CiRunner {\n id\n description\n runnerType\n shortSha\n version\n ipAddress\n paused\n locked\n jobCount\n tagList\n createdAt\n createdBy {\n ...User\n __typename\n }\n contactedAt\n status\n jobExecutionStatus\n userPermissions {\n updateRunner\n deleteRunner\n __typename\n }\n managers {\n count\n __typename\n }\n groups(first: 1) {\n nodes {\n id\n name\n fullName\n webUrl\n __typename\n }\n __typename\n }\n ownerProject {\n id\n name\n nameWithNamespace\n webUrl\n __typename\n }\n __typename\n}\n\nfragment User on User {\n id\n avatarUrl\n name\n username\n webUrl\n __typename\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n","query_analysis.duration_s":0.004544535999912114,"query_analysis.depth":5,"query_analysis.complexity":67,"query_analysis.used_fields":["CiRunner.id","CiRunner.description","CiRunner.runnerType","CiRunner.shortSha","CiRunner.version","CiRunner.ipAddress","CiRunner.paused","CiRunner.locked","CiRunner.jobCount","CiRunner.tagList","CiRunner.createdAt","User.id","User.avatarUrl","User.name","User.username","User.webUrl","User.__typename","UserCore.__typename","CiRunner.createdBy","CiRunner.contactedAt","CiRunner.status","CiRunner.jobExecutionStatus","RunnerPermissions.updateRunner","RunnerPermissions.deleteRunner","RunnerPermissions.__typename","CiRunner.userPermissions","CiRunnerManagerConnection.count","CiRunnerManagerConnection.__typename","CiRunner.managers","Group.id","Group.name","Group.fullName","Group.webUrl","Group.__typename","GroupConnection.nodes","GroupConnection.__typename","CiRunner.groups","Project.id","Project.name","Project.nameWithNamespace","Project.webUrl","Project.__typename","CiRunner.ownerProject","CiRunner.__typename","CiRunner.upgradeStatus","CiRunner.adminUrl","CiRunner.editAdminUrl","CiRunnerConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","CiRunnerConnection.pageInfo","CiRunnerConnection.__typename","Query.runners"],"query_analysis.used_deprecated_fields":["CiRunner.jobExecutionStatus","CiRunner.managers","CiRunner.upgradeStatus"]} {"severity":"INFO","time":"2023-07-08T09:35:31.057Z","correlation_id":"01H4TDPFP81EQ8JSMGQV284T7Y","meta.caller_id":"graphql:getCurrentLicense","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getCurrentLicense/zERbRVaI6kxVt-bcqnIdm7g4BLuEWCoZN9DIjCd7wbc=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.004421761999992668,"operation_name":"getCurrentLicense","operation_fingerprint":"getCurrentLicense/zERbRVaI6kxVt-bcqnIdm7g4BLuEWCoZN9DIjCd7wbc=","is_mutation":false,"variables":"{}","query_string":"query getCurrentLicense {\n currentLicense {\n ...License\n __typename\n }\n}\n\nfragment License on CurrentLicense {\n id\n type\n plan\n name\n email\n company\n startsAt\n expiresAt\n activatedAt\n lastSync\n usersInLicenseCount\n billableUsersCount\n maximumUserCount\n usersOverLicenseCount\n __typename\n}\n","query_analysis.duration_s":0.0008586399999330752,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-08T09:35:31.064Z","correlation_id":"01H4TDPFP81EQ8JSMGQV284T7Y","meta.caller_id":"graphql:getPastLicenseHistory","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getPastLicenseHistory/WfGLeIhF3IP13NhWd0aNbFSSKIR80O7yIpjg63_oT6k=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.0057902979999653326,"operation_name":"getPastLicenseHistory","operation_fingerprint":"getPastLicenseHistory/WfGLeIhF3IP13NhWd0aNbFSSKIR80O7yIpjg63_oT6k=","is_mutation":false,"variables":"{}","query_string":"query getPastLicenseHistory {\n licenseHistoryEntries {\n nodes {\n id\n type\n plan\n name\n email\n company\n usersInLicenseCount\n startsAt\n expiresAt\n activatedAt\n __typename\n }\n __typename\n }\n}\n","query_analysis.duration_s":0.0008586399999330752,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-08T09:35:31.070Z","correlation_id":"01H4TDPFP81EQ8JSMGQV284T7Y","meta.caller_id":"graphql:getFutureLicenseHistory","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getFutureLicenseHistory/CyB7CLSNF9zWDNlGSnx9QWMoIt8mwsfHRtJ663Qxn0Y=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.0037692340001740376,"operation_name":"getFutureLicenseHistory","operation_fingerprint":"getFutureLicenseHistory/CyB7CLSNF9zWDNlGSnx9QWMoIt8mwsfHRtJ663Qxn0Y=","is_mutation":false,"variables":"{}","query_string":"query getFutureLicenseHistory {\n subscriptionFutureEntries {\n nodes {\n ...FutureSubscription\n __typename\n }\n __typename\n }\n}\n\nfragment FutureSubscription on SubscriptionFutureEntry {\n type\n plan\n name\n email\n company\n usersInLicenseCount\n startsAt\n expiresAt\n __typename\n}\n","query_analysis.duration_s":0.0008586399999330752,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-08T09:36:17.923Z","correlation_id":"01H4TDQXAJ9D99QN6S71VE1STS","meta.caller_id":"graphql:getVariables","meta.remote_ip":"172.18.0.1","meta.feature_category":"continuous_integration","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getVariables/Bb2TnhQsCvgjT6-l1QkT6lIBUdnDE85F-4rUWzV7c0A=/2/m__gFtU9e510h5DhgY3P6pVREq0ZMZgl9AE1WUeylV8=","duration_s":0.06796718400005375,"operation_name":"getVariables","operation_fingerprint":"getVariables/Bb2TnhQsCvgjT6-l1QkT6lIBUdnDE85F-4rUWzV7c0A=","is_mutation":false,"variables":"{"sort"=\u003e"KEY_ASC", "first"=\u003e100}","query_string":"query getVariables($after: String, $before: String, $first: Int, $last: Int, $sort: CiVariableSort = KEY_ASC) {\n ciVariables(\n after: $after\n before: $before\n first: $first\n last: $last\n sort: $sort\n ) {\n pageInfo {\n ...PageInfo\n __typename\n }\n nodes {\n ...BaseCiVariable\n ... on CiInstanceVariable {\n masked\n protected\n raw\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment BaseCiVariable on CiVariable {\n __typename\n id\n key\n value\n variableType\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n","query_analysis.duration_s":0.009850929000094766,"query_analysis.depth":3,"query_analysis.complexity":36,"query_analysis.used_fields":["PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","CiInstanceVariableConnection.pageInfo","CiVariable.__typename","CiVariable.id","CiVariable.key","CiVariable.value","CiVariable.variableType","CiInstanceVariable.masked","CiInstanceVariable.protected","CiInstanceVariable.raw","CiInstanceVariable.__typename","CiInstanceVariableConnection.nodes","CiInstanceVariableConnection.__typename","Query.ciVariables"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-09T01:15:15.117Z","correlation_id":"01H4W3F67J3YS65FD2PZ69RM3V","meta.caller_id":"graphql:getCurrentLicense","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getCurrentLicense/zERbRVaI6kxVt-bcqnIdm7g4BLuEWCoZN9DIjCd7wbc=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.003517533994454425,"operation_name":"getCurrentLicense","operation_fingerprint":"getCurrentLicense/zERbRVaI6kxVt-bcqnIdm7g4BLuEWCoZN9DIjCd7wbc=","is_mutation":false,"variables":"{}","query_string":"query getCurrentLicense {\n currentLicense {\n ...License\n __typename\n }\n}\n\nfragment License on CurrentLicense {\n id\n type\n plan\n name\n email\n company\n startsAt\n expiresAt\n activatedAt\n lastSync\n usersInLicenseCount\n billableUsersCount\n maximumUserCount\n usersOverLicenseCount\n __typename\n}\n","query_analysis.duration_s":0.0007262479994096793,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-09T01:15:15.126Z","correlation_id":"01H4W3F67J3YS65FD2PZ69RM3V","meta.caller_id":"graphql:getPastLicenseHistory","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getPastLicenseHistory/WfGLeIhF3IP13NhWd0aNbFSSKIR80O7yIpjg63_oT6k=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.0034458919981261715,"operation_name":"getPastLicenseHistory","operation_fingerprint":"getPastLicenseHistory/WfGLeIhF3IP13NhWd0aNbFSSKIR80O7yIpjg63_oT6k=","is_mutation":false,"variables":"{}","query_string":"query getPastLicenseHistory {\n licenseHistoryEntries {\n nodes {\n id\n type\n plan\n name\n email\n company\n usersInLicenseCount\n startsAt\n expiresAt\n activatedAt\n __typename\n }\n __typename\n }\n}\n","query_analysis.duration_s":0.0007262479994096793,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-09T01:15:15.129Z","correlation_id":"01H4W3F67J3YS65FD2PZ69RM3V","meta.caller_id":"graphql:getFutureLicenseHistory","meta.remote_ip":"172.18.0.1","meta.feature_category":"sm_provisioning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getFutureLicenseHistory/CyB7CLSNF9zWDNlGSnx9QWMoIt8mwsfHRtJ663Qxn0Y=/0/RBNvo1WzZ4oRRq0W9-hknpT7T8If536DEMBg9hyq_4o=","duration_s":0.001695855004072655,"operation_name":"getFutureLicenseHistory","operation_fingerprint":"getFutureLicenseHistory/CyB7CLSNF9zWDNlGSnx9QWMoIt8mwsfHRtJ663Qxn0Y=","is_mutation":false,"variables":"{}","query_string":"query getFutureLicenseHistory {\n subscriptionFutureEntries {\n nodes {\n ...FutureSubscription\n __typename\n }\n __typename\n }\n}\n\nfragment FutureSubscription on SubscriptionFutureEntry {\n type\n plan\n name\n email\n company\n usersInLicenseCount\n startsAt\n expiresAt\n __typename\n}\n","query_analysis.duration_s":0.0007262479994096793,"query_analysis.depth":3,"query_analysis.complexity":24,"query_analysis.used_fields":["SubscriptionFutureEntry.type","SubscriptionFutureEntry.plan","SubscriptionFutureEntry.name","SubscriptionFutureEntry.email","SubscriptionFutureEntry.company","SubscriptionFutureEntry.usersInLicenseCount","SubscriptionFutureEntry.startsAt","SubscriptionFutureEntry.expiresAt","SubscriptionFutureEntry.__typename","SubscriptionFutureEntryConnection.nodes","SubscriptionFutureEntryConnection.__typename","Query.subscriptionFutureEntries"],"query_analysis.used_deprecated_fields":[]} {"severity":"INFO","time":"2023-07-11T20:24:30.498Z","correlation_id":"01H53A0ZE28NHP9SMGRA98HRQN","meta.caller_id":"graphql:getDashboardIssuesCount","meta.remote_ip":"172.18.0.1","meta.feature_category":"team_planning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getDashboardIssuesCount/ivt4gG24cOlmVqjeJeCwuSZu_DvCHagxXz38gYmdLmc=/6/RYkuBDLdxCDiFQgQDUmjgaofSYyhlgVZBHcknfeLNRU=","duration_s":0.0393290160000106,"operation_name":"getDashboardIssuesCount","operation_fingerprint":"getDashboardIssuesCount/ivt4gG24cOlmVqjeJeCwuSZu_DvCHagxXz38gYmdLmc=","is_mutation":false,"variables":"{"hideUsers"=\u003efalse, "isSignedIn"=\u003etrue, "sort"=\u003e"CREATED_DESC", "state"=\u003e"opened", "firstPageSize"=\u003e20, "assigneeUsernames"=\u003e"WizzardMaker"}","query_string":"query getDashboardIssuesCount($search: String, $assigneeId: String, $assigneeUsernames: [String!], $authorUsername: String, $confidential: Boolean, $labelName: [String], $milestoneTitle: [String], $milestoneWildcardId: MilestoneWildcardId, $myReactionEmoji: String, $types: [IssueType!], $in: [IssuableSearchableField!], $not: NegatedIssueFilterInput, $or: UnionedIssueFilterInput) {\n openedIssues: issues(\n state: opened\n search: $search\n assigneeId: $assigneeId\n assigneeUsernames: $assigneeUsernames\n authorUsername: $authorUsername\n confidential: $confidential\n labelName: $labelName\n milestoneTitle: $milestoneTitle\n milestoneWildcardId: $milestoneWildcardId\n myReactionEmoji: $myReactionEmoji\n types: $types\n in: $in\n not: $not\n or: $or\n ) {\n count\n __typename\n }\n closedIssues: issues(\n state: closed\n search: $search\n assigneeId: $assigneeId\n assigneeUsernames: $assigneeUsernames\n authorUsername: $authorUsername\n confidential: $confidential\n labelName: $labelName\n milestoneTitle: $milestoneTitle\n milestoneWildcardId: $milestoneWildcardId\n myReactionEmoji: $myReactionEmoji\n types: $types\n in: $in\n not: $not\n or: $or\n ) {\n count\n __typename\n }\n allIssues: issues(\n state: all\n search: $search\n assigneeId: $assigneeId\n assigneeUsernames: $assigneeUsernames\n authorUsername: $authorUsername\n confidential: $confidential\n labelName: $labelName\n milestoneTitle: $milestoneTitle\n milestoneWildcardId: $milestoneWildcardId\n myReactionEmoji: $myReactionEmoji\n types: $types\n in: $in\n not: $not\n or: $or\n ) {\n count\n __typename\n }\n}\n","query_analysis.duration_s":0.003177502000653476,"query_analysis.depth":2,"query_analysis.complexity":24,"query_analysis.used_fields":["IssueConnection.count","IssueConnection.__typename","Query.issues"],"query_analysis.used_deprecated_fields":["Query.issues"]} {"severity":"INFO","time":"2023-07-11T20:24:31.065Z","correlation_id":"01H53A0ZGHJYB39HK8K5SQ0E5X","meta.caller_id":"graphql:getDashboardIssuesEE","meta.remote_ip":"172.18.0.1","meta.feature_category":"team_planning","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","trace_type":"execute_query","query_fingerprint":"getDashboardIssuesEE/IAzApZlUPbk8SeVuSkisGelNFSCgsDZEwCaW6jPxUHw=/6/RYkuBDLdxCDiFQgQDUmjgaofSYyhlgVZBHcknfeLNRU=","duration_s":0.5283423029995902,"operation_name":"getDashboardIssuesEE","operation_fingerprint":"getDashboardIssuesEE/IAzApZlUPbk8SeVuSkisGelNFSCgsDZEwCaW6jPxUHw=","is_mutation":false,"variables":"{"hideUsers"=\u003efalse, "isSignedIn"=\u003etrue, "sort"=\u003e"CREATED_DESC", "state"=\u003e"opened", "firstPageSize"=\u003e20, "assigneeUsernames"=\u003e"WizzardMaker"}","query_string":"query getDashboardIssuesEE($hideUsers: Boolean = false, $isSignedIn: Boolean = false, $search: String, $sort: IssueSort, $state: IssuableState, $assigneeId: String, $assigneeUsernames: [String!], $authorUsername: String, $confidential: Boolean, $labelName: [String], $milestoneTitle: [String], $milestoneWildcardId: MilestoneWildcardId, $myReactionEmoji: String, $types: [IssueType!], $in: [IssuableSearchableField!], $not: NegatedIssueFilterInput, $or: UnionedIssueFilterInput, $afterCursor: String, $beforeCursor: String, $firstPageSize: Int, $lastPageSize: Int) {\n issues(\n search: $search\n sort: $sort\n state: $state\n assigneeId: $assigneeId\n assigneeUsernames: $assigneeUsernames\n authorUsername: $authorUsername\n confidential: $confidential\n labelName: $labelName\n milestoneTitle: $milestoneTitle\n milestoneWildcardId: $milestoneWildcardId\n myReactionEmoji: $myReactionEmoji\n types: $types\n in: $in\n not: $not\n or: $or\n after: $afterCursor\n before: $beforeCursor\n first: $firstPageSize\n last: $lastPageSize\n ) {\n nodes {\n ...IssueFragment\n blockingCount\n healthStatus\n reference(full: true)\n weight\n }\n pageInfo {\n ...PageInfo\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n\nfragment IssueFragment on Issue {\n id\n iid\n confidential\n createdAt\n downvotes\n dueDate\n hidden\n humanTimeEstimate\n mergeRequestsCount\n moved\n state\n title\n updatedAt\n closedAt\n upvotes\n userDiscussionsCount @include(if: $isSignedIn)\n webPath\n webUrl\n type\n assignees @skip(if: $hideUsers) {\n nodes {\n id\n avatarUrl\n name\n username\n webUrl\n }\n __typename\n }\n author @skip(if: $hideUsers) {\n id\n avatarUrl\n name\n username\n webUrl\n }\n labels {\n nodes {\n id\n color\n title\n description\n }\n __typename\n }\n milestone {\n id\n dueDate\n startDate\n webPath\n title\n }\n taskCompletionStatus {\n completedCount\n count\n __typename\n }\n __typename\n}\n","query_analysis.duration_s":0.0044921520002390025,"query_analysis.depth":5,"query_analysis.complexity":86,"query_analysis.used_fields":["Issue.id","Issue.iid","Issue.confidential","Issue.createdAt","Issue.downvotes","Issue.dueDate","Issue.hidden","Issue.humanTimeEstimate","Issue.mergeRequestsCount","Issue.moved","Issue.state","Issue.title","Issue.updatedAt","Issue.closedAt","Issue.upvotes","Issue.userDiscussionsCount","Issue.webPath","Issue.webUrl","Issue.type","UserCore.id","UserCore.avatarUrl","UserCore.name","UserCore.username","UserCore.webUrl","UserCoreConnection.nodes","UserCoreConnection.__typename","Issue.assignees","Issue.author","Label.id","Label.color","Label.title","Label.description","LabelConnection.nodes","LabelConnection.__typename","Issue.labels","Milestone.id","Milestone.dueDate","Milestone.startDate","Milestone.webPath","Milestone.title","Issue.milestone","TaskCompletionStatus.completedCount","TaskCompletionStatus.count","TaskCompletionStatus.__typename","Issue.taskCompletionStatus","Issue.__typename","Issue.blockingCount","Issue.healthStatus","Issue.reference","Issue.weight","IssueConnection.nodes","PageInfo.hasNextPage","PageInfo.hasPreviousPage","PageInfo.startCursor","PageInfo.endCursor","PageInfo.__typename","IssueConnection.pageInfo","IssueConnection.__typename","Query.issues"],"query_analysis.used_deprecated_fields":["Query.issues"]}
==> /var/log/gitlab/gitlab-rails/auth_json.log <== {"severity":"INFO","time":"2023-07-08T09:22:33.094Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-08T09:24:04.084Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-08T09:25:11.960Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-08T09:25:17.388Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T08:06:06.658Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T08:06:40.549Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T08:06:52.036Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T18:30:08.161Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T18:30:30.453Z","gitlab_throttle_user_allowlist":[]} {"severity":"INFO","time":"2023-07-11T18:30:49.794Z","gitlab_throttle_user_allowlist":[]}
==> /var/log/gitlab/gitlab-rails/sidekiq_client.log <== {"severity":"WARN","time":"2023-06-28T10:06:07.762Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby calling
Sidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-06-28T10:06:07.921Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-06-28T10:06:07.963Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-06-28T10:06:08.237Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"INFO","time":"2023-06-29T11:10:44.455Z","message":"Cron Jobs - deleted job with name: free_user_cap_data_remediation"} {"severity":"WARN","time":"2023-06-29T12:10:08.243Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-06-29T12:22:58.534Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-06-29T13:15:06.115Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-07-05T22:01:19.517Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"} {"severity":"WARN","time":"2023-07-09T01:15:01.679Z","message":"Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby callingSidekiq.strict_args!
during Sidekiq initialization.\n"}==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-06-29-09-44-55.log <== -- add_column(:application_settings, :pipeline_limit_per_project_user_sha, :integer, {:default=>0, :null=>false}) -> 0.0029s == 20220511191503 AddPipelineCreateLimitPerProjectUserShaToApplicationSettings: migrated (0.0029s)
== 20220513043344 RescheduleExpireOAuthTokens: migrating ====================== == 20220513043344 RescheduleExpireOAuthTokens: migrated (0.4513s) =============
== 20220517144749 RemoveVulnerabilityApprovalRules: migrating ================= == 20220517144749 RemoveVulnerabilityApprovalRules: migrated (0.0944s) ========
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2023-06-29-09-25-10.log <== -- indexes(:packages_build_infos) -> 0.0016s -- execute("SET statement_timeout TO 0") -> 0.0002s -- remove_index(:packages_build_infos, {:algorithm=>:concurrently, :name=>"index_packages_build_infos_package_id_pipeline_id"}) -> 0.0457s -- execute("RESET statement_timeout") -> 0.0003s == 20220413075921 UpdateIndexOnPackagesBuildInfos: migrated (0.1727s) =========
==> /var/log/gitlab/gitlab-rails/exceptions_json.log <== {"severity":"ERROR","time":"2023-07-06T07:53:09.272Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined method
starts_at' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.276Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methodexpires_at' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.281Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methodplan' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.285Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methodtrial?' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.291Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methodsubscription_id' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.299Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methoddaily_billable_users_count' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:in
block in value'","lib/gitlab/utils/usage_data.rb:200:inblock in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:in
with_metadata'","lib/gitlab/utils/usage_data.rb:198:inalt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:in
value'","lib/gitlab/usage/metric.rb:21:inblock in with_value'","lib/gitlab/usage/metric.rb:37:in
with_availability'","lib/gitlab/usage/metric.rb:21:inwith_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
call'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incompute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:in
block in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inmap'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
build'","lib/gitlab/usage/service_ping_report.rb:27:inwith_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:in
block in for'","lib/gitlab/usage/service_ping_report.rb:12:infor'","lib/service_ping/build_payload.rb:12:in
raw_payload'","lib/service_ping/build_payload.rb:15:infiltered_usage_data'","lib/service_ping/build_payload.rb:6:in
execute'","app/workers/gitlab_service_ping_worker.rb:40:inusage_data'","app/workers/gitlab_service_ping_worker.rb:35:in
block in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:inin_lock'","app/workers/gitlab_service_ping_worker.rb:31:in
perform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:incall'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in
call'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:incall'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:in
wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:inblock in call'","lib/gitlab/application_context.rb:118:in
block in use'","lib/gitlab/application_context.rb:118:inuse'","lib/gitlab/application_context.rb:57:in
with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:incall'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in
call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
block in call'","lib/gitlab/database/query_analyzer.rb:37:inwithin'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in
call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:incall'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in
call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:incall'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in
call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:inblock in call'","lib/gitlab/with_request_store.rb:17:in
enabling_request_store'","lib/gitlab/with_request_store.rb:10:inwith_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:inblock in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:inrun'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in
instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:incall'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in
block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:inwithin_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in
call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:incall'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in
call'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.303Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined methodlicensee_name' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/licensee_metrics.rb:10:in
block in \u003cclass:LicenseeMetrics\u003e'","lib/gitlab/usage/metrics/instrumentations/generic_metric.rb:37:inblock in value'","lib/gitlab/utils/usage_data.rb:200:in
block in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:inwith_metadata'","lib/gitlab/utils/usage_data.rb:198:in
alt_usage_data'","lib/gitlab/usage/metrics/instrumentations/generic_metric.rb:36:invalue'","lib/gitlab/usage/metric.rb:21:in
block in with_value'","lib/gitlab/usage/metric.rb:37:inwith_availability'","lib/gitlab/usage/metric.rb:21:in
with_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incall'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
compute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:inblock in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
map'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inbuild'","lib/gitlab/usage/service_ping_report.rb:27:in
with_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:inblock in for'","lib/gitlab/usage/service_ping_report.rb:12:in
for'","lib/service_ping/build_payload.rb:12:inraw_payload'","lib/service_ping/build_payload.rb:15:in
filtered_usage_data'","lib/service_ping/build_payload.rb:6:inexecute'","app/workers/gitlab_service_ping_worker.rb:40:in
usage_data'","app/workers/gitlab_service_ping_worker.rb:35:inblock in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:in
in_lock'","app/workers/gitlab_service_ping_worker.rb:31:inperform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:in
call'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:incall'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in
call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:inwrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in
block in call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/application_context.rb:57:inwith_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in
call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:incall'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:inblock in call'","lib/gitlab/database/query_analyzer.rb:37:in
within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:incall'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in
call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:incall'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in
call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:incall'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in
block in call'","lib/gitlab/with_request_store.rb:17:inenabling_request_store'","lib/gitlab/with_request_store.rb:10:in
with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:in
block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:inblock in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in
run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:ininstrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:in
call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:inblock in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in
within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:incall'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in
call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:incall'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.308Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined method
sha256' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:inblock in value'","lib/gitlab/utils/usage_data.rb:200:in
block in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:inwith_metadata'","lib/gitlab/utils/usage_data.rb:198:in
alt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:invalue'","lib/gitlab/usage/metric.rb:21:in
block in with_value'","lib/gitlab/usage/metric.rb:37:inwith_availability'","lib/gitlab/usage/metric.rb:21:in
with_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incall'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
compute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:inblock in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
map'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inbuild'","lib/gitlab/usage/service_ping_report.rb:27:in
with_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:inblock in for'","lib/gitlab/usage/service_ping_report.rb:12:in
for'","lib/service_ping/build_payload.rb:12:inraw_payload'","lib/service_ping/build_payload.rb:15:in
filtered_usage_data'","lib/service_ping/build_payload.rb:6:inexecute'","app/workers/gitlab_service_ping_worker.rb:40:in
usage_data'","app/workers/gitlab_service_ping_worker.rb:35:inblock in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:in
in_lock'","app/workers/gitlab_service_ping_worker.rb:31:inperform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:in
call'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:incall'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in
call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:inwrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in
block in call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/application_context.rb:57:inwith_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in
call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:incall'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:inblock in call'","lib/gitlab/database/query_analyzer.rb:37:in
within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:incall'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in
call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:incall'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in
call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:incall'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in
block in call'","lib/gitlab/with_request_store.rb:17:inenabling_request_store'","lib/gitlab/with_request_store.rb:10:in
with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:in
block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:inblock in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in
run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:ininstrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:in
call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:inblock in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in
within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:incall'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in
call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:incall'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-06T07:53:09.312Z","meta.caller_id":"GitlabServicePingWorker","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","meta.client_id":"ip/","exception.class":"NoMethodError","exception.message":"undefined method
add_ons' for nil:NilClass","exception.backtrace":["ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:42:inblock in value'","lib/gitlab/utils/usage_data.rb:200:in
block in alt_usage_data'","lib/gitlab/utils/usage_data.rb:48:inwith_metadata'","lib/gitlab/utils/usage_data.rb:198:in
alt_usage_data'","ee/lib/gitlab/usage/metrics/instrumentations/license_metric.rb:40:invalue'","lib/gitlab/usage/metric.rb:21:in
block in with_value'","lib/gitlab/usage/metric.rb:37:inwith_availability'","lib/gitlab/usage/metric.rb:21:in
with_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:incall'","lib/gitlab/usage/service_ping/instrumented_payload.rb:36:in
compute_instrumental_value'","lib/gitlab/usage/service_ping/instrumented_payload.rb:19:inblock in build'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:in
map'","lib/gitlab/usage/service_ping/instrumented_payload.rb:18:inbuild'","lib/gitlab/usage/service_ping_report.rb:27:in
with_instrumentation_classes'","lib/gitlab/usage/service_ping_report.rb:13:inblock in for'","lib/gitlab/usage/service_ping_report.rb:12:in
for'","lib/service_ping/build_payload.rb:12:inraw_payload'","lib/service_ping/build_payload.rb:15:in
filtered_usage_data'","lib/service_ping/build_payload.rb:6:inexecute'","app/workers/gitlab_service_ping_worker.rb:40:in
usage_data'","app/workers/gitlab_service_ping_worker.rb:35:inblock in perform'","lib/gitlab/exclusive_lease_helpers.rb:38:in
in_lock'","app/workers/gitlab_service_ping_worker.rb:31:inperform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:in
call'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:incall'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in
call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/sidekiq_middleware/worker_context.rb:11:inwrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in
block in call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/application_context.rb:57:inwith_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in
call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:incall'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:inblock in call'","lib/gitlab/database/query_analyzer.rb:37:in
within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:incall'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in
call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:incall'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in
call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:incall'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in
block in call'","lib/gitlab/with_request_store.rb:17:inenabling_request_store'","lib/gitlab/with_request_store.rb:10:in
with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:in
block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:inblock in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in
run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:ininstrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:in
call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:inblock in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in
within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:incall'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in
call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:incall'"],"user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"service_ping","tags.correlation_id":"c04cd818b6092142aa8d341ec120d3d9","extra.sidekiq":{"retry":3,"queue":"default","version":0,"queue_namespace":"cronjob","dead":false,"args":[],"class":"GitlabServicePingWorker","jid":"b1dc7d026207a53732b2ed08","created_at":1688442424.3723645,"meta.caller_id":"Cronjob","correlation_id":"c04cd818b6092142aa8d341ec120d3d9","meta.root_caller_id":"Cronjob","meta.feature_category":"service_ping","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f08b08648429384a6800590e6650412b8bf3ae483020a60c2e1d227e13e464e8","size_limiter":"validated","enqueued_at":1688629949.6904507,"error_message":"Unsuccessful response code: 406","error_class":"ServicePing::SubmitService::SubmissionError","failed_at":1688442523.669093,"retry_count":2,"retried_at":1688529036.9532511}} {"severity":"ERROR","time":"2023-07-08T21:26:33.916Z","correlation_id":"2aa5d9c6010f3c590faa206fc7c5f9c2","meta.caller_id":"ExternalServiceReactiveCachingWorker","meta.remote_ip":"172.18.0.1","meta.feature_category":"groups_and_projects","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","meta.related_class":"VersionCheck","meta.root_caller_id":"Dashboard::GroupsController#index","exception.class":"Gitlab::HTTP::BlockedUrlError","exception.message":"URL is blocked: Host cannot be resolved or invalid","exception.backtrace":["lib/gitlab/http_connection_adapter.rb:58:in
rescue in validate_url_with_proxy!'","lib/gitlab/http_connection_adapter.rb:50:invalidate_url_with_proxy!'","lib/gitlab/http_connection_adapter.rb:27:in
connection'","lib/gitlab/http.rb:65:inperform_request'","lib/gitlab/http.rb:85:in
try_get'","lib/version_check.rb:71:incalculate_reactive_cache'","app/models/concerns/reactive_caching.rb:94:in
block (2 levels) in exclusively_update_reactive_cache!'","app/models/concerns/reactive_caching.rb:153:inenqueuing_update'","app/models/concerns/reactive_caching.rb:93:in
block in exclusively_update_reactive_cache!'","app/models/concerns/reactive_caching.rb:143:inlocking_reactive_cache'","app/models/concerns/reactive_caching.rb:89:in
exclusively_update_reactive_cache!'","app/workers/concerns/reactive_cacheable_worker.rb:35:inperform'","lib/gitlab/sidekiq_middleware/defer_jobs.rb:33:in
call'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:incall'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in
perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:inperform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in
call'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:inwrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in
block in call'","lib/gitlab/application_context.rb:118:inblock in use'","lib/gitlab/application_context.rb:118:in
use'","lib/gitlab/application_context.rb:57:inwith_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in
call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:incall'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in
call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:inblock in call'","lib/gitlab/database/query_analyzer.rb:37:in
within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:incall'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in
call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:incall'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in
call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:incall'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in
block in call'","lib/gitlab/with_request_store.rb:17:inenabling_request_store'","lib/gitlab/with_request_store.rb:10:in
with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:incall'","lib/gitlab/sidekiq_middleware/server_metrics.rb:83:in
block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:inblock in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in
run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:ininstrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:82:in
call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:inblock in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in
within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:incall'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in
call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"],"exception.cause_class":"Gitlab::UrlBlocker::BlockedUrlError","user.username":"WizzardMaker","tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"groups_and_projects","tags.correlation_id":"2aa5d9c6010f3c590faa206fc7c5f9c2","extra.sidekiq":{"retry":3,"queue":"default","version":0,"class":"ExternalServiceReactiveCachingWorker","args":["VersionCheck","[FILTERED]"],"jid":"6db771bce31a00d4c8e3683f","created_at":1688808373.193478,"correlation_id":"2aa5d9c6010f3c590faa206fc7c5f9c2","meta.caller_id":"ExternalServiceReactiveCachingWorker","meta.remote_ip":"172.18.0.1","meta.feature_category":"groups_and_projects","meta.user":"WizzardMaker","meta.user_id":5,"meta.client_id":"user/5","meta.related_class":"VersionCheck","meta.root_caller_id":"Dashboard::GroupsController#index","worker_data_consistency":"always","size_limiter":"validated","scheduled_at":1688851573.1934164,"idempotency_key":"resque:gitlab:duplicate:default:28ead719eb8caefc84a596757ef3850b6aa8799ae42215a3ad988ecafc7925c7","enqueued_at":1688851577.8812397}}==> /var/log/gitlab/grafana/state <==
==> /var/log/gitlab/grafana/current <== 2023-06-29_11:29:02.97286 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972856Z","queryType":"arrow"} 2023-06-29_11:29:02.97286 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972860Z","queryType":"annotations"} 2023-06-29_11:29:02.97287 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972864Z","queryType":"table_static"} 2023-06-29_11:29:02.97287 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972867Z","queryType":"random_walk_with_error"} 2023-06-29_11:29:02.97287 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972871Z","queryType":"server_error_500"} 2023-06-29_11:29:02.97287 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972874Z","queryType":"logs"} 2023-06-29_11:29:02.97288 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2023-06-29T11:29:02.972877Z","queryType":"node_graph"} 2023-06-29_11:29:02.97288 {"@level":"debug","@message":"datasource: registering query type fallback handler","@timestamp":"2023-06-29T11:29:02.972881Z"} 2023-06-29_11:29:03.34040 t=2023-06-29T11:29:03+0000 lvl=eror msg="Failed to read plugin provisioning files from directory" logger=provisioning.plugins path=/var/opt/gitlab/grafana/provisioning/plugins error="open /var/opt/gitlab/grafana/provisioning/plugins: no such file or directory" 2023-06-29_11:29:03.34978 t=2023-06-29T11:29:03+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3000 protocol=http subUrl=/-/grafana socket=
==> /var/log/gitlab/redis-exporter/current <== 2023-07-11_18:28:45.94744 time="2023-07-11T18:28:45Z" level=info msg="Redis Metrics Exporter 1.46.0 build date: sha1: Go: go1.19.8 GOOS: linux GOARCH: amd64" 2023-07-11_18:28:45.96961 time="2023-07-11T18:28:45Z" level=info msg="Providing metrics at localhost:9121/metrics"
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== {"content_length_bytes":89,"correlation_id":"01H4N34S5J8AGKHXN3QVBFQ7N4","duration_ms":62,"level":"info","method":"GET","msg":"Finished HTTP request","status":200,"time":"2023-07-06T07:54:53Z","url":"http://unix/api/v4/internal/check"} {"content_length_bytes":89,"correlation_id":"01H4N3BF8TJNK3HHV407MM5J14","duration_ms":51,"level":"info","method":"GET","msg":"Finished HTTP request","status":200,"time":"2023-07-06T07:58:32Z","url":"http://unix/api/v4/internal/check"}
==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2023-07-11T22:01:01.670Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","jid":"941f28f219f16d7ea8a4d6c8","created_at":"2023-07-11T22:01:01.558Z","meta.caller_id":"Cronjob","correlation_id":"fe76912e93ef7a88f3dfa17846e78244","meta.root_caller_id":"Cronjob","meta.feature_category":"code_review_workflow","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.559Z","job_size_bytes":2,"pid":724,"message":"ScheduleMergeRequestCleanupRefsWorker JID-941f28f219f16d7ea8a4d6c8: done: 0.096376 sec","job_status":"done","scheduling_latency_s":0.014952,"redis_calls":2,"redis_duration_s":0.053012,"redis_read_bytes":2,"redis_write_bytes":186,"redis_queues_calls":2,"redis_queues_duration_s":0.053012,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.00512,"mem_objects":2949,"mem_bytes":385160,"mem_mallocs":1073,"mem_total_bytes":503120,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.096376,"completed_at":"2023-07-11T22:01:01.670Z","load_balancing_strategy":"primary","db_duration_s":0.0} {"severity":"INFO","time":"2023-07-11T22:01:01.680Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Metrics::GlobalMetricsUpdateWorker","jid":"83684f006ebb7e40fece2029","created_at":"2023-07-11T22:01:01.574Z","meta.caller_id":"Cronjob","correlation_id":"1392eefb7e2fab6af166e9db743a2f14","meta.root_caller_id":"Cronjob","meta.feature_category":"metrics","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:0ef60401f7cd15675ace9e5ac4456efd82434f42e048584516e1e7a85b24e5ba","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.575Z","job_size_bytes":2,"pid":724,"message":"Metrics::GlobalMetricsUpdateWorker JID-83684f006ebb7e40fece2029: done: 0.101888 sec","job_status":"done","scheduling_latency_s":0.002502,"redis_calls":4,"redis_duration_s":0.055722,"redis_read_bytes":4,"redis_write_bytes":494,"redis_queues_calls":2,"redis_queues_duration_s":0.000264,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.055458,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":308,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.004702,"mem_objects":3042,"mem_bytes":468024,"mem_mallocs":1086,"mem_total_bytes":589704,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.101888,"completed_at":"2023-07-11T22:01:01.680Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.0} {"severity":"INFO","time":"2023-07-11T22:01:01.772Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigrationWorker","jid":"3f7a16719584c9ab41de1358","created_at":"2023-07-11T22:01:01.682Z","meta.caller_id":"Cronjob","correlation_id":"e5d6d230bcc1ba22f8b77d3d440e1ad9","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","enqueued_at":"2023-07-11T22:01:01.682Z","job_size_bytes":2,"pid":724,"message":"Database::BatchedBackgroundMigrationWorker JID-3f7a16719584c9ab41de1358: start","job_status":"start","scheduling_latency_s":0.089611} {"severity":"INFO","time":"2023-07-11T22:01:01.808Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"e12828e650b98527bf69253a","created_at":"2023-07-11T22:01:01.522Z","meta.caller_id":"Cronjob","correlation_id":"fd2974abef2d7b37505b5164b0639350","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.523Z","job_size_bytes":2,"pid":724,"message":"Geo::SidekiqCronConfigWorker JID-e12828e650b98527bf69253a: done: 0.256185 sec","job_status":"done","scheduling_latency_s":0.0289,"redis_calls":163,"redis_duration_s":0.186317,"redis_read_bytes":59785,"redis_write_bytes":10827,"redis_queues_calls":163,"redis_queues_duration_s":0.186317,"redis_queues_read_bytes":59785,"redis_queues_write_bytes":10827,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.099912,"mem_objects":87463,"mem_bytes":85084800,"mem_mallocs":26338,"mem_total_bytes":88583320,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.256185,"completed_at":"2023-07-11T22:01:01.808Z","load_balancing_strategy":"primary","db_duration_s":0.0} {"severity":"INFO","time":"2023-07-11T22:01:01.681Z","class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","database":"ci","message":"skipping migration execution for database that shares database configuration with another database","retry":0} {"severity":"INFO","time":"2023-07-11T22:01:01.682Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Search::IndexCurationWorker","jid":"998cb532332d4bf81893c266","created_at":"2023-07-11T22:01:01.611Z","meta.caller_id":"Cronjob","correlation_id":"9f71fa8756d0bca0b61ea48562d01f92","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f1b35de73fa09aee87da69fc70f7f498516e5b000ec47c894d20a533c302983e","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.612Z","job_size_bytes":2,"pid":724,"message":"Search::IndexCurationWorker JID-998cb532332d4bf81893c266: start","job_status":"start","scheduling_latency_s":0.069551} {"severity":"INFO","time":"2023-07-11T22:01:01.814Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","jid":"227edf01fb5b6aa3a43e8c7e","created_at":"2023-07-11T22:01:01.586Z","meta.caller_id":"Cronjob","correlation_id":"c20ea674c4db7f18e0f618c2b7b76967","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857","enqueued_at":"2023-07-11T22:01:01.586Z","job_size_bytes":2,"pid":724,"message":"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-227edf01fb5b6aa3a43e8c7e: done: 0.22427 sec","job_status":"done","scheduling_latency_s":0.003477,"redis_calls":2,"redis_duration_s":0.001883,"redis_read_bytes":2,"redis_write_bytes":186,"redis_queues_calls":2,"redis_queues_duration_s":0.001883,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.002449,"mem_objects":1387,"mem_bytes":111880,"mem_mallocs":301,"mem_total_bytes":167360,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.22427,"completed_at":"2023-07-11T22:01:01.814Z","load_balancing_strategy":"primary","db_duration_s":0.0} {"severity":"INFO","time":"2023-07-11T22:01:01.810Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"401bab93bcb7ddac9bf5f3a6","created_at":"2023-07-11T22:01:01.601Z","meta.caller_id":"Cronjob","correlation_id":"73330aea1deff41d29b7f099f12ec377","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.602Z","job_size_bytes":2,"pid":724,"message":"ElasticIndexBulkCronWorker JID-401bab93bcb7ddac9bf5f3a6: done: 0.204303 sec","job_status":"done","scheduling_latency_s":0.004572,"redis_calls":2,"redis_duration_s":0.000283,"redis_read_bytes":2,"redis_write_bytes":186,"redis_queues_calls":2,"redis_queues_duration_s":0.000283,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.096,"db_main_duration_s":0.096,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.0055,"mem_objects":3289,"mem_bytes":399504,"mem_mallocs":1294,"mem_total_bytes":531064,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.204303,"completed_at":"2023-07-11T22:01:01.810Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.095865} {"severity":"INFO","time":"2023-07-11T22:01:01.832Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Search::IndexCurationWorker","jid":"998cb532332d4bf81893c266","created_at":"2023-07-11T22:01:01.611Z","meta.caller_id":"Cronjob","correlation_id":"9f71fa8756d0bca0b61ea48562d01f92","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f1b35de73fa09aee87da69fc70f7f498516e5b000ec47c894d20a533c302983e","size_limiter":"validated","enqueued_at":"2023-07-11T22:01:01.612Z","job_size_bytes":2,"pid":724,"message":"Search::IndexCurationWorker JID-998cb532332d4bf81893c266: done: 0.150505 sec","job_status":"done","scheduling_latency_s":0.069551,"redis_calls":2,"redis_duration_s":0.000258,"redis_read_bytes":2,"redis_write_bytes":186,"redis_queues_calls":2,"redis_queues_duration_s":0.000258,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.001994,"mem_objects":1432,"mem_bytes":113040,"mem_mallocs":321,"mem_total_bytes":170320,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.150505,"completed_at":"2023-07-11T22:01:01.832Z","load_balancing_strategy":"primary","db_duration_s":0.0} {"severity":"INFO","time":"2023-07-11T22:20:04.906Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","class":"Ci::StuckBuilds::DropRunningWorker","args":[],"jid":"2fa5e3fb6a0ebe746cd22fbc","created_at":"2023-07-11T22:00:03.228Z","meta.caller_id":"Cronjob","correlation_id":"750cd4cd9b944f1e4558ec9454827f34","meta.root_caller_id":"Cronjob","meta.feature_category":"continuous_integration","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","scheduled_at":"2023-07-11T22:20:03.228Z","idempotency_key":"resque:gitlab:duplicate:default:ea2a97d1a67ae4af54a74ae3b0ab2d60e87ef116206ede972bdbbdafa0a53cd4","enqueued_at":"2023-07-11T22:20:04.905Z","job_size_bytes":2,"pid":724,"message":"Ci::StuckBuilds::DropRunningWorker JID-2fa5e3fb6a0ebe746cd22fbc: start","job_status":"start","scheduling_latency_s":0.001329,"enqueue_latency_s":1.676986}
==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"01H53GTTKP4FKPPQJ0PB2JERJ2","duration_ms":10001,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2023-07-11T22:23:38Z","uri":"/help"} {"content_type":"text/plain","correlation_id":"01H53GTTKP4FKPPQJ0PB2JERJ2","duration_ms":10001,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":499,"system":"http","time":"2023-07-11T22:23:38Z","ttfb_ms":10001,"uri":"/help","user_agent":"curl/8.0.1-DEV","written_bytes":24} {"correlation_id":"01H53GWZ9PMXWWA3VKX7V9SSB9","duration_ms":10001,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2023-07-11T22:24:49Z","uri":"/help"} {"content_type":"text/plain","correlation_id":"01H53GWZ9PMXWWA3VKX7V9SSB9","duration_ms":10001,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":499,"system":"http","time":"2023-07-11T22:24:49Z","ttfb_ms":10001,"uri":"/help","user_agent":"curl/8.0.1-DEV","written_bytes":24} {"correlation_id":"01H53GWM5HD7S2VN0QBMCRS7B1","duration_ms":43399,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2023-07-11T22:25:11Z","uri":"/s4-plugins/netengine/-/milestones?state=opened"} {"content_type":"text/plain","correlation_id":"01H53GWM5HD7S2VN0QBMCRS7B1","duration_ms":43399,"host":"localhost:8010","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://REDACTED/s4-plugins/netengine/-/milestones","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":499,"system":"http","time":"2023-07-11T22:25:11Z","ttfb_ms":43399,"uri":"/s4-plugins/netengine/-/milestones?state=opened","user_agent":"Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)","written_bytes":24} {"correlation_id":"01H53GZ44FGPQN8Y61H6MBF8JD","duration_ms":10000,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2023-07-11T22:25:59Z","uri":"/help"} {"content_type":"text/plain","correlation_id":"01H53GZ44FGPQN8Y61H6MBF8JD","duration_ms":10000,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":499,"system":"http","time":"2023-07-11T22:25:59Z","ttfb_ms":10000,"uri":"/help","user_agent":"curl/8.0.1-DEV","written_bytes":24} {"correlation_id":"01H53H18VFPMDMN5ZD1SPYT5GW","duration_ms":10000,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"","time":"2023-07-11T22:27:10Z","uri":"/help"} {"content_type":"text/plain","correlation_id":"01H53H18VFPMDMN5ZD1SPYT5GW","duration_ms":10001,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":499,"system":"http","time":"2023-07-11T22:27:10Z","ttfb_ms":10001,"uri":"/help","user_agent":"curl/8.0.1-DEV","written_bytes":24}
==> /var/log/gitlab/postgres-exporter/state <==
==> /var/log/gitlab/postgres-exporter/current <== 2023-07-11_18:29:52.86548 ts=2023-07-11T18:29:52.865Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-07-11_18:29:52.86656 ts=2023-07-11T18:29:52.866Z caller=collector.go:196 level=error msg="collector failed" name=database duration_seconds=0.001010681 err="pq: the database system is starting up" 2023-07-11_18:29:52.86752 ts=2023-07-11T18:29:52.867Z caller=collector.go:196 level=error msg="collector failed" name=bgwriter duration_seconds=0.001891113 err="pq: the database system is starting up" 2023-07-11_18:29:53.86694 ts=2023-07-11T18:29:53.866Z caller=postgres_exporter.go:714 level=error err="Error opening connection to database (host=/var/opt/gitlab/postgresql%20user=gitlab-psql%20database=gitlabhq_production): pq: the database system is starting up" 2023-07-11_18:30:07.86626 ts=2023-07-11T18:30:07.866Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-07-11_18:30:07.86722 ts=2023-07-11T18:30:07.867Z caller=collector.go:196 level=error msg="collector failed" name=database duration_seconds=0.000900216 err="pq: the database system is starting up" 2023-07-11_18:30:07.86787 ts=2023-07-11T18:30:07.867Z caller=collector.go:196 level=error msg="collector failed" name=bgwriter duration_seconds=0.001421489 err="pq: the database system is starting up" 2023-07-11_18:30:08.86735 ts=2023-07-11T18:30:08.867Z caller=postgres_exporter.go:714 level=error err="Error opening connection to database (host=/var/opt/gitlab/postgresql%20user=gitlab-psql%20database=gitlabhq_production): pq: the database system is starting up" 2023-07-11_18:30:22.86635 ts=2023-07-11T18:30:22.866Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432 2023-07-11_18:30:22.96948 ts=2023-07-11T18:30:22.969Z caller=postgres_exporter.go:645 level=info msg="Semantic version changed" server=/var/opt/gitlab/postgresql:5432 from=0.0.0 to=13.11.0
==> /var/log/gitlab/puma/puma_stdout.log <== {"timestamp":"2023-07-09T10:23:56.167Z","pid":737,"message":"- Worker 7 (PID: 9326) booted in 0.59s, phase: 0"} {"timestamp":"2023-07-09T12:03:56.407Z","pid":737,"message":"- Worker 4 (PID: 12257) booted in 0.71s, phase: 0"} {"timestamp":"2023-07-09T12:18:54.549Z","pid":737,"message":"- Worker 3 (PID: 12711) booted in 0.45s, phase: 0"} {"timestamp":"2023-07-11T07:52:57.203Z","pid":737,"message":"! Terminating timed out worker (worker failed to check in within 60 seconds): 26698"} {"timestamp":"2023-07-11T07:53:32.425Z","pid":737,"message":"! Terminating timed out worker (worker failed to check in within 60 seconds): 1532"} {"timestamp":"2023-07-11T07:53:37.591Z","pid":737,"message":"! Terminating timed out worker (worker failed to check in within 60 seconds): 9326"} {"timestamp":"2023-07-11T07:53:41.723Z","pid":737,"message":"! Terminating timed out worker (worker failed to check in within 60 seconds): 31307"} === puma startup: 2023-07-11 08:07:35 +0000 === === puma startup: 2023-07-11 18:31:22 +0000 === {"timestamp":"2023-07-11T18:31:22.543Z","pid":654,"message":"- Worker 0 (PID: 778) booted in 0.35s, phase: 0"}
==> /var/log/gitlab/puma/state <==
==> /var/log/gitlab/puma/current <== 2023-07-11_18:31:22.04702 {"timestamp":"2023-07-11T18:31:22.046Z","pid":654,"message":"* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket"} 2023-07-11_18:31:22.04722 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"* Listening on http://127.0.0.1:8080"} 2023-07-11_18:31:22.04731 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"} 2023-07-11_18:31:22.04740 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! #\u003cThread:0x00007fa29cc27d88 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:91:in
sleep'"} 2023-07-11_18:31:22.04748 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! #\u003cThread:0x00007fa290014980 /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:81 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:83:in
sleep'"} 2023-07-11_18:31:22.04765 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"Use Ctrl-C to stop"} 2023-07-11_18:31:22.04768 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! WARNING: Detected running cluster mode with 1 worker."} 2023-07-11_18:31:22.04769 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! Running Puma in cluster mode with a single worker is often a misconfiguration."} 2023-07-11_18:31:22.04773 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead."} 2023-07-11_18:31:22.04776 {"timestamp":"2023-07-11T18:31:22.047Z","pid":654,"message":"! Set thesilence_single_worker_warning
option to silence this warning message."}==> /var/log/gitlab/puma/puma_stderr.log <== source=rack-timeout id=01H5244293JYVR32HYVQ38Z11D timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H52466XB2DY2QBBT25CXP73D timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H5248AKH1MWEVY2W00YH4A4G timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H5248BFVZ43C333XK2Q4HZBJ timeout=600000ms service=600001ms state=timed_out at=error === puma startup: 2023-07-11 18:31:22 +0000 === source=rack-timeout id=01H53FJ1D4B4WGV8CH06PHWFTB timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H53FJ7AQ1EDTG485BSPFTYJB timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H53FM679YTC4PAYG5TDTW10Q timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H53FPAV7YXP23EZPZ2QQN63R timeout=600000ms service=600000ms state=timed_out at=error source=rack-timeout id=01H53FRFET717RYF58GAAJASH6 timeout=600000ms service=600000ms state=timed_out at=error
==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <== {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:05:53Z","grpc.time_ms":0.163,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1184,"correlation_id":"66e07d6e399b72a0eb2e3fcea59b72bb","time":"2023-06-29T11:05:53.289Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:05:53Z","grpc.time_ms":0.208,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1185,"correlation_id":"67f8add28fc5c0c270c281e2e8cb4df8","time":"2023-06-29T11:05:53.289Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:08Z","grpc.time_ms":0.131,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1185,"correlation_id":"8a1903573351a7b19fc615d5dc46222d","time":"2023-06-29T11:06:08.291Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:08Z","grpc.time_ms":0.136,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1184,"correlation_id":"16921497dfdf06eb90ac1d4843d9393d","time":"2023-06-29T11:06:08.291Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:23Z","grpc.time_ms":0.115,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1184,"correlation_id":"2314b7a0c064989cb2d5d94cfc4da93e","time":"2023-06-29T11:06:23.293Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:23Z","grpc.time_ms":0.14,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1185,"correlation_id":"2faad52d605c268fd796f148ac680c39","time":"2023-06-29T11:06:23.293Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:38Z","grpc.time_ms":0.123,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1185,"correlation_id":"19a0602cf2fc9b215f5923605cc7dccd","time":"2023-06-29T11:06:38.295Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:38Z","grpc.time_ms":0.125,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1184,"correlation_id":"c772357d69a08e12262a1ae51a86dfb0","time":"2023-06-29T11:06:38.295Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:53Z","grpc.time_ms":0.108,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1185,"correlation_id":"4f9f6afa89b5ae1b4350b0eebd996d92","time":"2023-06-29T11:06:53.297Z"} {"type":"gitaly-ruby","grpc.start_time":"2023-06-29T11:06:53Z","grpc.time_ms":0.148,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":1184,"correlation_id":"57d754a199edf258d5cbd474ba08da95","time":"2023-06-29T11:06:53.297Z"}
==> /var/log/gitlab/alertmanager/state <==
==> /var/log/gitlab/alertmanager/current <== 2023-07-11_18:28:48.99360 ts=2023-07-11T18:28:48.993Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=master, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2023-07-11_18:28:48.99367 ts=2023-07-11T18:28:48.993Z caller=main.go:241 level=info build_context="(go=go1.19.8, user=GitLab-Omnibus, date=)" 2023-07-11_18:28:48.99621 ts=2023-07-11T18:28:48.996Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.18.0.2 port=9094 2023-07-11_18:28:49.00556 ts=2023-07-11T18:28:49.005Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-07-11_18:28:49.04502 ts=2023-07-11T18:28:49.044Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2023-07-11_18:28:49.05822 ts=2023-07-11T18:28:49.058Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2023-07-11_18:28:49.10531 ts=2023-07-11T18:28:49.063Z caller=tls_config.go:232 level=info msg="Listening on" address=127.0.0.1:9093 2023-07-11_18:28:49.10533 ts=2023-07-11T18:28:49.105Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=127.0.0.1:9093 2023-07-11_18:28:51.00583 ts=2023-07-11T18:28:51.005Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000209024s 2023-07-11_18:28:59.00810 ts=2023-07-11T18:28:59.008Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002507878s
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <== 2023-07-11_18:30:19.25702 FATAL: the database system is starting up 2023-07-11_18:30:20.99848 LOG: database system was not properly shut down; automatic recovery in progress 2023-07-11_18:30:21.09308 LOG: invalid record length at 0/480111B0: wanted 24, got 0 2023-07-11_18:30:21.09310 LOG: redo is not required 2023-07-11_18:30:21.52190 FATAL: the database system is starting up 2023-07-11_18:30:21.55122 FATAL: the database system is starting up 2023-07-11_18:30:21.55227 FATAL: the database system is starting up 2023-07-11_18:30:21.68676 FATAL: the database system is starting up 2023-07-11_18:30:21.69289 FATAL: the database system is starting up 2023-07-11_18:30:21.70278 LOG: database system is ready to accept connections
==> /var/log/gitlab/nginx/gitlab_registry_error.log <==
==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
==> /var/log/gitlab/nginx/access.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <== 2022/03/22 22:14:56 [crit] 650#0: *493 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2022/03/22 22:21:04 [crit] 645#0: *349 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2022/06/24 14:11:05 [crit] 399#0: *1723197 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2022/06/24 14:17:47 [crit] 399#0: *186 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2022/09/04 18:37:54 [crit] 383#0: *278 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2023/02/19 00:25:26 [crit] 413#0: *21656 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: REDACTED, request: "GET /help HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/help", host: "localhost" 2023/02/19 00:25:26 [crit] 413#0: *21658 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010" 2023/06/29 11:07:10 [crit] 654#0: *3404 connect() to unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket failed (2: No such file or directory) while connecting to upstream, client: 172.18.0.1, server: REDACTED, request: "POST /api/v4/jobs/request HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/api/v4/jobs/request", host: "localhost:8010"
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/gitlab_access.log <== 127.0.0.1 - - [11/Jul/2023:22:17:47 +0000] "GET /help HTTP/1.1" 499 0 "" "curl/8.0.1-DEV" - 127.0.0.1 - - [11/Jul/2023:22:25:59 +0000] "GET /help HTTP/1.1" 499 0 "" "curl/8.0.1-DEV" - 127.0.0.1 - - [11/Jul/2023:22:27:10 +0000] "GET /help HTTP/1.1" 499 0 "" "curl/8.0.1-DEV" -
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/gitlab-kas/state <==
==> /var/log/gitlab/gitlab-kas/current <== 2023-07-11_18:28:38.49500 {"level":"info","time":"2023-07-11T18:28:38.467Z","msg":"API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8153"} 2023-07-11_18:28:38.49514 {"level":"info","time":"2023-07-11T18:28:38.467Z","msg":"Private API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8155"} 2023-07-11_18:28:38.49521 {"level":"info","time":"2023-07-11T18:28:38.467Z","msg":"Agentk API endpoint is up","net_network":"tcp","net_address":"127.0.0.1:8150","is_websocket":true} 2023-07-11_18:28:38.52969 {"level":"info","time":"2023-07-11T18:28:38.529Z","msg":"Kubernetes API endpoint is up","mod_name":"kubernetes_api","net_network":"tcp","net_address":"127.0.0.1:8154"} 2023-07-11_18:28:38.52980 {"level":"info","time":"2023-07-11T18:28:38.529Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":"127.0.0.1:8151"}
==> /var/log/gitlab/gitlab-exporter/state <==
==> /var/log/gitlab/gitlab-exporter/current <== 2023-07-11_22:00:11.13900 127.0.0.1 - - [11/Jul/2023:22:00:10 UTC] "GET /database HTTP/1.1" 200 1795 2023-07-11_22:00:11.13917 - -> /database 2023-07-11_22:00:15.08677 127.0.0.1 - - [11/Jul/2023:22:00:15 UTC] "GET /ruby HTTP/1.1" 200 1074 2023-07-11_22:00:15.08682 - -> /ruby 2023-07-11_22:00:17.69650 127.0.0.1 - - [11/Jul/2023:22:00:17 UTC] "GET /sidekiq HTTP/1.1" 200 145992 2023-07-11_22:00:17.69654 - -> /sidekiq 2023-07-11_22:00:30.08460 127.0.0.1 - - [11/Jul/2023:22:00:30 UTC] "GET /ruby HTTP/1.1" 200 1076 2023-07-11_22:00:30.08468 - -> /ruby 2023-07-11_22:00:32.69745 127.0.0.1 - - [11/Jul/2023:22:00:32 UTC] "GET /sidekiq HTTP/1.1" 200 145992 2023-07-11_22:00:32.69750 - -> /sidekiq
==> /var/log/gitlab/redis/state <==
==> /var/log/gitlab/redis/current <== 2023-07-11_21:53:36.32608 5548:C 11 Jul 2023 21:53:36.326 * DB saved on disk 2023-07-11_21:53:36.32637 5548:C 11 Jul 2023 21:53:36.326 * RDB: 1 MB of memory used by copy-on-write 2023-07-11_21:53:36.39912 276:M 11 Jul 2023 21:53:36.399 * Background saving terminated with success 2023-07-11_21:58:37.00012 276:M 11 Jul 2023 21:58:37.000 * 10 changes in 300 seconds. Saving... 2023-07-11_21:58:37.00045 276:M 11 Jul 2023 21:58:37.000 * Background saving started by pid 5657 2023-07-11_21:58:37.20708 5657:C 11 Jul 2023 21:58:37.207 * DB saved on disk 2023-07-11_21:58:37.20735 5657:C 11 Jul 2023 21:58:37.207 * RDB: 1 MB of memory used by copy-on-write 2023-07-11_21:58:37.30126 276:M 11 Jul 2023 21:58:37.301 * Background saving terminated with success 2023-07-11_22:03:38.04207 276:M 11 Jul 2023 22:03:38.042 * 10 changes in 300 seconds. Saving... 2023-07-11_22:03:38.04267 276:M 11 Jul 2023 22:03:38.042 * Background saving started by pid 5756
==> /var/log/gitlab/registry/state <==
==> /var/log/gitlab/registry/current <== 2022-09-04_18:38:22.98877 time="2022-09-04T18:38:22.988Z" level=info msg="listening on 127.0.0.1:5000" environment=production go_version=go1.17.6 instance_id=ec415078-682a-4ab6-b5a4-a56a16da1a27 service=registry version=v3.31.0-gitlab 2022-09-04_18:39:13.53518 time="2022-09-04T18:39:13.535Z" level=info msg="attempting to stop server gracefully..." database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated 2022-09-04_18:39:13.53519 time="2022-09-04T18:39:13.535Z" level=info msg="graceful shutdown successful" database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated 2022-09-04_18:39:13.57765 time="2022-09-04T18:39:13.577Z" level=info msg="redis not configured" environment=production go_version=go1.17.6 instance_id=342e3e5e-682c-46fa-9714-b3c69c24466b service=registry version=v3.31.0-gitlab 2022-09-04_18:39:13.57776 time="2022-09-04T18:39:13.577Z" level=info msg="storage backend redirection enabled" environment=production go_version=go1.17.6 instance_id=342e3e5e-682c-46fa-9714-b3c69c24466b service=registry version=v3.31.0-gitlab 2022-09-04_18:39:13.57782 time="2022-09-04T18:39:13.577Z" level=info msg="using inmemory blob descriptor cache" environment=production go_version=go1.17.6 instance_id=342e3e5e-682c-46fa-9714-b3c69c24466b service=registry version=v3.31.0-gitlab 2022-09-04_18:39:13.57807 time="2022-09-04T18:39:13.577Z" level=info msg="Starting upload purge in 21m0s" environment=production go_version=go1.17.6 instance_id=342e3e5e-682c-46fa-9714-b3c69c24466b service=registry version=v3.31.0-gitlab 2022-09-04_18:39:13.58212 time="2022-09-04T18:39:13.582Z" level=info msg="listening on 127.0.0.1:5000" environment=production go_version=go1.17.6 instance_id=342e3e5e-682c-46fa-9714-b3c69c24466b service=registry version=v3.31.0-gitlab 2022-09-04_18:51:55.52492 time="2022-09-04T18:51:55.524Z" level=info msg="attempting to stop server gracefully..." database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated 2022-09-04_18:51:55.52495 time="2022-09-04T18:51:55.524Z" level=info msg="graceful shutdown successful" database_drain_timeout=0s http_drain_timeout=0s quit_signal=terminated
==> /var/log/gitlab/prometheus/state <==
==> /var/log/gitlab/prometheus/current <== 2023-07-11_20:29:11.72739 ts=2023-07-11T20:29:11.727Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=18.426754ms 2023-07-11_20:29:11.73490 ts=2023-07-11T20:29:11.734Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=8754 to_segment=8755 mint=1689098400000 2023-07-11_20:29:12.30741 ts=2023-07-11T20:29:12.307Z caller=head.go:1241 level=info component=tsdb msg="WAL checkpoint complete" first=8754 last=8755 duration=577.516699ms 2023-07-11_21:29:10.79747 ts=2023-07-11T21:29:10.797Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1689100147021 maxt=1689105600000 ulid=01H53DQAV0ZJJZWC64T5P2QPP0 duration=2.092399828s 2023-07-11_21:29:11.03373 ts=2023-07-11T21:29:11.033Z caller=db.go:1548 level=info component=tsdb msg="Deleting obsolete block" block=01H3WAC0VARPZN61C3Q3YJ72BK 2023-07-11_21:29:11.09169 ts=2023-07-11T21:29:11.091Z caller=head.go:1269 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=57.881811ms 2023-07-11_21:29:21.66373 ts=2023-07-11T21:29:21.663Z caller=compact.go:460 level=info component=tsdb msg="compact blocks" count=3 mint=1688947200241 maxt=1688968800000 ulid=01H53DQD8K8G92TD47KB1MAKGP sources="[01H4YVVTADRA6PGYNK8VK07BE7 01H4Z2QJ1ESM7BRERC7NWV9AK3 01H51ZRNBPA1RE9YVSRT237CGT]" duration=10.485422146s 2023-07-11_21:29:21.83660 ts=2023-07-11T21:29:21.836Z caller=db.go:1548 level=info component=tsdb msg="Deleting obsolete block" block=01H4Z2QJ1ESM7BRERC7NWV9AK3 2023-07-11_21:29:21.87637 ts=2023-07-11T21:29:21.876Z caller=db.go:1548 level=info component=tsdb msg="Deleting obsolete block" block=01H51ZRNBPA1RE9YVSRT237CGT 2023-07-11_21:29:21.92215 ts=2023-07-11T21:29:21.922Z caller=db.go:1548 level=info component=tsdb msg="Deleting obsolete block" block=01H4YVVTADRA6PGYNK8VK07BE7
==> /var/log/gitlab/logrotate/current <== 2023-07-11_21:37:56.01420 warning: Potentially dangerous mode on /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse: 0777 2023-07-11_21:37:56.01422 error: Ignoring /var/opt/gitlab/logrotate/logrotate.d/gitlab-workhorse because it is writable by group or others. 2023-07-11_21:37:56.02076 warning: Potentially dangerous mode on /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages: 0777 2023-07-11_21:37:56.02081 error: Ignoring /var/opt/gitlab/logrotate/logrotate.d/gitlab-pages because it is writable by group or others. 2023-07-11_21:37:56.02825 warning: Potentially dangerous mode on /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas: 0777 2023-07-11_21:37:56.02828 error: Ignoring /var/opt/gitlab/logrotate/logrotate.d/gitlab-kas because it is writable by group or others. 2023-07-11_21:37:56.03536 warning: Potentially dangerous mode on /var/opt/gitlab/logrotate/logrotate.d/gitaly: 0777 2023-07-11_21:37:56.03539 error: Ignoring /var/opt/gitlab/logrotate/logrotate.d/gitaly because it is writable by group or others. 2023-07-11_21:37:56.04305 warning: Potentially dangerous mode on /var/opt/gitlab/logrotate/logrotate.d/mailroom: 0777 2023-07-11_21:37:56.04307 error: Ignoring /var/opt/gitlab/logrotate/logrotate.d/mailroom because it is writable by group or others.
==> /var/log/gitlab/gitlab-rails/production.log <==
Rack::Timeout::RequestTimeoutException (Request ran for longer than 600000ms ):
lib/gitlab/database/postgresql_adapter/empty_query_ping.rb:15:in
exec' lib/gitlab/database/postgresql_adapter/empty_query_ping.rb:15:in
block in active?' lib/gitlab/database/postgresql_adapter/empty_query_ping.rb:14:inactive?' lib/gitlab/database/load_balancing/load_balancer.rb:110:in
read_write' lib/gitlab/database/load_balancing/connection_proxy.rb:126:inwrite_using_load_balancer' lib/gitlab/database/load_balancing/connection_proxy.rb:96:in
method_missing' app/models/concerns/cacheable_attributes.rb:19:incurrent_without_cache' app/models/concerns/cacheable_attributes.rb:55:in
current' lib/gitlab/current_settings.rb:60:inuncached_application_settings' lib/gitlab/current_settings.rb:42:in
ensure_application_settings!' lib/gitlab/current_settings.rb:15:inblock in current_application_settings' lib/gitlab/safe_request_store.rb:12:in
fetch' lib/gitlab/current_settings.rb:15:incurrent_application_settings' lib/gitlab/throttle.rb:13:in
settings' lib/gitlab/rack_attack/request.rb:95:inthrottle_unauthenticated_web?' lib/gitlab/rack_attack.rb:104:in
block in configure_throttles' lib/gitlab/middleware/read_only/controller.rb:50:incall' lib/gitlab/middleware/read_only.rb:18:in
call' lib/gitlab/middleware/same_site_cookies.rb:27:incall' lib/gitlab/middleware/basic_health_check.rb:25:in
call' lib/gitlab/middleware/handle_malformed_strings.rb:21:incall' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in
call' lib/gitlab/middleware/request_context.rb:15:incall' lib/gitlab/middleware/webhook_recursion_detection.rb:15:in
call' config/initializers/fix_local_cache_middleware.rb:11:incall' lib/gitlab/middleware/compressed_json.rb:44:in
call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:incall' lib/gitlab/middleware/sidekiq_web_static.rb:20:in
call' lib/gitlab/metrics/requests_rack_middleware.rb:79:incall' lib/gitlab/middleware/release_env.rb:13:in
call'
top
Output
top
Outputtop - 23:53:10 up 5:28, 0 users, load average: 58.57, 58.41, 58.29
Tasks: 121 total, 2 running, 119 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 7.2 sy, 0.0 ni, 73.4 id, 16.9 wa, 0.0 hi, 0.7 si, 0.0 st
MiB Mem : 9949.1 total, 1996.1 free, 5232.7 used, 2720.4 buff/cache
MiB Swap: 32768.0 total, 32768.0 free, 0.0 used. 4367.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
778 git 20 0 2207896 1.2g 18868 R 100.0 12.1 100:07.25 ruby
446 gitlab-+ 20 0 5313072 309052 71256 S 4.3 3.0 6:17.68 prometheus
724 git 20 0 3610264 1.2g 28220 S 1.3 12.0 17:39.80 ruby
276 gitlab-+ 20 0 70428 15812 6052 S 0.7 0.2 2:36.00 redis-server
444 gitlab-+ 20 0 1802924 52112 21084 S 0.3 0.5 0:12.69 alertmanager
1 root 20 0 4912 3488 3148 S 0.0 0.0 0:00.04 wrapper
24 root 20 0 2808 1312 1212 S 0.0 0.0 0:00.12 runsvdir
30 root 20 0 2656 1068 980 S 0.0 0.0 0:00.00 runsv
31 root 20 0 2804 1072 980 S 0.0 0.0 0:00.00 svlogd
32 root 20 0 15424 8568 7012 S 0.0 0.1 0:00.01 sshd
269 root 20 0 2656 1060 972 S 0.0 0.0 0:00.00 runsv
270 root 20 0 2656 1072 984 S 0.0 0.0 0:00.00 runsv
271 root 20 0 2656 1040 952 S 0.0 0.0 0:00.00 runsv
272 root 20 0 2804 1056 972 S 0.0 0.0 0:00.00 svlogd
273 root 20 0 2804 1080 996 S 0.0 0.0 0:00.00 svlogd
274 root 20 0 2804 1080 992 S 0.0 0.0 0:00.00 svlogd
310 git 20 0 1328484 19756 7152 S 0.0 0.2 0:01.88 gitaly-wrapper
316 git 20 0 2161168 286540 157828 S 0.0 2.8 0:31.64 gitaly
351 root 20 0 2656 1084 992 S 0.0 0.0 0:00.00 runsv
352 root 20 0 2804 1112 1020 S 0.0 0.0 0:00.00 svlogd
353 gitlab-+ 20 0 49020 28676 26816 D 0.0 0.3 0:00.36 postgres
378 root 20 0 2656 1064 972 S 0.0 0.0 0:00.00 runsv
379 root 20 0 2656 1060 972 S 0.0 0.0 0:00.00 runsv
gitlab-rake gitlab:check
does not return during the outage periods, but does not return any errors/problems during working time periods
Details of package version
Provide the package version installation details
Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-============================================================ un gitlab-ce (no description available) ii gitlab-ee 16.1.2-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) un gitlab-fips (no description available)
Environment details
- Operating System: WSL Windows Docker
- Installation Target, remove incorrect values:
- Docker 24.0.0.2
- Installation Type, remove incorrect values:
- Upgrade from version 14.9.0-ee (Upgraded with all necessary according to the upgrade manual)
- Is this a single or multiple node installation? Single Node
- Resources
- CPU: 16 Cores
- Memory total: 10GB with 32GB swap
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url '"**REDACTED**'gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = ""REDACTED" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = ""REDACTED" gitlab_rails['smtp_password'] = ""REDACTED" gitlab_rails['smtp_domain'] = ""REDACTED" gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['smtp_tls'] = false gitlab_rails['smtp_openssl_verify_mode'] = 'peer' gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_default_theme'] = 2 gitlab_rails['lfs_enabled'] = true gitlab_rails['omniauth_enabled'] = nil gitlab_rails['omniauth_allow_single_sign_on'] = ['github'] gitlab_rails['omniauth_providers'] = [ { "name" => "github", "app_id" => ""REDACTED", "app_secret" => "REDACTED", "args" => { "scope" => "user:email"} } ] gitlab_rails['env'] = { 'GITLAB_RAILS_RACK_TIMEOUT' => 600 } puma['per_worker_max_memory_mb'] = 2048 letsencrypt['enable'] = false