Upgrade to GitLab Chart 7.x results in reconcilation loop
Summary
Upgrading from Chart version 6.11.9 to 7.1.0 results in an infinite reconcilation loop, unfortunately I was not able to find out what the exact issue is.
Is it necessary to manually upgrade the Postgres as described in https://docs.gitlab.com/charts/releases/7_0.html#postgresql ?
Steps to reproduce
- Install GitLab Operator to version
0.21.0 - Upgrade GitLab using
spec.chart.versionfrom6.11.9to7.1.0
Configuration used
apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
name: gitlab
namespace: gitlab-system
spec:
chart:
version: "6.11.9" # -> 7.1.0
values:
global:
edition: ce
appConfig:
backups:
bucket: *******
tmpBucket: *******
omniauth:
enabled: true
allowSingleSignOn: ["openid_connect"]
autoLinkLdapUser: false
blockAutoCreatedUsers: true
providers:
- secret: *******
key: *******
hpa:
apiVersion: "autoscaling/v2"
batch:
cronJob:
apiVersion: "batch/v1"
hosts:
domain: *******
ingress:
enabled: false
configureCertmanager: false
minio:
persistence:
enabled: true
size: 220Gi
certmanager:
install: false
certmanager-issuer:
enabled: false
nginx-ingress:
enabled: false
prometheus:
install: false
gitlab-runner:
install: true
postgresql:
image:
tag: 13.8.0
gitlab:
gitaly:
persistence:
enabled: true
size: 200Gi
toolbox:
backups:
cron:
enabled: true
schedule: "0 0 * * *"
objectStorage:
backend: s3
config:
secret: *******
key: *******
persistence:
enabled: false
size: 400Gi
Current behavior
After upgrading, the pods look like this:
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
gitlab-controller-manager-67dbfbc459-v6bwx 2/2 Running 0 59m
gitlab-gitaly-0 1/1 Running 0 2m26s
gitlab-gitlab-exporter-56d74b69d6-q2wdr 1/1 Running 0 48m
gitlab-gitlab-shell-6956669f9b-66xnt 1/1 Running 0 48m
gitlab-gitlab-shell-6956669f9b-vhhd8 1/1 Running 0 47m
gitlab-kas-d77f8c697-j5blr 1/1 Running 0 47m
gitlab-kas-d77f8c697-qxb7j 1/1 Running 0 48m
gitlab-migrations-1-885-3-h7l5c 0/1 Completed 0 48m
gitlab-migrations-1-885-5-vd4wp 0/1 Completed 0 19m
gitlab-minio-6f4b59ff6d-95zkh 1/1 Running 0 2m26s
gitlab-minio-create-buckets-1-7n4ts 0/1 Completed 0 51m
gitlab-postgresql-0 2/2 Running 0 48m
gitlab-redis-master-0 2/2 Running 0 51m
gitlab-registry-7d8b454c4f-ctbtn 1/1 Running 0 47m
gitlab-registry-7d8b454c4f-wnxq4 1/1 Running 0 48m
gitlab-shared-secrets-1-256-wm8x4 0/1 Completed 0 36m
gitlab-shared-secrets-1-432-selfsign-jxpk7 0/1 Completed 0 49m
gitlab-shared-secrets-1-ckn-selfsign-pkp9k 0/1 Completed 0 2m48s
gitlab-shared-secrets-1-fjo-selfsign-445sh 0/1 Completed 0 51m
gitlab-shared-secrets-1-lp0-selfsign-2x8gl 0/1 Completed 0 20m
gitlab-shared-secrets-1-n6f-lfc9f 0/1 Completed 0 49m
gitlab-shared-secrets-1-q0z-selfsign-xv9j4 0/1 Completed 0 35m
gitlab-shared-secrets-1-qat-nxbgm 0/1 Completed 0 20m
gitlab-shared-secrets-1-rom-n85kd 0/1 Completed 0 3m8s
gitlab-shared-secrets-1-z5x-xcdpm 0/1 Completed 0 52m
gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv 1/1 Running 0 47m
gitlab-toolbox-6bd9d5ff59-77gpc 1/1 Running 0 48m
gitlab-webservice-default-7695f9dfc-d6pnw 2/2 Running 0 47m
gitlab-webservice-default-7695f9dfc-xrcj4 2/2 Running 0 47mand the operator is reconciling:
2023-06-25T09:22:10Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:22:10Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:22:10Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:22:10Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}The gitlab instance is still reachable and useable
Expected behavior
A successful upgrade
Versions
- Operator: 0.21.0
- Platform:
- Cloud: Hetzner Cloud
- Self-hosted: Kubernetes
- Kubernetes: (
kubectl version)- Client: 1.27.1
- Server: 1.25
Relevant logs
Operator Logs:
2023-06-25T09:39:42Z INFO template Rendering a new template. {"namespace": "gitlab-system", "releaseName": "gitlab", "hash": "14e63735-a37a-4461-bb30-1719fcef3885-6"}
2023-06-25T09:41:11Z DEBUG template The template is rendered. Check the warnings (if any). {"namespace": "gitlab-system", "releaseName": "gitlab", "hash": "14e63735-a37a-4461-bb30-1719fcef3885-6", "warnings": 0}
2023-06-25T09:41:11Z DEBUG template Caching the template. {"namespace": "gitlab-system", "releaseName": "gitlab", "hash": "14e63735-a37a-4461-bb30-1719fcef3885-6"}
2023-06-25T09:41:11Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.ConfigMap", "reference": "gitlab-system/gitlab-shared-secrets", "outcome": "updated"}
2023-06-25T09:41:11Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Job", "reference": "gitlab-system/gitlab-shared-secrets-1-rom", "outcome": "created"}
2023-06-25T09:41:11Z ERROR Reconciler error {"controller": "gitlab", "controllerGroup": "apps.gitlab.com", "controllerKind": "GitLab", "GitLab": {"name":"gitlab","namespace":"gitlab-system"}, "namespace": "gitlab-system", "name": "gitlab", "reconcileID": "c1e395f7-bc8d-4c34-8129-234f4b3104a6", "error": "Job.batch \"gitlab-shared-secrets-1-rom\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235
2023-06-25T09:41:11Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:11Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:11Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:11Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:21Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:21Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:31Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:31Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:31Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Job", "reference": "gitlab-system/gitlab-shared-secrets-1-ckn-selfsign", "outcome": "created"}
2023-06-25T09:41:31Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:32Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:41Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:41Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:51Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.ConfigMap", "reference": "gitlab-system/gitlab-postgresql-init-db", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-postgresql", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-postgresql-metrics", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.ConfigMap", "reference": "gitlab-system/gitlab-redis-health", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.ConfigMap", "reference": "gitlab-system/gitlab-redis-scripts", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-redis-headless", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-redis-master", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-redis-metrics", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.ConfigMap", "reference": "gitlab-system/gitlab-gitaly", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.StatefulSet", "reference": "gitlab-system/gitlab-gitaly", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Service", "reference": "gitlab-system/gitlab-gitaly", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab CreateOrPatch {"gitlab": "gitlab-system/gitlab", "type": "*v1.Deployment", "reference": "gitlab-system/gitlab-minio", "outcome": "updated"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:41:52Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:41:52Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:41:52Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:41:52Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:02Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:02Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:02Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:02Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:12Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:12Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:12Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:12Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:22Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:22Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:22Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:22Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:32Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:32Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:32Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:32Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:42Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:42Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:43Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:43Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
2023-06-25T09:42:53Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
2023-06-25T09:42:53Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
2023-06-25T09:42:53Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
2023-06-25T09:42:53Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
...Complete logs of namespace while operator is looping:
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:46:56Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:46:56Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:46:56Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:46:56Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}
[gitlab-webservice-default-7695f9dfc-xrcj4 gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ628KK5N8XN5G5B3RBM22","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.77.243:32768","remote_ip":"10.42.77.243","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:46:56Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-gitlab-shell-6956669f9b-66xnt gitlab-shell] {"component": "gitlab-shell","subcomponent":"ssh","level":"unknown","time":"2023-06-25T09:46:58Z","message":"kex_exchange_identification: Connection closed by remote host\r"}
[gitlab-webservice-default-7695f9dfc-xrcj4 gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ64S29FP8P9FK4D25FZRK","duration_ms":1,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.137.133:54138","remote_ip":"10.42.137.133","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:46:59Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-webservice-default-7695f9dfc-d6pnw gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ656EPTAGGM5PZBG7T8K7","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.77.243:35302","remote_ip":"10.42.77.243","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:46:59Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-webservice-default-7695f9dfc-xrcj4 webservice] {"component": "gitlab","subcomponent":"production_json","level":"info","method":"GET","path":"/-/readiness","format":"*/*","controller":"HealthController","action":"readiness","status":200,"time":"2023-06-25T09:47:00.098Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.008235,"mem_objects":1769,"mem_bytes":49912,"mem_mallocs":385,"mem_total_bytes":120672,"pid":40,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"f5124de4-ea7b-46e0-86c8-7e65896983eb","db_duration_s":0.0,"view_duration_s":0.00032,"duration_s":0.00119}
[gitlab-gitlab-shell-6956669f9b-vhhd8 gitlab-shell] {"component": "gitlab-shell","subcomponent":"ssh","level":"unknown","time":"2023-06-25T09:47:00Z","message":"kex_exchange_identification: Connection closed by remote host\r"}
[gitlab-webservice-default-7695f9dfc-d6pnw webservice] {"component": "gitlab","subcomponent":"production_json","level":"info","method":"GET","path":"/-/readiness","format":"*/*","controller":"HealthController","action":"readiness","status":200,"time":"2023-06-25T09:47:00.535Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.005577,"mem_objects":1769,"mem_bytes":49912,"mem_mallocs":385,"mem_total_bytes":120672,"pid":40,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"6c2a0c97-f820-4b32-bd9f-fec70ad6914d","db_duration_s":0.0,"view_duration_s":0.00019,"duration_s":0.00191}
[gitlab-webservice-default-7695f9dfc-d6pnw gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ67PYS762CVFAS5SW7PZ3","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.137.133:34132","remote_ip":"10.42.137.133","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:47:02Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-webservice-default-7695f9dfc-xrcj4 gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ684ACK8VHXZ5QP66VXNF","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.77.243:32768","remote_ip":"10.42.77.243","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:47:02Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-gitlab-shell-6956669f9b-66xnt gitlab-shell] {"component": "gitlab-shell","subcomponent":"ssh","level":"unknown","time":"2023-06-25T09:47:03Z","message":"kex_exchange_identification: Connection closed by remote host\r"}
[gitlab-webservice-default-7695f9dfc-xrcj4 webservice] {"component": "gitlab","subcomponent":"production_json","level":"info","method":"GET","path":"/-/readiness","format":"*/*","controller":"HealthController","action":"readiness","status":200,"time":"2023-06-25T09:47:05.102Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.010916,"mem_objects":1769,"mem_bytes":49912,"mem_mallocs":385,"mem_total_bytes":120672,"pid":40,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"6bc8e47f-3b18-48be-a50b-72368d39e1d1","db_duration_s":0.0,"view_duration_s":0.00035,"duration_s":0.00139}
[gitlab-webservice-default-7695f9dfc-xrcj4 gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ6AMQXFSQN552A392A4VR","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.137.133:54138","remote_ip":"10.42.137.133","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:47:05Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-gitlab-shell-6956669f9b-vhhd8 gitlab-shell] {"component": "gitlab-shell","subcomponent":"ssh","level":"unknown","time":"2023-06-25T09:47:05Z","message":"kex_exchange_identification: Connection closed by remote host\r"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.506Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:database_batched_background_migration", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"Database::BatchedBackgroundMigrationWorker", "jid"=>"3c91b3b9b738ab76da9d19f0", "created_at"=>1687686425.5046844, "meta.caller_id"=>"Cronjob", "correlation_id"=>"1f66fa345cf1b1083c3def64c4bd1bfa", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"database", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:database_batched_background_migration:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae", "duplicate-of"=>"c467b6a3a21864405be869b9", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"Database::BatchedBackgroundMigrationWorker JID-3c91b3b9b738ab76da9d19f0: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.511Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:loose_foreign_keys_cleanup", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"LooseForeignKeys::CleanupWorker", "jid"=>"9e9f379f3fcebc2a4529f834", "created_at"=>1687686425.5098588, "meta.caller_id"=>"Cronjob", "correlation_id"=>"158b80db0189d42c9576ee3d6b5f3442", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"cell", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:loose_foreign_keys_cleanup:2180384c8554b410820afa81ad3479301139c7627524f882c6bca44488742df9", "duplicate-of"=>"ba4798ece0c0b78b201f017a", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"LooseForeignKeys::CleanupWorker JID-9e9f379f3fcebc2a4529f834: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.517Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:metrics_global_metrics_update", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"Metrics::GlobalMetricsUpdateWorker", "jid"=>"069085c74896a5637ab72f8b", "created_at"=>1687686425.5163686, "meta.caller_id"=>"Cronjob", "correlation_id"=>"7b5b240f026217a05396bd4046b5e121", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"metrics", "worker_data_consistency"=>:sticky, "wal_locations"=>{}, "wal_location_source"=>:primary, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:metrics_global_metrics_update:0ef60401f7cd15675ace9e5ac4456efd82434f42e048584516e1e7a85b24e5ba", "duplicate-of"=>"801abcf6e2c9b024d548cb32", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"Metrics::GlobalMetricsUpdateWorker JID-069085c74896a5637ab72f8b: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-webservice-default-7695f9dfc-d6pnw webservice] {"component": "gitlab","subcomponent":"production_json","level":"info","method":"GET","path":"/-/readiness","format":"*/*","controller":"HealthController","action":"readiness","status":200,"time":"2023-06-25T09:47:05.535Z","params":[],"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_main_replica_duration_s":0.0,"cpu_s":0.005855,"mem_objects":1769,"mem_bytes":49912,"mem_mallocs":385,"mem_total_bytes":120672,"pid":40,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"d2e083be-235b-483c-b974-3ff102583a90","db_duration_s":0.0,"view_duration_s":0.00025,"duration_s":0.00086}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.561Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:user_status_cleanup_batch", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"UserStatusCleanup::BatchWorker", "jid"=>"1b7c7e85025449592c328843", "created_at"=>1687686425.5605705, "meta.caller_id"=>"Cronjob", "correlation_id"=>"1fdbbc5f29ef518d065c09dc28e4979f", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"user_profile", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:user_status_cleanup_batch:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf", "duplicate-of"=>"d36e61ac6f8b32acd978bc43", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"UserStatusCleanup::BatchWorker JID-1b7c7e85025449592c328843: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.610Z pid=24 tid=6bb8 class=CiPlatformMetricsUpdateCronWorker jid=8d84b1f44553270d1f93f135 INFO: start
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.613Z pid=24 tid=6bb8 class=CiPlatformMetricsUpdateCronWorker jid=8d84b1f44553270d1f93f135 INFO: arguments: []
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.656Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:schedule_merge_request_cleanup_refs", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"ScheduleMergeRequestCleanupRefsWorker", "jid"=>"6bf443d347e0874de43cd540", "created_at"=>1687686425.654613, "meta.caller_id"=>"Cronjob", "correlation_id"=>"f04c8d9f0a2ac851b79eb17b27c6d5cb", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"code_review_workflow", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:schedule_merge_request_cleanup_refs:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811", "duplicate-of"=>"668af1c4e9f747c46c784c7a", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"ScheduleMergeRequestCleanupRefsWorker JID-6bf443d347e0874de43cd540: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-webservice-default-7695f9dfc-d6pnw gitlab-workhorse] {"content_type":"","correlation_id":"01H3RZ6B25Q86VT37MPNVPPAKA","duration_ms":0,"host":"gitlab.claivolution.de","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.42.77.243:35302","remote_ip":"10.42.77.243","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-06-25T09:47:05Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.1.0 (15-1-stable; go1.17.7; linux/amd64)","written_bytes":0}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.681Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:database_batched_background_migration_ci_database", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", "jid"=>"4f27c07d50d460c86f7d62df", "created_at"=>1687686425.6794865, "meta.caller_id"=>"Cronjob", "correlation_id"=>"246496a1af1fc2c3b577cef4ce8cfda9", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"database", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:database_batched_background_migration_ci_database:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857", "duplicate-of"=>"1a7772559da78c5710c53569", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-4f27c07d50d460c86f7d62df: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:47:05.686Z pid=24 tid=6bb8 class=CiPlatformMetricsUpdateCronWorker jid=8d84b1f44553270d1f93f135 elapsed=0.075 INFO: done
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:47:06Z INFO controllers.GitLab Reconciling GitLab {"gitlab": "gitlab-system/gitlab"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:47:06Z DEBUG controllers.GitLab version information {"gitlab": "gitlab-system/gitlab", "upgrade": true, "current version": "6.11.9", "desired version": "7.1.0"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:47:06Z DEBUG controllers.GitLab createOrUpdate result {"gitlab": "gitlab-system/gitlab", "type": "*v1.Issuer", "reference": "gitlab-system/gitlab-issuer", "result": "updated"}
[gitlab-controller-manager-67dbfbc459-v6bwx manager] 2023-06-25T09:47:06Z INFO controllers.GitLab Core services are not ready. Waiting and retrying {"gitlab": "gitlab-system/gitlab", "interval": "10s"}I also notices some errors from sidekiq from time to time but I do not know if they are relevant:
gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.173Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:schedule_merge_request_cleanup_refs", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"ScheduleMergeRequestCleanupRefsWorker", "jid"=>"e150f6bafffcf2baf1fc5467", "created_at"=>1687686605.1687417, "meta.caller_id"=>"Cronjob", "correlation_id"=>"5ec9d16acee0388aeb907fdac9d25649", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"code_review_workflow", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:schedule_merge_request_cleanup_refs:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811", "duplicate-of"=>"668af1c4e9f747c46c784c7a", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"ScheduleMergeRequestCleanupRefsWorker JID-e150f6bafffcf2baf1fc5467: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.173Z","correlation_id":"8ac211235debe5d30a94b584745008a6","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/gitlab_schemas_metrics.rb:16:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"pages","tags.correlation_id":"8ac211235debe5d30a94b584745008a6","extra.sidekiq":{"retry":false,"queue":"cronjob:pages_domain_ssl_renewal_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"PagesDomainSslRenewalCronWorker","jid":"aae0c3cedb9cdc47b56f1ff6","created_at":1687686605.130864,"meta.caller_id":"Cronjob","correlation_id":"8ac211235debe5d30a94b584745008a6","meta.root_caller_id":"Cronjob","meta.feature_category":"pages","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:pages_domain_ssl_renewal_cron:cee20cd61cae5986aa71b6fa6dd981aaba3c2a4e4980220c0e15411712f7b4e3","size_limiter":"validated","enqueued_at":1687686605.1330335}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.178Z pid=24 tid=6b8g class=ContainerRegistry::CleanupWorker jid=4c3762b28aa763e5f8c5c615 elapsed=0.064 INFO: done
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.181Z","correlation_id":"8ac211235debe5d30a94b584745008a6","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:50:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"pages","tags.correlation_id":"8ac211235debe5d30a94b584745008a6","extra.sidekiq":{"retry":false,"queue":"cronjob:pages_domain_ssl_renewal_cron","version":0,"queue_namespace":"cronjob","args":[],"class":"PagesDomainSslRenewalCronWorker","jid":"aae0c3cedb9cdc47b56f1ff6","created_at":1687686605.130864,"meta.caller_id":"Cronjob","correlation_id":"8ac211235debe5d30a94b584745008a6","meta.root_caller_id":"Cronjob","meta.feature_category":"pages","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:cronjob:pages_domain_ssl_renewal_cron:cee20cd61cae5986aa71b6fa6dd981aaba3c2a4e4980220c0e15411712f7b4e3","size_limiter":"validated","enqueued_at":1687686605.1330335}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.196Z pid=24 tid=6b20 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=1a87ba750fd48c2b7efa1b0d INFO: start
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.197Z pid=24 tid=6b74 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=0e15bcd563a56d86f1da6a55 INFO: start
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.197Z pid=24 tid=6b20 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=1a87ba750fd48c2b7efa1b0d INFO: arguments: []
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.206Z pid=24 tid=6bg0 class=PagesDomainSslRenewalCronWorker jid=aae0c3cedb9cdc47b56f1ff6 elapsed=0.072 INFO: done
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.199Z pid=24 tid=6aug class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=94cbcee1f7f9689506bb476c INFO: start
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.204Z pid=24 tid=6b74 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=0e15bcd563a56d86f1da6a55 INFO: arguments: []
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.210Z pid=24 tid=6aug class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=94cbcee1f7f9689506bb476c INFO: arguments: []
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.204Z pid=24 tid=6918 INFO: {"retry"=>false, "queue"=>"cronjob:database_batched_background_migration_ci_database", "version"=>0, "queue_namespace"=>:cronjob, "args"=>[], "class"=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", "jid"=>"4955548e33457cfa3cff5e62", "created_at"=>1687686605.2006369, "meta.caller_id"=>"Cronjob", "correlation_id"=>"02c8c7ed834825833b9e790ea76769a7", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"database", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:cronjob:database_batched_background_migration_ci_database:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857", "duplicate-of"=>"1a7772559da78c5710c53569", "job_size_bytes"=>2, "pid"=>24, "job_status"=>"deduplicated", "message"=>"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-4955548e33457cfa3cff5e62: deduplicated: until executing", "deduplication.type"=>"until executing"}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.213Z","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/gitlab_schemas_metrics.rb:16:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"container_registry","tags.correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","extra.sidekiq":{"retry":0,"queue":"container_repository:container_expiration_policies_cleanup_container_repository","version":0,"status_expiration":1800,"queue_namespace":"container_repository","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","args":[],"jid":"1a87ba750fd48c2b7efa1b0d","created_at":1687686605.169361,"meta.caller_id":"ContainerExpirationPolicyWorker","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":1687686605.179243}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.217Z pid=24 tid=6bc8 class=ContainerExpirationPolicyWorker jid=061ddc83006f39c79f9dd207 elapsed=0.147 INFO: done
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.199Z pid=24 tid=6bfo class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=988b8845b15800adb3fe54da INFO: start
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.223Z","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/gitlab_schemas_metrics.rb:16:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"container_registry","tags.correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","extra.sidekiq":{"retry":0,"queue":"container_repository:container_expiration_policies_cleanup_container_repository","version":0,"status_expiration":1800,"queue_namespace":"container_repository","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","args":[],"jid":"0e15bcd563a56d86f1da6a55","created_at":1687686605.169361,"meta.caller_id":"ContainerExpirationPolicyWorker","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":1687686605.179243}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.226Z","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:50:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"container_registry","tags.correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","extra.sidekiq":{"retry":0,"queue":"container_repository:container_expiration_policies_cleanup_container_repository","version":0,"status_expiration":1800,"queue_namespace":"container_repository","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","args":[],"jid":"1a87ba750fd48c2b7efa1b0d","created_at":1687686605.169361,"meta.caller_id":"ContainerExpirationPolicyWorker","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":1687686605.179243}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.229Z pid=24 tid=6bfo class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=988b8845b15800adb3fe54da INFO: arguments: []
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] {"component": "gitlab","subcomponent":"exceptions_json","level":"error","severity":"ERROR","time":"2023-06-25T09:50:05.242Z","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","exception.class":"ActiveRecord::ConnectionNotEstablished","exception.message":"connection to server at \"10.43.32.178\", port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\n","exception.backtrace":["lib/gitlab/database/load_balancing/primary_host.rb:44:in `connection'","lib/gitlab/database/load_balancing/load_balancer.rb:54:in `read'","lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `read_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:55:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/query_analyzers/ci/partitioning_routing_analyzer.rb:15:in `enabled?'","lib/gitlab/database/query_analyzer.rb:49:in `block in begin!'","lib/gitlab/database/query_analyzer.rb:46:in `select'","lib/gitlab/database/query_analyzer.rb:46:in `begin!'","lib/gitlab/database/query_analyzer.rb:34:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/arguments_logger.rb:10:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'"],"exception.cause_class":"PG::ConnectionBad","user.username":null,"tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"container_registry","tags.correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","extra.sidekiq":{"retry":0,"queue":"container_repository:container_expiration_policies_cleanup_container_repository","version":0,"status_expiration":1800,"queue_namespace":"container_repository","class":"ContainerExpirationPolicies::CleanupContainerRepositoryWorker","args":[],"jid":"1a87ba750fd48c2b7efa1b0d","created_at":1687686605.169361,"meta.caller_id":"ContainerExpirationPolicyWorker","correlation_id":"a55ad0ca1f30f6c720e83832d7f77bfb","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","worker_data_consistency":"always","size_limiter":"validated","enqueued_at":1687686605.179243}}
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.263Z pid=24 tid=6aug class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=94cbcee1f7f9689506bb476c elapsed=0.064 INFO: done
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.270Z pid=24 tid=6b74 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=0e15bcd563a56d86f1da6a55 elapsed=0.074 INFO: done
[gitlab-sidekiq-all-in-1-v2-f548bf964-kzdtv sidekiq] 2023-06-25T09:50:05.274Z pid=24 tid=6b20 class=ContainerExpirationPolicies::CleanupContainerRepositoryWorker jid=1a87ba750fd48c2b7efa1b0d elapsed=0.078 INFO: done