Skip to content

v16.2.3: can't push/pull to self-hosted instance

Summary

Can't push/pull to GitLab via SSH when OAuth-only authentication is enabled.

Steps to reproduce

  1. Use following docker-compose manifest (don't forget about internal reverse proxy ingress):
version: '3.6'
services:
  gitlab:
    image: 'gitlab/gitlab-ee:16.2.3-ee.0'
    restart: always
    hostname: 'git.company.tld'
    environment:
      GITLAB_OMNIBUS_CONFIG: | 
        external_url 'https://git.company.tld'
        nginx['listen_https'] = false
        nginx['listen_port'] = 8929
        nginx['proxy_set_headers'] = {
            "X-Forwarded-Proto" => "http",
            "X-Forwarded-Port" => "8929"
        }
        gitlab_rails['omniauth_providers'] = [
          {
            name: "google_oauth2",
            label: "Google Workspace",
            app_id: "[REDACTED]",
            app_secret: "[REDACTED]",
            args: { access_type: "offline", approval_prompt: "" }
          }
        ]
        gitlab_rails['gitlab_shell_ssh_port'] = 10022
        gitlab_rails['omniauth_enabled'] = true
        gitlab_rails['omniauth_allow_single_sign_on'] = ['google_oauth2']
        gitlab_rails['omniauth_sync_profile_from_provider'] = ['google_oauth2']
        gitlab_rails['omniauth_block_auto_created_users'] = false
    ports:
      - '8929:8929'
      - '10022:22'
    volumes:
      - './gitlab/config:/etc/gitlab'
      - './gitlab/logs:/var/log/gitlab'
      - './gitlab/data:/var/opt/gitlab'
    shm_size: '256m'
    container_name: gitlab
    networks:
      - cicd
  1. Provision your account, grant admin rights via built-in user and grant group ownership via web interface.
  2. Create empty Git repo, and try to pull it.

Example Project

Empty repo on self-hosted GitLab.

What is the current bug behavior?

$ git clone ssh://git@git.company.tld:10022/test_group/test_repo.git
10:02:52.666942 git.c:462               trace: built-in: git clone ssh://git@git.company.tld:10022/test_group/test_repo.git
Cloning into 'test_repo'...
10:02:52.678187 run-command.c:661       trace: run_command: unset GIT_DIR; GIT_PROTOCOL=version=2 'ssh -vvv' -o SendEnv=GIT_PROTOCOL -p 10022 git@git.company.tld 'git-upload-pack '\''/test_group/test_repo.git'\'''
OpenSSH_9.3p2, OpenSSL 3.1.2 1 Aug 2023
debug1: Reading configuration data /home/ivdok/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/ivdok/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/ivdok/.ssh/known_hosts2'
debug2: resolving "git.company.tld" port 10022
debug3: resolve_host: lookup git.company.tld:10022
debug3: ssh_connect_direct: entering
debug1: Connecting to git.company.tld [172.31.104.169] port 10022.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/ivdok/.ssh/id_rsa type 0
debug1: identity file /home/ivdok/.ssh/id_rsa-cert type -1
debug1: identity file /home/ivdok/.ssh/id_ecdsa type -1
debug1: identity file /home/ivdok/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/ivdok/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/ivdok/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/ivdok/.ssh/id_ed25519 type -1
debug1: identity file /home/ivdok/.ssh/id_ed25519-cert type -1
debug1: identity file /home/ivdok/.ssh/id_ed25519_sk type -1
debug1: identity file /home/ivdok/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/ivdok/.ssh/id_xmss type -1
debug1: identity file /home/ivdok/.ssh/id_xmss-cert type -1
debug1: identity file /home/ivdok/.ssh/id_dsa type -1
debug1: identity file /home/ivdok/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to git.company.tld:10022 as 'git'
debug3: put_host_port: [git.company.tld]:10022
debug3: record_hostkey: found key type ED25519 in file /home/ivdok/.ssh/known_hosts:408
debug3: load_hostkeys_file: loaded 1 keys from [git.company.tld]:10022
debug1: load_hostkeys: fopen /home/ivdok/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:nlUAFae5qJ3aZ5yc9+k9/XTcgSEb7Nq8gTgPgo26ZXA
debug3: put_host_port: [172.31.104.169]:10022
debug3: put_host_port: [git.company.tld]:10022
debug3: record_hostkey: found key type ED25519 in file /home/ivdok/.ssh/known_hosts:408
debug3: load_hostkeys_file: loaded 1 keys from [git.company.tld]:10022
debug1: load_hostkeys: fopen /home/ivdok/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[git.company.tld]:10022' is known and matches the ED25519 host key.
debug1: Found key in /home/ivdok/.ssh/known_hosts:408
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/ivdok/.ssh/id_rsa RSA SHA256:rnYIVyocrhOxVKJnqp0JiPno2I1Z70LsUxVwbO4uXak
debug1: Will attempt key: /home/ivdok/.ssh/id_ecdsa 
debug1: Will attempt key: /home/ivdok/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/ivdok/.ssh/id_ed25519 
debug1: Will attempt key: /home/ivdok/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/ivdok/.ssh/id_xmss 
debug1: Will attempt key: /home/ivdok/.ssh/id_dsa 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/ivdok/.ssh/id_rsa RSA SHA256:rnYIVyocrhOxVKJnqp0JiPno2I1Z70LsUxVwbO4uXak
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/ivdok/.ssh/id_rsa RSA SHA256:rnYIVyocrhOxVKJnqp0JiPno2I1Z70LsUxVwbO4uXak
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:rnYIVyocrhOxVKJnqp0JiPno2I1Z70LsUxVwbO4uXak
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:rnYIVyocrhOxVKJnqp0JiPno2I1Z70LsUxVwbO4uXak
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to git.company.tld ([172.31.104.169]:10022) using "publickey".
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: filesystem
debug3: client_repledge: enter
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received RSA key SHA256:Sa0qS+3x3axejUm+HuBZWvTpu2FfSUkzMqUfIguVWWI
debug3: client_input_hostkeys: received ECDSA key SHA256:59pp+JGi2sdIux5F8XiDavtlF2226KMSXGxyuQ3xOJw
debug3: client_input_hostkeys: received ED25519 key SHA256:nlUAFae5qJ3aZ5yc9+k9/XTcgSEb7Nq8gTgPgo26ZXA
debug3: put_host_port: [git.company.tld]:10022
debug1: client_input_hostkeys: searching /home/ivdok/.ssh/known_hosts for [git.company.tld]:10022 / (none)
debug3: hostkeys_foreach: reading file "/home/ivdok/.ssh/known_hosts"
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key under different name/addr at /home/ivdok/.ssh/known_hosts:1
debug3: hostkeys_find: found ssh-rsa key under different name/addr at /home/ivdok/.ssh/known_hosts:42
debug3: hostkeys_find: found ssh-ed25519 key under different name/addr at /home/ivdok/.ssh/known_hosts:43
debug3: hostkeys_find: found ssh-ed25519 key at /home/ivdok/.ssh/known_hosts:408
debug1: client_input_hostkeys: searching /home/ivdok/.ssh/known_hosts2 for [git.company.tld]:10022 / (none)
debug1: client_input_hostkeys: hostkeys file /home/ivdok/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 2 new, 18446744073709551615 retained, 2 incomplete match. 0 to remove
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug3: client_repledge: enter
debug3: receive packet: type 4
debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:55: key options: command user-rc
debug3: receive packet: type 4
debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:55: key options: command user-rc
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: set_sock_tos: set socket 3 IP_TOS 0x20
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env WINDOWID
debug3: Ignored env COLORTERM
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env LANGUAGE
debug3: Ignored env GIT_TRACE
debug3: Ignored env GIT_SSH_COMMAND
debug3: Ignored env DOTNET_ROOT
debug3: Ignored env MOZ_DRM_DEVICE
debug3: Ignored env SHELL_SESSION_ID
debug3: Ignored env MEMORY_PRESSURE_WRITE
debug3: Ignored env LIBVA_DRIVER_NAME
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env LC_MONETARY
debug3: Ignored env GTK_RC_FILES
debug3: Ignored env XCURSOR_SIZE
debug1: channel 0: setting env GIT_PROTOCOL = "version=2"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env EDITOR
debug3: Ignored env XDG_SEAT
debug3: Ignored env PWD
debug3: Ignored env XDG_SESSION_DESKTOP
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env SYSTEMD_EXEC_PID
debug3: Ignored env _
debug3: Ignored env XAUTHORITY
debug3: Ignored env GIT_CURL_VERBOSE
debug3: Ignored env MOTD_SHOWN
debug3: Ignored env GTK2_RC_FILES
debug3: Ignored env GIT_EXEC_PATH
debug3: Ignored env HOME
debug3: Ignored env LANG
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env KONSOLE_DBUS_SERVICE
debug3: Ignored env MEMORY_PRESSURE_WATCH
debug3: Ignored env WAYLAND_DISPLAY
debug3: Ignored env KONSOLE_DBUS_SESSION
debug3: Ignored env PROFILEHOME
debug3: Ignored env PROFILE_SH_INCLUDED
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env INVOCATION_ID
debug3: Ignored env KONSOLE_VERSION
debug3: Ignored env MANAGERPID
debug3: Ignored env DOTNET_BUNDLE_EXTRACT_BASE_DIR
debug3: Ignored env MOZ_DISABLE_RDD_SANDBOX
debug3: Ignored env KDE_SESSION_UID
debug3: Ignored env EGL_PLATFORM
debug3: Ignored env NVD_BACKEND
debug3: Ignored env XDG_SESSION_CLASS
debug3: Ignored env TERM
debug3: Ignored env USER
debug3: Ignored env COLORFGBG
debug3: Ignored env PLASMA_USE_QT_SCALING
debug3: Ignored env KDE_SESSION_VERSION
debug3: Ignored env PAM_KWALLET5_LOGIN
debug3: Ignored env QT_WAYLAND_FORCE_DPI
debug3: Ignored env DISPLAY
debug3: Ignored env SHLVL
debug3: Ignored env MOZ_ENABLE_WAYLAND
debug3: Ignored env LC_MEASUREMENT
debug3: Ignored env XDG_VTNR
debug3: Ignored env UBUNTU_MENUPROXY
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env MOZ_PLUGIN_PATH
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env DEBUGINFOD_URLS
debug3: Ignored env LC_TIME
debug3: Ignored env QT_AUTO_SCREEN_SCALE_FACTOR
debug3: Ignored env JOURNAL_STREAM
debug3: Ignored env XCURSOR_THEME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env KDE_FULL_SESSION
debug3: Ignored env NLSPATH
debug3: Ignored env PATH
debug3: Ignored env GTK_USE_PORTAL
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env KDE_APPLICATIONS_AS_SCOPE
debug3: Ignored env MAIL
debug3: Ignored env _JAVA_OPTIONS
debug3: Ignored env OLDPWD
debug1: Sending command: git-upload-pack '/test_group/test_repo.git'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug3: client_repledge: enter
debug1: pledge: fork
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 299
remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found or you don't have permission to view it.
remote: 
remote: ========================================================================
remote: 
debug2: channel 0: written 299 to efd 6
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: chan_shutdown_read: channel 0: (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: chan_shutdown_write: channel 0: (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1 io 0x00/0x00)

debug3: send packet: type 1
Transferred: sent 3280, received 3160 bytes, in 0.6 seconds
Bytes per second: sent 5474.5, received 5274.2
debug1: Exit status 1

What is the expected correct behavior?

Empty git repo should be clonable. Locally created repo should be force-pushable (but exhibits the same behaviour).

Relevant logs and/or screenshots

Largely irrelevant:

==> /var/log/gitlab/postgres-exporter/current <==
2023-08-08_07:11:48.51648 ts=2023-08-08T07:11:48.516Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.182715542 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-08-04 09:10:37.523279 +0000 GMT\") to a float64: invalid syntax"
2023-08-08_07:11:48.59760 ts=2023-08-08T07:11:48.597Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.263769098 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-08-04 09:10:37.523279 +0000 GMT\") to a float64: invalid syntax"

==> /var/log/gitlab/gitlab-exporter/current <==
2023-08-08_07:12:00.33711 127.0.0.1 - - [08/Aug/2023:07:12:00 UTC] "GET /database HTTP/1.1" 200 1952
2023-08-08_07:12:00.33722 - -> /database
2023-08-08_07:12:00.65825 127.0.0.1 - - [08/Aug/2023:07:12:00 UTC] "GET /sidekiq HTTP/1.1" 200 135828
2023-08-08_07:12:00.65835 - -> /sidekiq
2023-08-08_07:12:00.70095 127.0.0.1 - - [08/Aug/2023:07:12:00 UTC] "GET /ruby HTTP/1.1" 200 1084
2023-08-08_07:12:00.70114 - -> /ruby

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2023-08-08T07:12:01.390Z","params":[],"redis_calls":5,"redis_duration_s":0.002588,"redis_read_bytes":995,"redis_write_bytes":387,"redis_feature_flag_calls":5,"redis_feature_flag_duration_s":0.002588,"redis_feature_flag_read_bytes":995,"redis_feature_flag_write_bytes":387,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.119026,"mem_objects":4104,"mem_bytes":2803688,"mem_mallocs":4107,"mem_total_bytes":2967848,"pid":677,"worker_id":"puma_3","rate_limiting_gates":[],"correlation_id":"e850d78c-03ad-42fe-806a-b0d1e5ee4100","db_duration_s":0.0,"view_duration_s":0.00081,"duration_s":0.10959}

==> /var/log/gitlab/postgres-exporter/current <==
2023-08-08_07:12:03.40447 ts=2023-08-08T07:12:03.404Z caller=collector.go:190 level=error msg="collector failed" name=replication duration_seconds=0.071294288 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-08-04 09:10:37.523279 +0000 GMT\") to a float64: invalid syntax"
2023-08-08_07:12:03.49847 ts=2023-08-08T07:12:03.498Z caller=collector.go:190 level=error msg="collector failed" name=postmaster duration_seconds=0.165326947 err="sql: Scan error on column index 0, name \"pg_postmaster_start_time\": converting driver.Value type time.Time (\"2023-08-04 09:10:37.523279 +0000 GMT\") to a float64: invalid syntax"

==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2023-08-08T07:12:06.817Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"UserStatusCleanup::BatchWorker","jid":"263aa800ea8f1cc906470af0","created_at":"2023-08-08T07:12:06.812Z","meta.caller_id":"Cronjob","correlation_id":"19554b5e393f35e36db0df7d4b35827f","meta.root_caller_id":"Cronjob","meta.feature_category":"user_profile","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.813Z","job_size_bytes":2,"pid":469,"message":"UserStatusCleanup::BatchWorker JID-263aa800ea8f1cc906470af0: start","job_status":"start","scheduling_latency_s":0.003874}
{"severity":"INFO","time":"2023-08-08T07:12:06.822Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ObjectStorage::DeleteStaleDirectUploadsWorker","jid":"324508c934835650b13d4ab9","created_at":"2023-08-08T07:12:06.739Z","meta.caller_id":"Cronjob","correlation_id":"5420ac9df5ddc9c6813f1cc13edcdfad","meta.root_caller_id":"Cronjob","meta.feature_category":"build_artifacts","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:da92be7816ff691bea6b0627c50cb73b98e4b5009a988cab37d277c0df9aa326","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.796Z","job_size_bytes":2,"pid":469,"message":"ObjectStorage::DeleteStaleDirectUploadsWorker JID-324508c934835650b13d4ab9: start","job_status":"start","scheduling_latency_s":0.024851}
{"severity":"INFO","time":"2023-08-08T07:12:06.911Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ObjectStorage::DeleteStaleDirectUploadsWorker","jid":"324508c934835650b13d4ab9","created_at":"2023-08-08T07:12:06.739Z","meta.caller_id":"Cronjob","correlation_id":"5420ac9df5ddc9c6813f1cc13edcdfad","meta.root_caller_id":"Cronjob","meta.feature_category":"build_artifacts","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:da92be7816ff691bea6b0627c50cb73b98e4b5009a988cab37d277c0df9aa326","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.796Z","job_size_bytes":2,"pid":469,"message":"ObjectStorage::DeleteStaleDirectUploadsWorker JID-324508c934835650b13d4ab9: done: 0.090941 sec","job_status":"done","scheduling_latency_s":0.024851,"redis_calls":12,"redis_duration_s":0.014707999999999999,"redis_read_bytes":1514,"redis_write_bytes":876,"redis_feature_flag_calls":8,"redis_feature_flag_duration_s":0.012155,"redis_feature_flag_read_bytes":1510,"redis_feature_flag_write_bytes":636,"redis_queues_calls":2,"redis_queues_duration_s":0.001709,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000844,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":54,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.020552,"mem_objects":3435,"mem_bytes":398088,"mem_mallocs":994,"mem_total_bytes":535488,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.object_storage_delete_stale_direct_uploads_worker.total_pending_entries":0,"extra.object_storage_delete_stale_direct_uploads_worker.total_deleted_stale_entries":0,"extra.object_storage_delete_stale_direct_uploads_worker.execution_timeout":false,"duration_s":0.090941,"completed_at":"2023-08-08T07:12:06.911Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.0,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:06.913Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"LooseForeignKeys::CleanupWorker","jid":"15859aa16672736f2e204d37","created_at":"2023-08-08T07:12:06.877Z","meta.caller_id":"Cronjob","correlation_id":"264c5d4ac7f3f9f8c61b4547b9a298af","meta.root_caller_id":"Cronjob","meta.feature_category":"cell","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:2180384c8554b410820afa81ad3479301139c7627524f882c6bca44488742df9","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.883Z","job_size_bytes":2,"pid":469,"message":"LooseForeignKeys::CleanupWorker JID-15859aa16672736f2e204d37: start","job_status":"start","scheduling_latency_s":0.030523}
{"severity":"INFO","time":"2023-08-08T07:12:06.941Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"UserStatusCleanup::BatchWorker","jid":"263aa800ea8f1cc906470af0","created_at":"2023-08-08T07:12:06.812Z","meta.caller_id":"Cronjob","correlation_id":"19554b5e393f35e36db0df7d4b35827f","meta.root_caller_id":"Cronjob","meta.feature_category":"user_profile","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:63596f84f2aa01599e5d284b58c5b497d0d152817e7aa09bae73984558ee32bf","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.813Z","job_size_bytes":2,"pid":469,"message":"UserStatusCleanup::BatchWorker JID-263aa800ea8f1cc906470af0: done: 0.123837 sec","job_status":"done","scheduling_latency_s":0.003874,"redis_calls":6,"redis_duration_s":0.032433000000000003,"redis_read_bytes":798,"redis_write_bytes":538,"redis_feature_flag_calls":4,"redis_feature_flag_duration_s":0.031704,"redis_feature_flag_read_bytes":796,"redis_feature_flag_write_bytes":352,"redis_queues_calls":2,"redis_queues_duration_s":0.000729,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.015,"db_main_duration_s":0.015,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.014046,"mem_objects":2751,"mem_bytes":273168,"mem_mallocs":867,"mem_total_bytes":383208,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.123837,"completed_at":"2023-08-08T07:12:06.940Z","load_balancing_strategy":"primary","db_duration_s":0.014928,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.029Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"UpdateAllMirrorsWorker","jid":"9db8fcf005a6b6e1dbb7f7f2","created_at":"2023-08-08T07:12:07.020Z","meta.caller_id":"Cronjob","correlation_id":"0642aa2f0e46b9577cef7f466b9da2bf","meta.root_caller_id":"Cronjob","meta.feature_category":"source_code_management","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:06f0ae3fb264ecea0081f8b5bcdfdf63d385e3db07629cf8d60f5d6f216854cc","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.022Z","job_size_bytes":2,"pid":469,"message":"UpdateAllMirrorsWorker JID-9db8fcf005a6b6e1dbb7f7f2: start","job_status":"start","scheduling_latency_s":0.007101}
{"severity":"INFO","time":"2023-08-08T07:12:07.030Z","retry":0,"queue":"default","version":1,"queue_namespace":"cronjob","args":[],"class":"Database::CiNamespaceMirrorsConsistencyCheckWorker","jid":"e70ccd91870f8ab3d692d698","created_at":"2023-08-08T07:12:06.982Z","meta.caller_id":"Cronjob","correlation_id":"df2937474e87082afc1abb50e2a695fb","meta.root_caller_id":"Cronjob","meta.feature_category":"cell","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:a4d60c973c162c31ab7807150de2d79cb56e2b9c2a13ce2c456ceab058031670","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.986Z","job_size_bytes":2,"pid":469,"message":"Database::CiNamespaceMirrorsConsistencyCheckWorker JID-e70ccd91870f8ab3d692d698: start","job_status":"start","scheduling_latency_s":0.043887}
{"severity":"INFO","time":"2023-08-08T07:12:07.033Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Users::MigrateRecordsToGhostUserInBatchesWorker","jid":"017888a9b51b2723e8148e86","created_at":"2023-08-08T07:12:06.999Z","meta.caller_id":"Cronjob","correlation_id":"3aca805992280897a181e85f4ba93b32","meta.root_caller_id":"Cronjob","meta.feature_category":"user_profile","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:4bdb3193c92ce2ad4e73652ad55816e507f3dd7f07576150fc7c08572353b9e8","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.000Z","job_size_bytes":2,"pid":469,"message":"Users::MigrateRecordsToGhostUserInBatchesWorker JID-017888a9b51b2723e8148e86: start","job_status":"start","scheduling_latency_s":0.033162}
{"severity":"INFO","time":"2023-08-08T07:12:07.089Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::IncidentSlaExceededCheckWorker","jid":"33036c92bb969b6f18abd963","created_at":"2023-08-08T07:12:07.052Z","meta.caller_id":"Cronjob","correlation_id":"de24796612567520e19a59eec8dc79f9","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.055Z","job_size_bytes":2,"pid":469,"message":"IncidentManagement::IncidentSlaExceededCheckWorker JID-33036c92bb969b6f18abd963: start","job_status":"start","scheduling_latency_s":0.033898}
{"severity":"INFO","time":"2023-08-08T07:12:07.283Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","jid":"c52481c27504d11508750f52","created_at":"2023-08-08T07:12:07.086Z","meta.caller_id":"Cronjob","correlation_id":"29cb9b780738c4c620456cc99d5a5368","meta.root_caller_id":"Cronjob","meta.feature_category":"code_review_workflow","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.100Z","job_size_bytes":2,"pid":469,"message":"ScheduleMergeRequestCleanupRefsWorker JID-c52481c27504d11508750f52: start","job_status":"start","scheduling_latency_s":0.182774}
{"severity":"INFO","time":"2023-08-08T07:12:07.285Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker","jid":"dae154c3a95051bc6524d5e1","created_at":"2023-08-08T07:12:07.166Z","meta.caller_id":"Cronjob","correlation_id":"ac334be1e1425956ee5fefcc01204381","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:979447f3f3db1dd62a2679e7ff5835fcbb6ccf42394193a5feb300b5ecfe13e6","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.168Z","job_size_bytes":2,"pid":469,"message":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker JID-dae154c3a95051bc6524d5e1: start","job_status":"start","scheduling_latency_s":0.117142}
{"severity":"INFO","time":"2023-08-08T07:12:07.287Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","jid":"7a839becc42ab95122ba6687","created_at":"2023-08-08T07:12:07.137Z","meta.caller_id":"Cronjob","correlation_id":"de0d868e5abf30c4f3ebc3147242ccc6","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857","enqueued_at":"2023-08-08T07:12:07.139Z","job_size_bytes":2,"pid":469,"message":"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-7a839becc42ab95122ba6687: start","job_status":"start","scheduling_latency_s":0.148224}
{"severity":"INFO","time":"2023-08-08T07:12:07.289Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"011a555397eb3fe07ce48bdd","created_at":"2023-08-08T07:12:07.126Z","meta.caller_id":"Cronjob","correlation_id":"dbc5b6c7fa7b0c907588f6ac812aa133","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.128Z","job_size_bytes":2,"pid":469,"message":"Geo::SidekiqCronConfigWorker JID-011a555397eb3fe07ce48bdd: start","job_status":"start","scheduling_latency_s":0.161169}
{"severity":"INFO","time":"2023-08-08T07:12:07.327Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ScheduleMergeRequestCleanupRefsWorker","jid":"c52481c27504d11508750f52","created_at":"2023-08-08T07:12:07.086Z","meta.caller_id":"Cronjob","correlation_id":"29cb9b780738c4c620456cc99d5a5368","meta.root_caller_id":"Cronjob","meta.feature_category":"code_review_workflow","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:33e8a9dcd4c9780ad0ea123ad7ccbabde1aa1e90ffcbb928434ba4b5800a5811","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.100Z","job_size_bytes":2,"pid":469,"message":"ScheduleMergeRequestCleanupRefsWorker JID-c52481c27504d11508750f52: done: 0.045 sec","job_status":"done","scheduling_latency_s":0.182774,"redis_calls":4,"redis_duration_s":0.001311,"redis_read_bytes":397,"redis_write_bytes":337,"redis_feature_flag_calls":2,"redis_feature_flag_duration_s":0.000824,"redis_feature_flag_read_bytes":395,"redis_feature_flag_write_bytes":151,"redis_queues_calls":2,"redis_queues_duration_s":0.000487,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.013948,"mem_objects":4057,"mem_bytes":442376,"mem_mallocs":1528,"mem_total_bytes":604656,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.045,"completed_at":"2023-08-08T07:12:07.327Z","load_balancing_strategy":"primary","db_duration_s":0.0,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.389Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Search::IndexCurationWorker","jid":"d11fbb4dbc9fd5da493c1bfb","created_at":"2023-08-08T07:12:07.290Z","meta.caller_id":"Cronjob","correlation_id":"30db98beb236f3dceee08d73d2ce89ea","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f1b35de73fa09aee87da69fc70f7f498516e5b000ec47c894d20a533c302983e","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.333Z","job_size_bytes":2,"pid":469,"message":"Search::IndexCurationWorker JID-d11fbb4dbc9fd5da493c1bfb: start","job_status":"start","scheduling_latency_s":0.056752}
{"severity":"INFO","time":"2023-08-08T07:12:07.414Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Users::MigrateRecordsToGhostUserInBatchesWorker","jid":"017888a9b51b2723e8148e86","created_at":"2023-08-08T07:12:06.999Z","meta.caller_id":"Cronjob","correlation_id":"3aca805992280897a181e85f4ba93b32","meta.root_caller_id":"Cronjob","meta.feature_category":"user_profile","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:4bdb3193c92ce2ad4e73652ad55816e507f3dd7f07576150fc7c08572353b9e8","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.000Z","job_size_bytes":2,"pid":469,"message":"Users::MigrateRecordsToGhostUserInBatchesWorker JID-017888a9b51b2723e8148e86: done: 0.380691 sec","job_status":"done","scheduling_latency_s":0.033162,"redis_calls":5,"redis_duration_s":0.196222,"redis_read_bytes":203,"redis_write_bytes":847,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000347,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":100,"redis_queues_calls":2,"redis_queues_duration_s":0.106229,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.089646,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":561,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.116,"db_main_duration_s":0.116,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.011044,"mem_objects":2433,"mem_bytes":236448,"mem_mallocs":735,"mem_total_bytes":333768,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.380691,"completed_at":"2023-08-08T07:12:07.414Z","load_balancing_strategy":"primary","db_duration_s":0.116191,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.396Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::IncidentSlaExceededCheckWorker","jid":"33036c92bb969b6f18abd963","created_at":"2023-08-08T07:12:07.052Z","meta.caller_id":"Cronjob","correlation_id":"de24796612567520e19a59eec8dc79f9","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:d27ff4fa309181f602a07a33de23313c3104da78503f956ce0f32a9b265b6d48","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.055Z","job_size_bytes":2,"pid":469,"message":"IncidentManagement::IncidentSlaExceededCheckWorker JID-33036c92bb969b6f18abd963: done: 0.30779 sec","job_status":"done","scheduling_latency_s":0.033898,"redis_calls":3,"redis_duration_s":0.013732,"redis_read_bytes":201,"redis_write_bytes":289,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000213,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":103,"redis_queues_calls":2,"redis_queues_duration_s":0.013519,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.042,"db_main_duration_s":0.042,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.011085,"mem_objects":2420,"mem_bytes":145952,"mem_mallocs":689,"mem_total_bytes":242752,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.30779,"completed_at":"2023-08-08T07:12:07.396Z","load_balancing_strategy":"primary","db_duration_s":0.041727,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.548Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Search::IndexCurationWorker","jid":"d11fbb4dbc9fd5da493c1bfb","created_at":"2023-08-08T07:12:07.290Z","meta.caller_id":"Cronjob","correlation_id":"30db98beb236f3dceee08d73d2ce89ea","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:f1b35de73fa09aee87da69fc70f7f498516e5b000ec47c894d20a533c302983e","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.333Z","job_size_bytes":2,"pid":469,"message":"Search::IndexCurationWorker JID-d11fbb4dbc9fd5da493c1bfb: done: 0.158409 sec","job_status":"done","scheduling_latency_s":0.056752,"redis_calls":4,"redis_duration_s":0.001292,"redis_read_bytes":400,"redis_write_bytes":322,"redis_feature_flag_calls":2,"redis_feature_flag_duration_s":0.000637,"redis_feature_flag_read_bytes":398,"redis_feature_flag_write_bytes":136,"redis_queues_calls":2,"redis_queues_duration_s":0.000655,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.007078,"mem_objects":2106,"mem_bytes":138400,"mem_mallocs":564,"mem_total_bytes":222640,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.158409,"completed_at":"2023-08-08T07:12:07.548Z","load_balancing_strategy":"primary","db_duration_s":0.0,"urgency":"throttled","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.736Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker","jid":"dae154c3a95051bc6524d5e1","created_at":"2023-08-08T07:12:07.166Z","meta.caller_id":"Cronjob","correlation_id":"ac334be1e1425956ee5fefcc01204381","meta.root_caller_id":"Cronjob","meta.feature_category":"incident_management","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:979447f3f3db1dd62a2679e7ff5835fcbb6ccf42394193a5feb300b5ecfe13e6","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.168Z","job_size_bytes":2,"pid":469,"message":"IncidentManagement::PendingEscalations::ScheduleCheckCronWorker JID-dae154c3a95051bc6524d5e1: done: 0.451209 sec","job_status":"done","scheduling_latency_s":0.117142,"redis_calls":3,"redis_duration_s":0.183261,"redis_read_bytes":201,"redis_write_bytes":302,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000235,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":116,"redis_queues_calls":2,"redis_queues_duration_s":0.183026,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.163,"db_main_duration_s":0.163,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.01417,"mem_objects":2836,"mem_bytes":218344,"mem_mallocs":1239,"mem_total_bytes":331784,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.451209,"completed_at":"2023-08-08T07:12:07.736Z","load_balancing_strategy":"primary","db_duration_s":0.162748,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.795Z","retry":0,"queue":"default","version":1,"queue_namespace":"cronjob","args":[],"class":"Database::CiNamespaceMirrorsConsistencyCheckWorker","jid":"e70ccd91870f8ab3d692d698","created_at":"2023-08-08T07:12:06.982Z","meta.caller_id":"Cronjob","correlation_id":"df2937474e87082afc1abb50e2a695fb","meta.root_caller_id":"Cronjob","meta.feature_category":"cell","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:a4d60c973c162c31ab7807150de2d79cb56e2b9c2a13ce2c456ceab058031670","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.986Z","job_size_bytes":2,"pid":469,"message":"Database::CiNamespaceMirrorsConsistencyCheckWorker JID-e70ccd91870f8ab3d692d698: done: 0.765123 sec","job_status":"done","scheduling_latency_s":0.043887,"redis_calls":5,"redis_duration_s":0.130919,"redis_read_bytes":204,"redis_write_bytes":475,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.006136,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":103,"redis_queues_calls":2,"redis_queues_duration_s":0.003728,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.121055,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":186,"db_count":5,"db_write_count":0,"db_cached_count":1,"db_replica_count":0,"db_primary_count":5,"db_main_count":4,"db_ci_count":1,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.505,"db_main_duration_s":0.233,"db_ci_duration_s":0.273,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.021404,"mem_objects":3437,"mem_bytes":285952,"mem_mallocs":669,"mem_total_bytes":423432,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.database_ci_namespace_mirrors_consistency_check_worker.results":{"next_start_id":1,"matches":10,"mismatches":0,"batches":1,"mismatches_details":[],"start_id":1},"duration_s":0.765123,"completed_at":"2023-08-08T07:12:07.795Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.5049,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.896Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Geo::SidekiqCronConfigWorker","jid":"011a555397eb3fe07ce48bdd","created_at":"2023-08-08T07:12:07.126Z","meta.caller_id":"Cronjob","correlation_id":"dbc5b6c7fa7b0c907588f6ac812aa133","meta.root_caller_id":"Cronjob","meta.feature_category":"geo_replication","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:ee3c9c9863d06b67487cd0dcfc522a28b6dc54e34bf5abd078467931033818a0","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.128Z","job_size_bytes":2,"pid":469,"message":"Geo::SidekiqCronConfigWorker JID-011a555397eb3fe07ce48bdd: done: 0.60618 sec","job_status":"done","scheduling_latency_s":0.161169,"redis_calls":166,"redis_duration_s":0.42738699999999996,"redis_read_bytes":61020,"redis_write_bytes":11066,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000397,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":81,"redis_queues_calls":165,"redis_queues_duration_s":0.42699,"redis_queues_read_bytes":60821,"redis_queues_write_bytes":10985,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.241749,"mem_objects":89147,"mem_bytes":86374816,"mem_mallocs":26975,"mem_total_bytes":89940696,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.60618,"completed_at":"2023-08-08T07:12:07.895Z","load_balancing_strategy":"primary","db_duration_s":0.0,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.391Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigrationWorker","jid":"5abf1f08fc2e2087ac725cae","created_at":"2023-08-08T07:12:07.348Z","meta.caller_id":"Cronjob","correlation_id":"786d3e6d96523b612e604f0d351e2c6d","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","enqueued_at":"2023-08-08T07:12:07.350Z","job_size_bytes":2,"pid":469,"message":"Database::BatchedBackgroundMigrationWorker JID-5abf1f08fc2e2087ac725cae: start","job_status":"start","scheduling_latency_s":0.040926}
{"severity":"INFO","time":"2023-08-08T07:12:07.811Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"ba12f0f7b7f81d95fc6473a3","created_at":"2023-08-08T07:12:07.505Z","meta.caller_id":"Cronjob","correlation_id":"07c92ae1c55a3e396a7ff6a42c0e8506","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.702Z","job_size_bytes":2,"pid":469,"message":"ElasticIndexBulkCronWorker JID-ba12f0f7b7f81d95fc6473a3: start","job_status":"start","scheduling_latency_s":0.108731}
{"severity":"INFO","time":"2023-08-08T07:12:07.883Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Metrics::GlobalMetricsUpdateWorker","jid":"8a306d9a7a2c1964c4618a6e","created_at":"2023-08-08T07:12:07.717Z","meta.caller_id":"Cronjob","correlation_id":"06aeda75d65a1b0b0000def2638af2d0","meta.root_caller_id":"Cronjob","meta.feature_category":"metrics","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:0ef60401f7cd15675ace9e5ac4456efd82434f42e048584516e1e7a85b24e5ba","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.718Z","job_size_bytes":2,"pid":469,"message":"Metrics::GlobalMetricsUpdateWorker JID-8a306d9a7a2c1964c4618a6e: start","job_status":"start","scheduling_latency_s":0.165163}
{"severity":"INFO","time":"2023-08-08T07:12:07.402Z","class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","database":"ci","message":"skipping migration execution for database that shares database configuration with another database","retry":0}
{"severity":"INFO","time":"2023-08-08T07:12:08.019Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigration::CiDatabaseWorker","jid":"7a839becc42ab95122ba6687","created_at":"2023-08-08T07:12:07.137Z","meta.caller_id":"Cronjob","correlation_id":"de0d868e5abf30c4f3ebc3147242ccc6","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:6ba8adee4a8c1e77d2f087a2765c43226ceffa1fd65abc34b95725a7c9abd857","enqueued_at":"2023-08-08T07:12:07.139Z","job_size_bytes":2,"pid":469,"message":"Database::BatchedBackgroundMigration::CiDatabaseWorker JID-7a839becc42ab95122ba6687: done: 0.731683 sec","job_status":"done","scheduling_latency_s":0.148224,"redis_calls":3,"redis_duration_s":0.030563,"redis_read_bytes":201,"redis_write_bytes":293,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.017094,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":107,"redis_queues_calls":2,"redis_queues_duration_s":0.013469,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.009242,"mem_objects":1847,"mem_bytes":167720,"mem_mallocs":465,"mem_total_bytes":241600,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.731683,"completed_at":"2023-08-08T07:12:08.019Z","load_balancing_strategy":"primary","db_duration_s":0.0,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:07.991Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker","jid":"653713c395f1693489a421da","created_at":"2023-08-08T07:12:07.842Z","meta.caller_id":"Cronjob","correlation_id":"fc55c9ceea81848dd1194426e5d28a68","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:ba175455ba4249d8c954504ae77a1c3684a61c72005cc3a3032e14c625ef4d8a","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.843Z","job_size_bytes":2,"pid":469,"message":"ElasticIndexInitialBulkCronWorker JID-653713c395f1693489a421da: start","job_status":"start","scheduling_latency_s":0.148331}
{"severity":"INFO","time":"2023-08-08T07:12:07.725Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"UpdateAllMirrorsWorker","jid":"9db8fcf005a6b6e1dbb7f7f2","created_at":"2023-08-08T07:12:07.020Z","meta.caller_id":"Cronjob","correlation_id":"0642aa2f0e46b9577cef7f466b9da2bf","meta.root_caller_id":"Cronjob","meta.feature_category":"source_code_management","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:06f0ae3fb264ecea0081f8b5bcdfdf63d385e3db07629cf8d60f5d6f216854cc","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.022Z","job_size_bytes":2,"pid":469,"message":"UpdateAllMirrorsWorker JID-9db8fcf005a6b6e1dbb7f7f2: done: 0.696412 sec","job_status":"done","scheduling_latency_s":0.007101,"redis_calls":10,"redis_duration_s":0.33198099999999997,"redis_read_bytes":716,"redis_write_bytes":920,"redis_cache_calls":1,"redis_cache_duration_s":0.010424,"redis_cache_read_bytes":114,"redis_cache_write_bytes":96,"redis_feature_flag_calls":3,"redis_feature_flag_duration_s":0.000568,"redis_feature_flag_read_bytes":596,"redis_feature_flag_write_bytes":225,"redis_queues_calls":2,"redis_queues_duration_s":0.213395,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":4,"redis_shared_state_duration_s":0.107594,"redis_shared_state_read_bytes":4,"redis_shared_state_write_bytes":413,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.055,"db_main_duration_s":0.055,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.027626,"mem_objects":6278,"mem_bytes":780008,"mem_mallocs":2665,"mem_total_bytes":1031128,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.696412,"completed_at":"2023-08-08T07:12:07.725Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.055208,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:08.096Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Metrics::GlobalMetricsUpdateWorker","jid":"8a306d9a7a2c1964c4618a6e","created_at":"2023-08-08T07:12:07.717Z","meta.caller_id":"Cronjob","correlation_id":"06aeda75d65a1b0b0000def2638af2d0","meta.root_caller_id":"Cronjob","meta.feature_category":"metrics","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:0ef60401f7cd15675ace9e5ac4456efd82434f42e048584516e1e7a85b24e5ba","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.718Z","job_size_bytes":2,"pid":469,"message":"Metrics::GlobalMetricsUpdateWorker JID-8a306d9a7a2c1964c4618a6e: done: 0.212473 sec","job_status":"done","scheduling_latency_s":0.165163,"redis_calls":5,"redis_duration_s":0.040001,"redis_read_bytes":203,"redis_write_bytes":581,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.010798,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":87,"redis_queues_calls":2,"redis_queues_duration_s":0.028527,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000676,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":308,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_main_count":0,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"db_main_duration_s":0.0,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.013273,"mem_objects":3934,"mem_bytes":493272,"mem_mallocs":1456,"mem_total_bytes":650632,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.212473,"completed_at":"2023-08-08T07:12:08.096Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.0,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:08.079Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexBulkCronWorker","jid":"ba12f0f7b7f81d95fc6473a3","created_at":"2023-08-08T07:12:07.505Z","meta.caller_id":"Cronjob","correlation_id":"07c92ae1c55a3e396a7ff6a42c0e8506","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:f252f68f3cc1cae1877f9a0e1f5b889102a68a9e335a3a5a9e683c7bdf0507f5","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.702Z","job_size_bytes":2,"pid":469,"message":"ElasticIndexBulkCronWorker JID-ba12f0f7b7f81d95fc6473a3: done: 0.268163 sec","job_status":"done","scheduling_latency_s":0.108731,"redis_calls":3,"redis_duration_s":0.020904,"redis_read_bytes":201,"redis_write_bytes":265,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.01057,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":79,"redis_queues_calls":2,"redis_queues_duration_s":0.010334,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.016,"db_main_duration_s":0.016,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.019203,"mem_objects":4233,"mem_bytes":488768,"mem_mallocs":1684,"mem_total_bytes":658088,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.268163,"completed_at":"2023-08-08T07:12:08.079Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.016059,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:08.132Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Database::BatchedBackgroundMigrationWorker","jid":"5abf1f08fc2e2087ac725cae","created_at":"2023-08-08T07:12:07.348Z","meta.caller_id":"Cronjob","correlation_id":"786d3e6d96523b612e604f0d351e2c6d","meta.root_caller_id":"Cronjob","meta.feature_category":"database","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae","enqueued_at":"2023-08-08T07:12:07.350Z","job_size_bytes":2,"pid":469,"message":"Database::BatchedBackgroundMigrationWorker JID-5abf1f08fc2e2087ac725cae: done: 0.741242 sec","job_status":"done","scheduling_latency_s":0.040926,"redis_calls":3,"redis_duration_s":0.028538,"redis_read_bytes":201,"redis_write_bytes":281,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.020725,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":95,"redis_queues_calls":2,"redis_queues_duration_s":0.007813,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.053,"db_main_duration_s":0.053,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.021902,"mem_objects":4553,"mem_bytes":510448,"mem_mallocs":1972,"mem_total_bytes":692568,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.741242,"completed_at":"2023-08-08T07:12:08.132Z","load_balancing_strategy":"primary","db_duration_s":0.052733,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:08.154Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"ElasticIndexInitialBulkCronWorker","jid":"653713c395f1693489a421da","created_at":"2023-08-08T07:12:07.842Z","meta.caller_id":"Cronjob","correlation_id":"fc55c9ceea81848dd1194426e5d28a68","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","worker_data_consistency":"sticky","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:default:ba175455ba4249d8c954504ae77a1c3684a61c72005cc3a3032e14c625ef4d8a","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:07.843Z","job_size_bytes":2,"pid":469,"message":"ElasticIndexInitialBulkCronWorker JID-653713c395f1693489a421da: done: 0.162541 sec","job_status":"done","scheduling_latency_s":0.148331,"redis_calls":3,"redis_duration_s":0.023535,"redis_read_bytes":199,"redis_write_bytes":272,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.00061,"redis_feature_flag_read_bytes":197,"redis_feature_flag_write_bytes":86,"redis_queues_calls":2,"redis_queues_duration_s":0.022925,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.032,"db_main_duration_s":0.032,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.017277,"mem_objects":4185,"mem_bytes":488816,"mem_mallocs":1684,"mem_total_bytes":656216,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.162541,"completed_at":"2023-08-08T07:12:08.154Z","load_balancing_strategy":"primary_no_wal","db_duration_s":0.031448,"urgency":"low","target_duration_s":300}
{"severity":"INFO","time":"2023-08-08T07:12:08.158Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"LooseForeignKeys::CleanupWorker","jid":"15859aa16672736f2e204d37","created_at":"2023-08-08T07:12:06.877Z","meta.caller_id":"Cronjob","correlation_id":"264c5d4ac7f3f9f8c61b4547b9a298af","meta.root_caller_id":"Cronjob","meta.feature_category":"cell","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:2180384c8554b410820afa81ad3479301139c7627524f882c6bca44488742df9","size_limiter":"validated","enqueued_at":"2023-08-08T07:12:06.883Z","job_size_bytes":2,"pid":469,"message":"LooseForeignKeys::CleanupWorker JID-15859aa16672736f2e204d37: done: 1.245208 sec","job_status":"done","scheduling_latency_s":0.030523,"redis_calls":5,"redis_duration_s":0.014929,"redis_read_bytes":203,"redis_write_bytes":571,"redis_feature_flag_calls":1,"redis_feature_flag_duration_s":0.000222,"redis_feature_flag_read_bytes":199,"redis_feature_flag_write_bytes":84,"redis_queues_calls":2,"redis_queues_duration_s":0.005006,"redis_queues_read_bytes":2,"redis_queues_write_bytes":186,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.009701,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":301,"db_count":13,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":13,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.385,"db_main_duration_s":0.385,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.056511,"mem_objects":9438,"mem_bytes":580056,"mem_mallocs":5603,"mem_total_bytes":957576,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.loose_foreign_keys_cleanup_worker.stats":{"over_limit":false,"delete_count_by_table":{},"update_count_by_table":{},"delete_count":0,"update_count":0,"connection":"main"},"duration_s":1.245208,"completed_at":"2023-08-08T07:12:08.158Z","load_balancing_strategy":"primary","db_duration_s":1.15604,"urgency":"low","target_duration_s":300}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-08-08T07:12:08Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-08-08T07:12:08Z"}

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.4.14
Bundler Version:2.4.16
Rake Version:   13.0.6
Redis Version:  7.0.12
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        16.2.3-ee
Revision:       7ac2b0a343c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.11
URL:            https://git.company.tld
HTTP Clone URL: https://git.company.tld/some-group/some-project.git
SSH Clone URL:  ssh://git@git.company.tld:10022/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        14.23.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
35/1 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 5
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by ivdok