redis issue with gitlab
after running gitlab omnibus in a container:
podman run --name gitlab \
-p 0.0.0.0:8082:80 \
-p 0.0.0.0:2222:22 \
--shm-size 10gb \
--restart always \
docker.io/gitlab/gitlab-ee:latesti check the logs:
==> /var/log/gitlab/gitlab-workhorse/current <==
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:13:48Z"}
{"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2023-04-14T13:13:48Z"}
permissions:
root@4fa0fb4534ec:/var/opt/gitlab# ls -la
total 120
drwxr-xr-x 23 root root 4096 Apr 14 13:05 .
drwxr-xr-x 3 root root 4096 Jan 30 21:48 ..
-rw-r--r-- 1 git git 220 Feb 25 2020 .bash_logout
-rw-r--r-- 1 git git 3771 Feb 25 2020 .bashrc
drwxr-xr-x 2 git git 4096 Apr 14 13:02 .bundle
-rw-r--r-- 1 git git 298 Apr 14 13:02 .gitconfig
-rw-r--r-- 1 git git 807 Feb 25 2020 .profile
drwx------ 2 git git 4096 Apr 14 13:02 .ssh
drwxr-x--- 3 gitlab-prometheus root 4096 Apr 14 13:04 alertmanager
drwx------ 2 git root 4096 Apr 14 13:02 backups
-rw------- 1 root root 38 Apr 14 13:04 bootstrapped
drwxr-xr-x 2 gitlab-consul gitlab-consul 4096 Jan 30 21:48 consul
drwx------ 3 git git 4096 Apr 14 13:02 git-data
drwx------ 3 git root 4096 Apr 14 13:05 gitaly
drwxr-xr-x 3 git root 4096 Apr 14 13:02 gitlab-ci
drwxr-xr-x 2 git root 4096 Apr 14 13:04 gitlab-exporter
drwx------ 2 git root 4096 Apr 14 13:03 gitlab-kas
drwxr-xr-x 9 git root 4096 Apr 14 13:04 gitlab-rails
drwx------ 2 git root 4096 Apr 14 13:02 gitlab-shell
drwxr-x--- 3 git gitlab-www 4096 Apr 14 13:04 gitlab-workhorse
drwx------ 3 root root 4096 Apr 14 13:12 logrotate
drwxr-xr-x 2 mattermost mattermost 4096 Jan 30 21:48 mattermost
drwxr-x--- 9 root gitlab-www 4096 Apr 14 13:04 nginx
drwx------ 2 gitlab-psql root 4096 Apr 14 13:05 postgres-exporter
drwxr-xr-x 3 gitlab-psql gitlab-psql 4096 Apr 14 13:03 postgresql
drwxr-x--- 4 gitlab-prometheus gitlab-prometheus 4096 Apr 14 13:04 prometheus
-rw-r--r-- 1 root root 356 Apr 14 13:05 public_attributes.json
drwxr-x--- 2 gitlab-redis git 4096 Apr 14 13:18 redis
drwxr-xr-x 2 registry registry 4096 Jan 30 21:48 registry
-rw-r--r-- 1 root root 40 Apr 14 13:02 trusted-certs-directory-hash
redis permissions
root@4fa0fb4534ec:/var/opt/gitlab/redis# ls -la
total 236
drwxr-x--- 2 gitlab-redis git 4096 Apr 14 13:18 .
drwxr-xr-x 23 root root 4096 Apr 14 13:05 ..
-rw-r--r-- 1 gitlab-redis gitlab-redis 220 Feb 25 2020 .bash_logout
-rw-r--r-- 1 gitlab-redis gitlab-redis 3771 Feb 25 2020 .bashrc
-rw-r--r-- 1 gitlab-redis gitlab-redis 807 Feb 25 2020 .profile
-rw------- 1 gitlab-redis gitlab-redis 134939 Apr 14 13:18 dump.rdb
-rw-r--r-- 1 gitlab-redis root 83307 Apr 14 13:02 redis.conf
srwxrwxrwx 1 gitlab-redis gitlab-redis 0 Apr 14 13:02 redis.socket
pinging
root@4fa0fb4534ec:/var/opt/gitlab/redis# whoami
root
root@4fa0fb4534ec:/var/opt/gitlab/redis# redis-cli -s ./redis.socket ping
PONGredis log:
root@4fa0fb4534ec:/var/log/gitlab/redis# ls -la
total 16
drwx------ 2 gitlab-redis root 4096 Apr 14 13:02 .
drwxr-xr-x 20 root root 4096 Apr 14 13:04 ..
lrwxrwxrwx 1 root root 31 Apr 14 13:02 config -> /opt/gitlab/sv/redis/log/config
-rw-r--r-- 1 root root 4624 Apr 14 13:23 current
-rw------- 1 root root 0 Apr 14 13:02 lock
root@4fa0fb4534ec:/var/log/gitlab/redis# cat current
2023-04-14_13:02:57.21575 510:C 14 Apr 2023 13:02:57.215 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2023-04-14_13:02:57.21582 510:C 14 Apr 2023 13:02:57.215 # Redis version=6.2.8, bits=64, commit=423c78f4, modified=1, pid=510, just started
2023-04-14_13:02:57.21583 510:C 14 Apr 2023 13:02:57.215 # Configuration loaded
2023-04-14_13:02:57.21607 510:M 14 Apr 2023 13:02:57.216 * monotonic clock: POSIX clock_gettime
2023-04-14_13:02:57.21641 _._
2023-04-14_13:02:57.21642 _.-``__ ''-._
2023-04-14_13:02:57.21642 _.-`` `. `_. ''-._ Redis 6.2.8 (423c78f4/1) 64 bit
2023-04-14_13:02:57.21642 .-`` .-```. ```\/ _.,_ ''-._
2023-04-14_13:02:57.21643 ( ' , .-` | `, ) Running in standalone mode
2023-04-14_13:02:57.21643 |`-._`-...-` __...-.``-._|'` _.-'| Port: 0
2023-04-14_13:02:57.21643 | `-._ `._ / _.-' | PID: 510
2023-04-14_13:02:57.21643 `-._ `-._ `-./ _.-' _.-'
2023-04-14_13:02:57.21644 |`-._`-._ `-.__.-' _.-'_.-'|
2023-04-14_13:02:57.21644 | `-._`-._ _.-'_.-' | https://redis.io
2023-04-14_13:02:57.21644 `-._ `-._`-.__.-'_.-' _.-'
2023-04-14_13:02:57.21644 |`-._`-._ `-.__.-' _.-'_.-'|
2023-04-14_13:02:57.21644 | `-._`-._ _.-'_.-' |
2023-04-14_13:02:57.21645 `-._ `-._`-.__.-'_.-' _.-'
2023-04-14_13:02:57.21645 `-._ `-.__.-' _.-'
2023-04-14_13:02:57.21645 `-._ _.-'
2023-04-14_13:02:57.21646 `-.__.-'
2023-04-14_13:02:57.21646
2023-04-14_13:02:57.21646 510:M 14 Apr 2023 13:02:57.216 # Server initialized
2023-04-14_13:02:57.21646 510:M 14 Apr 2023 13:02:57.216 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2023-04-14_13:02:57.21668 510:M 14 Apr 2023 13:02:57.216 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
2023-04-14_13:07:58.03472 510:M 14 Apr 2023 13:07:58.034 * 10 changes in 300 seconds. Saving...
2023-04-14_13:07:58.03506 510:M 14 Apr 2023 13:07:58.035 * Background saving started by pid 1403
2023-04-14_13:07:58.04004 1403:C 14 Apr 2023 13:07:58.040 * DB saved on disk
2023-04-14_13:07:58.04033 1403:C 14 Apr 2023 13:07:58.040 * RDB: 0 MB of memory used by copy-on-write
2023-04-14_13:07:58.13536 510:M 14 Apr 2023 13:07:58.135 * Background saving terminated with success
2023-04-14_13:12:59.00852 510:M 14 Apr 2023 13:12:59.008 * 10 changes in 300 seconds. Saving...
2023-04-14_13:12:59.00885 510:M 14 Apr 2023 13:12:59.008 * Background saving started by pid 1501
2023-04-14_13:12:59.01400 1501:C 14 Apr 2023 13:12:59.013 * DB saved on disk
2023-04-14_13:12:59.01435 1501:C 14 Apr 2023 13:12:59.014 * RDB: 0 MB of memory used by copy-on-write
2023-04-14_13:12:59.10925 510:M 14 Apr 2023 13:12:59.109 * Background saving terminated with success
2023-04-14_13:18:00.01204 510:M 14 Apr 2023 13:18:00.012 * 10 changes in 300 seconds. Saving...
2023-04-14_13:18:00.01237 510:M 14 Apr 2023 13:18:00.012 * Background saving started by pid 1604
2023-04-14_13:18:00.01992 1604:C 14 Apr 2023 13:18:00.019 * DB saved on disk
2023-04-14_13:18:00.02023 1604:C 14 Apr 2023 13:18:00.020 * RDB: 1 MB of memory used by copy-on-write
2023-04-14_13:18:00.11308 510:M 14 Apr 2023 13:18:00.113 * Background saving terminated with success
2023-04-14_13:23:01.08436 510:M 14 Apr 2023 13:23:01.084 * 10 changes in 300 seconds. Saving...
2023-04-14_13:23:01.08484 510:M 14 Apr 2023 13:23:01.084 * Background saving started by pid 1711
2023-04-14_13:23:01.09448 1711:C 14 Apr 2023 13:23:01.094 * DB saved on disk
2023-04-14_13:23:01.09490 1711:C 14 Apr 2023 13:23:01.094 * RDB: 2 MB of memory used by copy-on-write
2023-04-14_13:23:01.18573 510:M 14 Apr 2023 13:23:01.185 * Background saving terminated with success
gitlab rails log:
root@4fa0fb4534ec:/var/log/gitlab/redis# cat /var/log/gitlab/gitlab-rails/exceptions_json.log | grep error
{"severity":"ERROR","time":"2023-04-14T13:03:45.766Z","correlation_id":"34784b65fe27b6915a16e906685a1577","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:433:in `user_commit_files'","lib/gitlab/git/repository.rb:990:in `block in commit_files'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:989:in `commit_files'","app/models/repository.rb:877:in `block in commit_files'","app/models/repository.rb:860:in `with_cache_hooks'","app/models/repository.rb:877:in `commit_files'","app/models/repository.rb:835:in `create_file'","app/services/files/create_service.rb:16:in `create_transformed_commit'","app/services/files/create_service.rb:10:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","app/services/projects/create_service.rb:197:in `create_readme'","app/services/projects/create_service.rb:132:in `after_create_actions'","ee/app/services/ee/projects/create_service.rb:62:in `after_create_actions'","app/services/projects/create_service.rb:76:in `block in execute'","lib/gitlab/application_context.rb:115:in `block in use'","lib/gitlab/application_context.rb:115:in `use'","lib/gitlab/application_context.rb:55:in `with_context'","app/services/projects/create_service.rb:75:in `execute'","ee/app/services/ee/projects/create_service.rb:31:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:333:in `block in transaction'","lib/gitlab/database.rb:332:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"34784b65fe27b6915a16e906685a1577"}
i'm in a container, so i tried to do:
root@4fa0fb4534ec:/var/log/gitlab/redis# sysctl vm.overcommit_memory=1
sysctl: setting key "vm.overcommit_memory", ignoring: Read-only file systemworkhorse:
root@4fa0fb4534ec:/var/log/gitlab/gitlab-workhorse# cat current | grep error
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:04:23Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:04:33Z","uri":""}
{"correlation_id":"01GXZXW58QNZD9Z53EKCMRKB4E","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:04:33Z","uri":"/help"}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:04:43Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:04:53Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:05:03Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: no such file or directory","level":"error","method":"GET","msg":"","time":"2023-04-14T13:05:13Z","uri":""}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:05:23Z"}
{"error":"context deadline exceeded","level":"fatal","msg":"shutting down","time":"2023-04-14T13:05:40Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:06:41Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:07:42Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:08:43Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:09:44Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:10:45Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:11:46Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:12:47Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:13:48Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:14:49Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:15:50Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:16:51Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:17:52Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:18:53Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:19:54Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:20:55Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:21:56Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:22:57Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:23:58Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:24:59Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:26:00Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:27:01Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:28:02Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:29:03Z"}
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2023-04-14T13:30:04Z"}
rails exceptions:
root@4fa0fb4534ec:/var/log/gitlab/gitlab-workhorse# cat /var/log/gitlab/gitlab-rails/exceptions_json.log
# Logfile created on 2023-04-14 13:03:45 +0000 by logger.rb/v1.4.2
{"severity":"ERROR","time":"2023-04-14T13:03:45.588Z","correlation_id":"34784b65fe27b6915a16e906685a1577","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"AuthorizedProjectUpdate::ProjectRecalculateWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:156:in `setup_authorizations'","app/services/projects/create_service.rb:126:in `after_create_actions'","ee/app/services/ee/projects/create_service.rb:62:in `after_create_actions'","app/services/projects/create_service.rb:76:in `block in execute'","lib/gitlab/application_context.rb:115:in `block in use'","lib/gitlab/application_context.rb:115:in `use'","lib/gitlab/application_context.rb:55:in `with_context'","app/services/projects/create_service.rb:75:in `execute'","ee/app/services/ee/projects/create_service.rb:31:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:333:in `block in transaction'","lib/gitlab/database.rb:332:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"34784b65fe27b6915a16e906685a1577"}
{"severity":"ERROR","time":"2023-04-14T13:03:45.601Z","correlation_id":"34784b65fe27b6915a16e906685a1577","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Projects::PostCreationWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/projects/create_service.rb:130:in `after_create_actions'","ee/app/services/ee/projects/create_service.rb:62:in `after_create_actions'","app/services/projects/create_service.rb:76:in `block in execute'","lib/gitlab/application_context.rb:115:in `block in use'","lib/gitlab/application_context.rb:115:in `use'","lib/gitlab/application_context.rb:55:in `with_context'","app/services/projects/create_service.rb:75:in `execute'","ee/app/services/ee/projects/create_service.rb:31:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:333:in `block in transaction'","lib/gitlab/database.rb:332:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"34784b65fe27b6915a16e906685a1577"}
{"severity":"ERROR","time":"2023-04-14T13:03:45.766Z","correlation_id":"34784b65fe27b6915a16e906685a1577","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:433:in `user_commit_files'","lib/gitlab/git/repository.rb:990:in `block in commit_files'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:989:in `commit_files'","app/models/repository.rb:877:in `block in commit_files'","app/models/repository.rb:860:in `with_cache_hooks'","app/models/repository.rb:877:in `commit_files'","app/models/repository.rb:835:in `create_file'","app/services/files/create_service.rb:16:in `create_transformed_commit'","app/services/files/create_service.rb:10:in `create_commit!'","app/services/commits/create_service.rb:30:in `execute'","app/services/projects/create_service.rb:197:in `create_readme'","app/services/projects/create_service.rb:132:in `after_create_actions'","ee/app/services/ee/projects/create_service.rb:62:in `after_create_actions'","app/services/projects/create_service.rb:76:in `block in execute'","lib/gitlab/application_context.rb:115:in `block in use'","lib/gitlab/application_context.rb:115:in `use'","lib/gitlab/application_context.rb:55:in `with_context'","app/services/projects/create_service.rb:75:in `execute'","ee/app/services/ee/projects/create_service.rb:31:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:333:in `block in transaction'","lib/gitlab/database.rb:332:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"34784b65fe27b6915a16e906685a1577"}
{"severity":"ERROR","time":"2023-04-14T13:03:45.789Z","correlation_id":"34784b65fe27b6915a16e906685a1577","exception.class":"Sidekiq::Worker::EnqueueFromTransactionError","exception.message":"Pages::InvalidateDomainCacheWorker.perform_async cannot be enqueued inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n","exception.backtrace":["config/initializers/forbid_sidekiq_in_transactions.rb:28:in `raise_inside_transaction_exception'","config/initializers/forbid_sidekiq_in_transactions.rb:43:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","lib/gitlab/event_store/subscription.rb:16:in `consume_event'","lib/gitlab/event_store/store.rb:33:in `block in publish'","lib/gitlab/event_store/store.rb:32:in `each'","lib/gitlab/event_store/store.rb:32:in `publish'","lib/gitlab/event_store.rb:17:in `publish'","app/services/projects/create_service.rb:320:in `publish_event'","app/services/projects/create_service.rb:135:in `after_create_actions'","ee/app/services/ee/projects/create_service.rb:62:in `after_create_actions'","app/services/projects/create_service.rb:76:in `block in execute'","lib/gitlab/application_context.rb:115:in `block in use'","lib/gitlab/application_context.rb:115:in `use'","lib/gitlab/application_context.rb:55:in `with_context'","app/services/projects/create_service.rb:75:in `execute'","ee/app/services/ee/projects/create_service.rb:31:in `execute'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:59:in `create_project'","app/models/concerns/stepable.rb:14:in `call'","app/models/concerns/stepable.rb:14:in `block in execute_steps'","app/models/concerns/stepable.rb:13:in `each'","app/models/concerns/stepable.rb:13:in `inject'","app/models/concerns/stepable.rb:13:in `execute_steps'","lib/gitlab/database_importers/self_monitoring/project/create_service.rb:27:in `execute'","(eval):3:in `block (2 levels) in run_file'","lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'","lib/gitlab/database.rb:333:in `block in transaction'","lib/gitlab/database.rb:332:in `transaction'","lib/tasks/gitlab/db.rake:104:in `block (3 levels) in \u003ctop (required)\u003e'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":"34784b65fe27b6915a16e906685a1577"}
some extra:
inside container:
root@4fa0fb4534ec:/var/log/gitlab/gitlab-workhorse# df -h
Filesystem Size Used Avail Use% Mounted on
fuse-overlayfs 16G 13G 2.2G 86% /
tmpfs 64M 0 64M 0% /dev
tmpfs 1.6G 53M 1.5G 4% /etc/hosts
/dev/nvme0n1p1 16G 13G 2.2G 86% /etc/gitlab
shm 9.4G 6.7M 9.4G 1% /dev/shm
udev 7.7G 0 7.7G 0% /dev/tty
outside container:
admin@ip-10-235-77-28:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.7G 0 7.7G 0% /dev
tmpfs 1.6G 448K 1.6G 1% /run
/dev/nvme0n1p1 16G 13G 2.2G 86% /
tmpfs 7.7G 168K 7.7G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/nvme0n1p15 124M 11M 114M 9% /boot/efi
tmpfs 1.6G 53M 1.5G 4% /run/user/1000
admin@ip-10-235-77-28:~$ cat /proc/meminfo
MemTotal: 16033748 kB
MemFree: 733352 kB
MemAvailable: 11544572 kB
Buffers: 490824 kB
Cached: 9788624 kB
SwapCached: 0 kB
Active: 2735172 kB
Inactive: 11418820 kB
Active(anon): 70020 kB
Inactive(anon): 3877940 kB
Active(file): 2665152 kB
Inactive(file): 7540880 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 512 kB
Writeback: 0 kB
AnonPages: 3842872 kB
Mapped: 514788 kB
Shmem: 81524 kB
KReclaimable: 941864 kB
Slab: 1052432 kB
SReclaimable: 941864 kB
SUnreclaim: 110568 kB
KernelStack: 8384 kB
PageTables: 24600 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8016872 kB
Committed_AS: 4101300 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 25668 kB
VmallocChunk: 0 kB
Percpu: 11936 kB
AnonHugePages: 483328 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 278440 kB
DirectMap2M: 15083520 kB
root@4fa0fb4534ec:/var/opt/gitlab# free -h
total used free shared buff/cache available
Mem: 15Gi 3.9Gi 744Mi 79Mi 10Gi 11Gi
Swap: 0B 0B 0B
root@4fa0fb4534ec:/var/opt/gitlab# vmstat --unit mb
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 807 502 10989 0 0 1 131 4 3 7 2 92 0 0
so this was impacting my ability to do object storage, so i started from the most basic container with no volumes. this seems to be an issue with any image i use. so, not sure what is going on - hoping there's a solution.
Edited by avnav88