Error 500 after 14.5.0 CE upgrade; restarting gitlab docker container helps for a certain time

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

After upgrading for Gitlab 14.5.0 CE by loading the newer docker image I get error 500 after a certain time of Gitlab running. But the error behavior is really strange: if you e.g. try to open a Gitlab page, then the error 500 occurs, clicking reload the page loads with no error. Error 500 to working operation is approximately in a 1:10 ratio.

Also upgrading to 14.5.1 and 14.5.2 did not bring any improvement, the error still occurs after a few hours of Gitlab running.

Running a check everything looks ok:

root@gitlab:/# gitlab-rake gitlab:check --trace
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
GitLab Instance / Monitoring ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 32
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

But sometimes even during the check the 500 error occurs:

root@gitlab:/# gitlab-rake gitlab:check --trace
** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: FAILED - 500 Internal Server Error
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

In the file /var/log/gitlab/gitlab-rails/production.log if found two different error stack traces:

Started GET "/prg/prg.git/info/refs?service=git-upload-pack" for 172.18.0.57 at 2021-12-10 06:22:05 +0000
Processing by Repositories::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "repository_path"=>"prg/prg.git"}
Completed 500 Internal Server Error in 4ms (ActiveRecord: 0.8ms | Elasticsearch: 0.0ms | Allocations: 2727)
  
IOError (unmapped file):
  
lib/gitlab/metrics/subscribers/rails_cache.rb:62:in `observe'
lib/gitlab/metrics/subscribers/rails_cache.rb:12:in `cache_read'
lib/feature.rb:86:in `enabled?'
app/models/concerns/routable.rb:118:in `full_path'
lib/gitlab/repo_path.rb:69:in `redirected?'
lib/gitlab/repo_path.rb:31:in `block in parse'
lib/gitlab/repo_path.rb:19:in `each'
lib/gitlab/repo_path.rb:19:in `parse'
app/controllers/repositories/git_http_client_controller.rb:95:in `parse_repo_path'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

and after a certain time with Gitlab running also this one:

Started GET "/lem/lem-frontend.git/info/refs?service=git-upload-pack" for 172.18.0.57 at 2021-12-10 07:09:51 +0000
Processing by Repositories::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "repository_path"=>"lem/lem-frontend.git"}
Completed 500 Internal Server Error in 98ms (ActiveRecord: 32.9ms | Elasticsearch: 0.0ms | Allocations: 55915)
  
IOError (unmapped file):
  
lib/gitlab/instrumentation/redis_base.rb:102:in `instance_observe_duration'
lib/gitlab/instrumentation/redis_interceptor.rb:29:in `ensure in call'
lib/gitlab/instrumentation/redis_interceptor.rb:36:in `call'
lib/feature.rb:86:in `enabled?'
app/models/concerns/routable.rb:118:in `full_path'
lib/gitlab/repo_path.rb:69:in `redirected?'
lib/gitlab/repo_path.rb:31:in `block in parse'
lib/gitlab/repo_path.rb:19:in `each'
lib/gitlab/repo_path.rb:19:in `parse'
app/controllers/repositories/git_http_client_controller.rb:95:in `parse_repo_path'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Is there a solution for this issue?