Self-hosted GitLab CE times out graphql requests until restart, when left unused for a while
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
After GitLab is unused for a few days (not uncommon, single-user instance) the web-ui returns An error occurred loading the projects. Please refresh the page to try again., doesn't load projects, and /api/graphql returns Timeout on validation of query
Steps to reproduce
Don't use a single-user/standalone instance for a few days (maybe a week), then log in
Example Project
Not applicable
What is the current bug behavior?
I run a self-hosted GitLab CE (18.0.0-ce.0 as of today, but I've been having this problem for a year or so, on many versions) on a server with an 8-core Core(TM) i7-2600, 16GB RAM, 2x RAID1 "spinning rust" 4TB HDDs, running Debian Stable (12.11).
I am the sole user of the instance, and it's running behing Apache 2.4.62 using a slightly modified version of the Apache 2.4 recipe from https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache
After a few days of the server sitting there doing nothing (I don't use the UI that much) when I login to the UI I get An error occurred loading the projects. Please refresh the page to try again. when loading my projects. Reloading doesn't work. A few pages work, some others don't (don't have a list, sorry).
A gitlab-ctl restart fixes it, until it happens again.
The /api/graphql endpoint returns
{"errors":[{"message":"Timeout on validation of query","locations":[],"extensions":{"code":"validationTimeout"}}]}/var/log/gitlab/gitlab-rails/production_json.log (I only have this one) doesn't show anything interesting: all the statuses >= 400 are some 401s that I get before login.
If I go on the server and curl 'http://127.0.0.1:8181/api/graphql' -X POST [...] I get the same error; on port 8181 is gitlab-workhorse (gitlab_workhorse['listen_addr'] = "127.0.0.1:8181" from gitlab.rb), but /var/log/gitlab/gitlab-workhorse/current has barely anything happening when I refresh the page (and get the error). See [1] below (I tried to use a collapsible section and markdown destroyed formatting).
I doubt the \\z has anything to do with this, because - as I mentioned - restarting the instance solves this (albeit temporarily).
Things I tried:
- optimize the instance using https://docs.gitlab.com/omnibus/settings/memory_constrained_envs/ both with and without the experimental features
- I'm fairly sure I used a config option for single-node/standalone mode but I can't find it anymore
- A full reinstall of the instance (with a backup/restore, but I don't want to do this again)
I assumed it was an issue of caching something that only appeared after not using the instance for a while, but would resolve itself after a couple of page reloads, as the cache gets refilled, but it doesn't seem to be the case.
And the weirdest part of all this os that the performance bar - despite the page taking seconds to load (~10s after login, then a "slightly faster" 5-ish seconds) - shows everything as if it was ok, which it very clearly isn't).
I'm open to suggestions, or testing (although I need the instance now so I'll go ahead and restart GitLab now, meaning I may need it to "crash" again before I can test things).
What is the expected correct behavior?
GitLab loads projects and pages correctly
Relevant logs and/or screenshots
[1]
/var/log/gitlab/gitlab-workhorse/current extract
$ cat workhorse |jq '. | select(.uri != "/api/v4/jobs/request")'
{
"backend_id": "rails",
"content_type": "text/html; charset=utf-8",
"correlation_id": "01JWSAN27481B77ZK8D1VMEZ5P",
"duration_ms": 16696,
"host": "MY_SERVER_URL",
"level": "info",
"method": "GET",
"msg": "access",
"proto": "HTTP/1.1",
"read_bytes": 1594,
"referrer": "https://MY_SERVER_URL/users/sign_in",
"remote_addr": "[REDACTED_IP]:48520",
"remote_ip": "REDACTED_IP",
"route": "",
"route_id": "default",
"status": 200,
"system": "http",
"time": "2025-06-03T00:02:23+02:00",
"ttfb_ms": 16695,
"uri": "/dashboard/projects/member",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
"written_bytes": 27722
}
{
"backend_id": "rails",
"content_type": "application/json; charset=utf-8",
"correlation_id": "01JWSANJX9JYJVRBVYE2KBJ6X1",
"duration_ms": 3345,
"host": "MY_SERVER_URL",
"level": "info",
"method": "POST",
"msg": "access",
"proto": "HTTP/1.1",
"read_bytes": 3715,
"referrer": "https://MY_SERVER_URL/dashboard/projects/member",
"remote_addr": "[REDACTED_IP]:57128",
"remote_ip": "REDACTED_IP",
"route": "^/api/graphql\\z",
"route_id": "api_graphql",
"status": 200,
"system": "http",
"time": "2025-06-03T00:02:26+02:00",
"ttfb_ms": 3345,
"uri": "/api/graphql",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
"written_bytes": 114
}
{
"backend_id": "rails",
"content_type": "application/json; charset=utf-8",
"correlation_id": "01JWSANJWQFQMAMPN78WWPNV6S",
"duration_ms": 4614,
"host": "MY_SERVER_URL",
"level": "info",
"method": "POST",
"msg": "access",
"proto": "HTTP/1.1",
"read_bytes": 2196,
"referrer": "https://MY_SERVER_URL/dashboard/projects/member",
"remote_addr": "[REDACTED_IP]:57118",
"remote_ip": "REDACTED_IP",
"route": "^/api/graphql\\z",
"route_id": "api_graphql",
"status": 200,
"system": "http",
"time": "2025-06-03T00:02:28+02:00",
"ttfb_ms": 4614,
"uri": "/api/graphql",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
"written_bytes": 114
}
All the uncommented lines from my gitlab.rb
$ cat gitlab.rb |grep -vE "^#|^$"
external_url 'https://REDACTED'
gitlab_rails['gitlab_default_theme'] = 2
gitlab_rails['gitlab_default_projects_features_wiki'] = false
gitlab_rails['gitlab_default_projects_features_container_registry'] = false
gitlab_rails['trusted_proxies'] = [ '127.0.0.1/8' , 'IP', 'ANOTHER IP', 'AND AN IPV6 ONE']
gitlab_rails['usage_ping_enabled'] = false
gitlab_rails['backup_keep_time'] = 259200
gitlab_workhorse['listen_network'] = "tcp"
gitlab_workhorse['listen_addr'] = "127.0.0.1:8181"
gitlab_workhorse['trusted_cidrs_for_x_forwarded_for'] = [ '127.0.0.1/8' , 'IP', 'ANOTHER IP', 'AND AN IPV6 ONE']
web_server['external_users'] = ['www-data']
nginx['enable'] = false
gitlab_rails['gitlab_kas_enabled'] = false
mattermost['enable'] = false
registry_nginx['enable'] = false
prometheus_monitoring['enable'] = falseOutput of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 12 Current User: git Using RVM: no Ruby Version: 3.2.5 Gem Version: 3.6.7 Bundler Version:2.6.5 Rake Version: 13.0.6 Redis Version: 7.2.7 Sidekiq Version:7.3.9 Go Version: go1.19.8 linux/amd64 GitLab information Version: 18.0.0 Revision: c481e1bd1b8 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 16.8 URL: https://REDACTED HTTP Clone URL: https://REDACTED/some-group/some-project.git SSH Clone URL: git@REDACTED:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.41.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 18.0.0 - default Git Version: 2.49.0.gl2
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.41.0 ? ... OK (14.41.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 1/1 ... yes 1/2 ... yes 1/3 ... yes 2/6 ... yes 3/7 ... yes 3/11 ... yes 3/12 ... yes 3/14 ... yes 4/15 ... yes 4/16 ... yes 3/17 ... yes 5/18 ... yes 3/19 ... yes 3/20 ... yes 3/21 ... yes 3/22 ... yes 3/23 ... yes 3/24 ... yes 3/25 ... yes 3/26 ... yes 8/27 ... yes 3/28 ... yes 3/29 ... yes 4/30 ... yes 4/31 ... yes 4/32 ... yes 4/33 ... yes 4/35 ... yes 4/36 ... yes 3/37 ... yes 3/38 ... yes 3/39 ... yes 3/40 ... yes 3/41 ... yes 8/42 ... yes 3/43 ... yes 4/45 ... yes 4/46 ... yes 4/47 ... yes 4/48 ... yes 3/49 ... yes 3/50 ... yes 19/52 ... yes 4/53 ... yes 3/54 ... yes 3/55 ... yes 19/56 ... yes 88/57 ... yes 4/58 ... yes 3/59 ... yes 19/60 ... yes 3/61 ... yes 19/62 ... yes Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.2.5) Git user has default SSH configuration? ... yes Active users: ... 9 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished