GitLab's backing PostgreSQL server leaking memory
Summary
I don't have a good understanding of PostgreSQL memory allocation or GitLab's usage of the backing databases, but we're seeing this behavior with our RDS Postgres instance's memory usage:
As you can probably guess, memory started going down when people came to work and started using GitLab. Memory does not get freed overnight, which is why I believe this to be a bug in GitLab. The effect is that GitLab gets crazy slow after a day or two when PostgreSQL starts using swap and eventually fills that up as well.
Running SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state='idle' causes an instant drop in memory usage to nearly zero as seen here twice:
Steps to reproduce
Well, that's difficult to tell. We use the docker image for 8.14 with Postgres 9.6. We have ~100 users and ~200 projects, and we also use GitLab CI and the Docker Registry extensively—we have ~50k builds and ~1 TB of Docker images (we build and keep images for almost every commit).
Expected behavior
We'd like some free memory.
Actual behavior
We don't have free memory
Relevant logs and/or screenshots
Output of SELECT * FROM pg_stat_activity ORDER BY state_change DESC NULLS LAST:
datid datname pid usesysid usename application_name client_addr client_hostname client_port backend_start xact_start query_start state_change wait_event_type wait_event state backend_xid backend_xmin query
16400 gitlab 4257 16386 gitlab 10.2.12.113 46480 2016-12-09 18:27:50.366929 2016-12-09 18:28:16.664650 2016-12-09 18:28:16.665108 2016-12-09 18:28:16.665109 active 956159 "SELECT *
FROM pg_stat_activity
ORDER BY state_change DESC"
16400 gitlab 17136 16386 gitlab unicorn worker[3] -D -E produc...service/gitlab-rails/config.ru 10.2.41.111 54887 2016-12-09 15:41:34.915917 2016-12-09 18:28:16.214298 2016-12-09 18:28:16.214359 idle SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = $1 LEFT JOIN project_features ON ci_builds.gl_project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."gl_project_id", count(*) AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'running' AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') GROUP BY "ci_builds"."gl_project_id") AS project_builds ON ci_builds.gl_project_id=project_builds.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = $2 AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = $3 AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) ORDER BY COALESCE(project_builds.running_builds, 0) ASC, ci_builds.id ASC
16400 gitlab 24457 16386 gitlab unicorn worker[4] -D -E produc...service/gitlab-rails/config.ru 10.2.41.111 42141 2016-12-09 16:04:47.580832 2016-12-09 18:28:15.193014 2016-12-09 18:28:15.193064 idle SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = $1 LEFT JOIN project_features ON ci_builds.gl_project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."gl_project_id", count(*) AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'running' AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') GROUP BY "ci_builds"."gl_project_id") AS project_builds ON ci_builds.gl_project_id=project_builds.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = $2 AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = $3 AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) ORDER BY COALESCE(project_builds.running_builds, 0) ASC, ci_builds.id ASC
16400 gitlab 15479 16386 gitlab unicorn worker[1] -D -E produc...service/gitlab-rails/config.ru 10.2.41.111 51237 2016-12-09 15:36:09.706232 2016-12-09 18:28:14.742151 2016-12-09 18:28:14.742204 idle SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = $1 LEFT JOIN project_features ON ci_builds.gl_project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."gl_project_id", count(*) AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'running' AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') GROUP BY "ci_builds"."gl_project_id") AS project_builds ON ci_builds.gl_project_id=project_builds.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = $2 AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = $3 AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) ORDER BY COALESCE(project_builds.running_builds, 0) ASC, ci_builds.id ASC
16400 gitlab 30858 16386 gitlab unicorn worker[0] -D -E produc...service/gitlab-rails/config.ru 10.2.41.111 40874 2016-12-09 18:09:20.109876 2016-12-09 18:28:13.710635 2016-12-09 18:28:13.710684 idle SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = $1 LEFT JOIN project_features ON ci_builds.gl_project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."gl_project_id", count(*) AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'running' AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') GROUP BY "ci_builds"."gl_project_id") AS project_builds ON ci_builds.gl_project_id=project_builds.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = $2 AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = $3 AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) ORDER BY COALESCE(project_builds.running_builds, 0) ASC, ci_builds.id ASC
16400 gitlab 15867 16386 gitlab unicorn worker[2] -D -E produc...service/gitlab-rails/config.ru 10.2.41.111 52061 2016-12-09 15:37:30.705083 2016-12-09 18:28:10.737390 2016-12-09 18:28:10.737442 idle SELECT "ci_builds".* FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = $1 LEFT JOIN project_features ON ci_builds.gl_project_id = project_features.project_id LEFT JOIN (SELECT "ci_builds"."gl_project_id", count(*) AS running_builds FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'running' AND "ci_builds"."runner_id" IN (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') GROUP BY "ci_builds"."gl_project_id") AS project_builds ON ci_builds.gl_project_id=project_builds.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = $2 AND "ci_builds"."runner_id" IS NULL AND "projects"."shared_runners_enabled" = $3 AND (project_features.builds_access_level IS NULL or project_features.builds_access_level > 0) ORDER BY COALESCE(project_builds.running_builds, 0) ASC, ci_builds.id ASC
16400 gitlab 3060 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50993 2016-12-09 11:28:43.335757 2016-12-09 18:20:11.650823 2016-12-09 18:20:11.652441 idle UPDATE "projects" SET "last_repository_check_failed" = 'f', "last_repository_check_at" = '2016-12-09 17:20:11.650121' WHERE "projects"."id" = $1
16400 gitlab 20110 16386 gitlab sidekiq 4.2.1 gitlab-rails [2 of 25 busy] 10.2.41.111 45322 2016-12-08 19:04:12.975701 2016-12-09 18:18:06.813149 2016-12-09 18:18:06.813182 idle SELECT sg.stage FROM (SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."commit_id" = $1) sg GROUP BY stage ORDER BY max(stage_idx), stage
16400 gitlab 3065 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50999 2016-12-09 11:28:43.489473 2016-12-09 18:18:06.754000 2016-12-09 18:18:06.754021 idle SELECT "licenses".* FROM "licenses" ORDER BY "licenses"."id" DESC LIMIT 1
16400 gitlab 20088 16386 gitlab sidekiq 4.2.1 gitlab-rails [0 of 25 busy] 10.2.41.111 45205 2016-12-08 19:04:05.214691 2016-12-09 18:18:06.729926 2016-12-09 18:18:06.729965 idle SELECT "services".* FROM "services" WHERE "services"."project_id" = $1 AND "services"."pipeline_events" = $2 AND "services"."active" = $3 ORDER BY "services"."id" DESC
16400 gitlab 20089 16386 gitlab sidekiq 4.2.1 gitlab-rails [0 of 25 busy] 10.2.41.111 45211 2016-12-08 19:04:05.691278 2016-12-09 18:18:06.666252 2016-12-09 18:18:06.666286 idle SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."deleted_at" IS NULL AND "merge_requests"."source_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) AND "merge_requests"."source_branch" = $2 ORDER BY "merge_requests"."id" DESC
16400 gitlab 19552 16386 gitlab sidekiq 4.2.1 gitlab-rails [0 of 25 busy] 10.2.41.111 44118 2016-12-08 19:02:24.927230 2016-12-09 18:18:06.663585 2016-12-09 18:18:06.663670 idle COMMIT
16400 gitlab 24986 16386 gitlab sidekiq 4.2.1 gitlab-rails [9 of 25 busy] 10.2.41.111 56206 2016-12-08 19:19:51.679863 2016-12-09 18:18:06.660614 2016-12-09 18:18:06.660738 idle COMMIT
16400 gitlab 20118 16386 gitlab sidekiq 4.2.1 gitlab-rails [2 of 25 busy] 10.2.41.111 45330 2016-12-08 19:04:13.372042 2016-12-09 18:18:06.650921 2016-12-09 18:18:06.650945 idle UPDATE "merge_request_metrics" SET "latest_build_started_at" = '2016-12-09 17:17:29.067425', "latest_build_finished_at" = '2016-12-09 17:18:06.613153', "pipeline_id" = 15844 WHERE 1=0
16400 gitlab 3061 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50994 2016-12-09 11:28:43.359613 2016-12-09 18:18:06.633131 2016-12-09 18:18:06.634694 idle COMMIT
16400 gitlab 24976 16386 gitlab sidekiq 4.2.1 gitlab-rails [6 of 25 busy] 10.2.41.111 56160 2016-12-08 19:19:49.103285 2016-12-09 18:18:06.583914 2016-12-09 18:18:06.583946 idle SELECT COUNT(*) FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."gl_project_id" = $1 AND "ci_builds"."status" IN ('running', 'pending')
16400 gitlab 3064 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50998 2016-12-09 11:28:43.470776 2016-12-09 18:18:06.522197 2016-12-09 18:18:06.522237 idle SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."id" = $1 LIMIT 1
16400 gitlab 20499 16386 gitlab sidekiq 4.2.1 gitlab-rails [0 of 25 busy] 10.2.41.111 46157 2016-12-08 19:05:23.670600 2016-12-09 18:17:44.681304 2016-12-09 18:17:44.681338 idle SELECT COUNT(*) FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."gl_project_id" = $1 AND "ci_builds"."status" IN ('running', 'pending')
16400 gitlab 20109 16386 gitlab sidekiq 4.2.1 gitlab-rails [2 of 25 busy] 10.2.41.111 45313 2016-12-08 19:04:12.109832 2016-12-09 18:17:44.675743 2016-12-09 18:17:44.675863 idle COMMIT
16400 gitlab 24987 16386 gitlab sidekiq 4.2.1 gitlab-rails [9 of 25 busy] 10.2.41.111 56217 2016-12-08 19:19:51.810100 2016-12-09 18:17:44.576562 2016-12-09 18:17:44.576592 idle SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."id" = $1 LIMIT 1
16400 gitlab 24988 16386 gitlab sidekiq 4.2.1 gitlab-rails [9 of 25 busy] 10.2.41.111 56223 2016-12-08 19:19:51.897814 2016-12-09 18:17:34.367377 2016-12-09 18:17:34.367401 idle SELECT "web_hooks".* FROM "web_hooks" WHERE "web_hooks"."type" IN ('SystemHook') AND "web_hooks"."push_events" = $1 ORDER BY "web_hooks"."id" DESC
16400 gitlab 3062 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50995 2016-12-09 11:28:43.380693 2016-12-09 18:17:34.326390 2016-12-09 18:17:34.326430 idle SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."deleted_at" IS NULL AND "merge_requests"."id" = $1 ORDER BY "merge_requests"."id" DESC LIMIT 1
16400 gitlab 24978 16386 gitlab sidekiq 4.2.1 gitlab-rails [9 of 25 busy] 10.2.41.111 56196 2016-12-08 19:19:50.612973 2016-12-09 18:17:32.743723 2016-12-09 18:17:32.743770 idle SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."deleted_at" IS NULL AND "merge_requests"."id" = $1 ORDER BY "merge_requests"."id" DESC LIMIT 1
16400 gitlab 6954 16386 gitlab sidekiq 4.2.1 gitlab-rails [1 of 25 busy] 10.2.41.111 56135 2016-12-09 08:13:07.618196 2016-12-09 18:17:31.543469 2016-12-09 18:17:31.543505 idle SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."deleted_at" IS NULL AND "merge_requests"."id" = $1 ORDER BY "merge_requests"."id" DESC LIMIT 1
16400 gitlab 24985 16386 gitlab sidekiq 4.2.1 gitlab-rails [9 of 25 busy] 10.2.41.111 56205 2016-12-08 19:19:51.633565 2016-12-09 18:17:30.520077 2016-12-09 18:17:30.520110 idle SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" DESC LIMIT 1
16400 gitlab 3063 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 50997 2016-12-09 11:28:43.399724 2016-12-09 18:17:30.426457 2016-12-09 18:17:30.426481 idle SELECT "services".* FROM "services" WHERE "services"."project_id" = $1 AND "services"."note_events" = $2 AND "services"."active" = $3 ORDER BY "services"."id" DESC
16400 gitlab 20111 16386 gitlab sidekiq 4.2.1 gitlab-rails [2 of 25 busy] 10.2.41.111 45324 2016-12-08 19:04:13.043413 2016-12-09 18:17:30.232322 2016-12-09 18:17:30.232349 idle SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."deleted_at" IS NULL AND "namespaces"."id" = $1 ORDER BY "namespaces"."id" DESC LIMIT 1
16400 gitlab 6955 16386 gitlab sidekiq 4.2.1 gitlab-rails [23 of 25 busy] 10.2.41.111 56164 2016-12-09 08:13:10.391657 2016-12-09 18:17:30.096237 2016-12-09 18:17:30.096414 idle SELECT "schema_migrations".* FROM "schema_migrations"
16400 gitlab 3066 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 51000 2016-12-09 11:28:43.508573 2016-12-09 18:17:30.054194 2016-12-09 18:17:30.054246 idle SELECT "services".* FROM "services" WHERE "services"."project_id" = $1 AND "services"."pipeline_events" = $2 AND "services"."active" = $3 ORDER BY "services"."id" DESC
16400 gitlab 3067 16386 gitlab sidekiq 4.2.1 gitlab-rails [17 of 25 busy] 10.2.41.111 51001 2016-12-09 11:28:43.532990 2016-12-09 18:17:28.248380 2016-12-09 18:17:28.248406 idle SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."deleted_at" IS NULL AND "namespaces"."id" = $1 ORDER BY "namespaces"."id" DESC LIMIT 1
16400 gitlab 24977 16386 gitlab sidekiq 4.2.1 gitlab-rails [6 of 25 busy] 10.2.41.111 56163 2016-12-08 19:19:49.345440 2016-12-09 18:17:27.745746 2016-12-09 18:17:27.745779 idle SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" DESC LIMIT 1
16400 gitlab 3709 16386 gitlab 10.2.12.113 46373 2016-12-09 16:42:25.097485 2016-12-09 16:42:26.334794 2016-12-09 16:42:26.335218 idle SELECT * FROM pg_stat_activity
16400 gitlab 19470 16386 gitlab /opt/gitlab/embedded/service/gem/ruby/2.3.0/bin/sidekiq 10.2.41.111 43837 2016-12-08 19:02:03.411133 2016-12-08 19:02:07.528756 2016-12-08 19:02:07.528884 idle " SELECT a.attname, format_type(a.atttypid, a.atttypmod),
pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '""todos""'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
"
16402 rdsadmin 3969 10 rdsadmin <insufficient privilege>PostgreSQL table/index sizes
table_name table_size indices_size total_size
"public"."merge_request_diffs" 187 MB 664 kB 188 MB
"public"."notes" 14 MB 67 MB 81 MB
"public"."ci_builds" 27 MB 47 MB 74 MB
"public"."events" 46 MB 7872 kB 54 MB
"public"."issues" 5464 kB 33 MB 38 MB
"public"."sent_notifications" 12 MB 2696 kB 15 MB
"public"."merge_requests" 1904 kB 9168 kB 11 MB
"public"."ci_commits" 3384 kB 4320 kB 7704 kB
"public"."todos" 752 kB 1592 kB 2344 kB
"public"."labels" 728 kB 976 kB 1704 kB
[smaller tables are probably irrelevant so trimmed]Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab application Check
I, [2016-12-09T18:47:01.378385 #2616] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
Checking GitLab Shell ...
GitLab Shell version >= 4.0.0 ? ... OK (4.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
8/7 ... ok
5/9 ... ok
7/12 ... repository is empty
8/13 ... ok
109/15 ... ok
26/17 ... ok
7/19 ... ok
29/20 ... repository is empty
29/22 ... ok
36/24 ... ok
36/25 ... ok
5/26 ... ok
30/27 ... ok
26/30 ... ok
5/32 ... ok
26/33 ... ok
3/38 ... ok
36/39 ... repository is empty
3/40 ... ok
2/42 ... ok
2/43 ... repository is empty
2/44 ... ok
7/45 ... ok
59/46 ... ok
5/47 ... ok
51/48 ... ok
33/52 ... ok
60/54 ... repository is empty
60/55 ... ok
60/56 ... ok
36/57 ... ok
11/58 ... ok
59/59 ... ok
26/60 ... ok
59/66 ... ok
52/67 ... ok
59/68 ... ok
52/69 ... ok
59/70 ... ok
69/72 ... ok
72/73 ... ok
81/74 ... ok
40/75 ... ok
56/76 ... repository is empty
60/77 ... repository is empty
81/80 ... ok
81/81 ... ok
81/82 ... ok
51/83 ... ok
81/84 ... ok
56/85 ... repository is empty
111/91 ... ok
51/94 ... ok
40/95 ... ok
29/96 ... repository is empty
2/97 ... ok
59/98 ... ok
7/99 ... ok
51/101 ... ok
3/102 ... ok
59/103 ... ok
26/104 ... ok
59/106 ... ok
60/107 ... ok
40/108 ... ok
4/109 ... ok
51/110 ... ok
53/111 ... ok
51/113 ... ok
81/114 ... ok
8/115 ... ok
26/116 ... ok
43/118 ... ok
43/119 ... ok
43/120 ... ok
40/121 ... ok
60/122 ... ok
109/123 ... ok
109/124 ... ok
59/125 ... ok
59/126 ... ok
59/127 ... ok
64/128 ... ok
55/129 ... ok
8/130 ... ok
8/132 ... ok
111/133 ... repository is empty
111/134 ... ok
109/135 ... ok
7/139 ... ok
111/140 ... ok
28/141 ... repository is empty
43/142 ... ok
59/144 ... ok
109/145 ... ok
69/146 ... ok
18/147 ... ok
3/149 ... ok
2/151 ... ok
2/152 ... ok
43/155 ... ok
43/159 ... ok
14/160 ... ok
2/161 ... ok
111/162 ... ok
18/163 ... ok
51/164 ... ok
15/165 ... ok
119/166 ... ok
109/167 ... ok
26/168 ... ok
51/169 ... ok
5/170 ... ok
111/172 ... ok
111/173 ... ok
52/174 ... ok
59/175 ... ok
109/176 ... ok
51/177 ... ok
36/178 ... ok
43/179 ... ok
128/181 ... ok
69/182 ... ok
59/183 ... ok
133/184 ... ok
11/185 ... ok
61/187 ... ok
51/188 ... ok
59/189 ... ok
5/190 ... ok
5/191 ... ok
59/192 ... ok
51/193 ... ok
59/194 ... ok
36/195 ... repository is empty
26/196 ... ok
43/197 ... ok
26/198 ... repository is empty
59/199 ... ok
8/200 ... ok
59/201 ... ok
103/202 ... ok
91/203 ... repository is empty
91/204 ... repository is empty
59/205 ... ok
111/206 ... ok
88/207 ... ok
43/208 ... ok
119/209 ... ok
51/210 ... ok
51/211 ... ok
59/212 ... ok
74/213 ... ok
88/215 ... ok
51/217 ... ok
111/218 ... ok
138/219 ... ok
82/220 ... ok
109/221 ... repository is empty
51/222 ... ok
51/223 ... ok
109/226 ... ok
36/227 ... ok
2/228 ... ok
5/229 ... ok
5/230 ... ok
3/231 ... ok
3/232 ... ok
3/233 ... repository is empty
36/234 ... ok
7/235 ... ok
109/236 ... ok
30/237 ... ok
51/238 ... ok
40/239 ... ok
18/240 ... ok
43/242 ... ok
59/243 ... repository is empty
91/244 ... repository is empty
43/245 ... ok
42/246 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
<trimmed>
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
8/7 ... yes
5/9 ... yes
7/12 ... yes
8/13 ... yes
109/15 ... yes
26/17 ... yes
7/19 ... yes
29/20 ... yes
29/22 ... yes
36/24 ... yes
36/25 ... yes
5/26 ... yes
30/27 ... yes
26/30 ... yes
5/32 ... yes
26/33 ... yes
3/38 ... yes
36/39 ... yes
3/40 ... yes
2/42 ... yes
2/43 ... yes
2/44 ... yes
7/45 ... yes
59/46 ... yes
5/47 ... yes
51/48 ... yes
33/52 ... yes
60/54 ... yes
60/55 ... yes
60/56 ... yes
36/57 ... yes
11/58 ... yes
59/59 ... yes
26/60 ... yes
59/66 ... yes
52/67 ... yes
59/68 ... yes
52/69 ... yes
59/70 ... yes
69/72 ... yes
72/73 ... yes
81/74 ... yes
40/75 ... yes
56/76 ... yes
60/77 ... yes
81/80 ... yes
81/81 ... yes
81/82 ... yes
51/83 ... yes
81/84 ... yes
56/85 ... yes
111/91 ... yes
51/94 ... yes
40/95 ... yes
29/96 ... yes
2/97 ... yes
59/98 ... yes
7/99 ... yes
51/101 ... yes
3/102 ... yes
59/103 ... yes
26/104 ... yes
59/106 ... yes
60/107 ... yes
40/108 ... yes
4/109 ... yes
51/110 ... yes
53/111 ... yes
51/113 ... yes
81/114 ... yes
8/115 ... yes
26/116 ... yes
43/118 ... yes
43/119 ... yes
43/120 ... yes
40/121 ... yes
60/122 ... yes
109/123 ... yes
109/124 ... yes
59/125 ... yes
59/126 ... yes
59/127 ... yes
64/128 ... yes
55/129 ... yes
8/130 ... yes
8/132 ... yes
111/133 ... yes
111/134 ... yes
109/135 ... yes
7/139 ... yes
111/140 ... yes
28/141 ... yes
43/142 ... yes
59/144 ... yes
109/145 ... yes
69/146 ... yes
18/147 ... yes
3/149 ... yes
2/151 ... yes
2/152 ... yes
43/155 ... yes
43/159 ... yes
14/160 ... yes
2/161 ... yes
111/162 ... yes
18/163 ... yes
51/164 ... yes
15/165 ... yes
119/166 ... yes
109/167 ... yes
26/168 ... yes
51/169 ... yes
5/170 ... yes
111/172 ... yes
111/173 ... yes
52/174 ... yes
59/175 ... yes
109/176 ... yes
51/177 ... yes
36/178 ... yes
43/179 ... yes
128/181 ... yes
69/182 ... yes
59/183 ... yes
133/184 ... yes
11/185 ... yes
61/187 ... yes
51/188 ... yes
59/189 ... yes
5/190 ... yes
5/191 ... yes
59/192 ... yes
51/193 ... yes
59/194 ... yes
36/195 ... yes
26/196 ... yes
43/197 ... yes
26/198 ... yes
59/199 ... yes
8/200 ... yes
59/201 ... yes
103/202 ... yes
91/203 ... yes
91/204 ... yes
59/205 ... yes
111/206 ... yes
88/207 ... yes
43/208 ... yes
119/209 ... yes
51/210 ... yes
51/211 ... yes
59/212 ... yes
74/213 ... yes
88/215 ... yes
51/217 ... yes
111/218 ... yes
138/219 ... yes
82/220 ... yes
109/221 ... yes
51/222 ... yes
51/223 ... yes
109/226 ... yes
36/227 ... yes
2/228 ... yes
5/229 ... yes
5/230 ... yes
3/231 ... yes
3/232 ... yes
3/233 ... yes
36/234 ... yes
7/235 ... yes
109/236 ... yes
30/237 ... yes
51/238 ... yes
40/239 ... yes
18/240 ... yes
43/242 ... yes
59/243 ... yes
91/244 ... yes
43/245 ... yes
42/246 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 111
Elasticsearch version 2.4.x? ... yes (2.4.2)
Elasticsearch has plugin delete-by-query installed? ... yes
Checking GitLab ... FinishedResults of GitLab environment info
I, [2016-12-09T18:59:08.112435 #3604] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:1.13.6
Rake Version: 10.5.0
Sidekiq Version:4.2.1
GitLab information
Version: 8.14.3-ee
Revision: 8282833
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.1
URL: https://<masked>
HTTP Clone URL: https://<masked>/some-group/some-project.git
SSH Clone URL: git@<masked>:some-group/some-project.git
Elasticsearch: no # note from issue author: this is weird, we actually have ES configured and it's working properly
Geo: no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: github
GitLab Shell
Version: 4.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/gitEdited by 🤖 GitLab Bot 🤖