Instance-wide variables not available in all pipelines
Summary
I was just setting up our GitLab (v13.1.1, updated yesterday) to use instance-wide variables instead of group variables within our pipelines. It seems there was an issue, perhaps with caching, as not all repos got the instance-level variables.
I worked in the following order:
- Set 4 instance-wide variables via curl. All are protected and masked.
- Check those variables by using curl to GET them afterwards. Everything looked right.
- Remove the group level variables (same names, also protected and masked)
- Start some jobs...
- A few tests I did worked well
- At least one repo did not seem to have the variable available
The group variables had been set with a script using the following curl-call for each group: curl --request POST --header "PRIVATE-TOKEN: <TOKEN>" "<URL>/api/v4/groups/<GROUP-NAME>/variables" --form "key=<KEY>" --form "value=<VALUE>" --form "masked=true"
I've removed all group variables using a Python script using python-gitlab. It iterated over each group and removed the variables.
Some data about our environment:
- GitLab instance is running on a dedicated server since multiple years
- 89 groups
- 1250 projects
- 214 users
- GitLab Community Edition
Project data where variables were not available
{ "id": 1290, "description": "Formerly know as oes", "name": "OrbatEditorServer2", "name_with_namespace": "SimulatedScenario / SimulatedEntities / OrbatEditorServer2", "path": "OrbatEditorServer2", "path_with_namespace": "SimulatedScenario/SimulatedEntities/OrbatEditorServer2", "created_at": "2018-08-21T06:20:39.948Z", "default_branch": "master", "tag_list": [], "ssh_url_to_repo": "...", "http_url_to_repo": "...", "web_url": "...", "readme_url": null, "avatar_url": null, "star_count": 1, "forks_count": 0, "last_activity_at": "2020-07-02T06:14:46.538Z", "namespace": { "id": 76, "name": "SimulatedEntities", "path": "SimulatedEntities", "kind": "group", "full_path": "SimulatedScenario/SimulatedEntities", "parent_id": 66, "avatar_url": null, "web_url": "..." }, "_links": { "self": "...", "issues": "...", "merge_requests": "...", "repo_branches": "...", "labels": "...", "events": "...", "members": "..." }, "empty_repo": false, "archived": false, "visibility": "internal", "resolve_outdated_diff_discussions": false, "container_registry_enabled": true, "issues_enabled": true, "merge_requests_enabled": true, "wiki_enabled": true, "jobs_enabled": true, "snippets_enabled": true, "can_create_merge_request_in": true, "issues_access_level": "enabled", "repository_access_level": "enabled", "merge_requests_access_level": "enabled", "forking_access_level": "enabled", "wiki_access_level": "enabled", "builds_access_level": "enabled", "snippets_access_level": "enabled", "pages_access_level": "public", "emails_disabled": null, "shared_runners_enabled": true, "lfs_enabled": true, "creator_id": 63, "import_status": "none", "import_error": null, "open_issues_count": 0, "runners_token": "TdUqEtmgfZHWKvgE79od", "ci_default_git_depth": null, "public_jobs": true, "build_git_strategy": "fetch", "build_timeout": 3600, "auto_cancel_pending_pipelines": "enabled", "build_coverage_regex": null, "ci_config_path": null, "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, "allow_merge_on_skipped_pipeline": null, "request_access_enabled": false, "only_allow_merge_if_all_discussions_are_resolved": false, "remove_source_branch_after_merge": null, "printing_merge_request_link_enabled": true, "merge_method": "merge", "suggestion_commit_message": null, "auto_devops_enabled": false, "auto_devops_deploy_strategy": "continuous", "autoclose_referenced_issues": true, "repository_storage": "default", "permissions": { "project_access": null, "group_access": { "access_level": 30, "notification_level": 3 } } }
Pipeline data of the failed pipeline
{ "id": 164618, "sha": "3d188341afb9e8ccf6f24bc8b4679b3d7d92f626", "ref": "master", "status": "failed", "created_at": "2020-07-02T06:27:48.924Z", "updated_at": "2020-07-02T06:31:03.293Z", "web_url": "...", "before_sha": "0000000000000000000000000000000000000000", "tag": false, "yaml_errors": null, "user": { "id": 159, "name": "...", "username": "...", "state": "active", "avatar_url": "...", "web_url": "..." }, "started_at": "2020-07-02T06:27:51.896Z", "finished_at": "2020-07-02T06:31:03.286Z", "committed_at": null, "duration": 17, "coverage": null, "detailed_status": { "icon": "status_failed", "text": "failed", "label": "failed", "group": "failed", "tooltip": "failed", "has_details": false, "details_path": "/SimulatedScenario/SimulatedEntities/OrbatEditorServer2/-/pipelines/164618", "illustration": null, "favicon": "/assets/ci_favicons/favicon_status_failed-41304d7f7e3828808b0c26771f0309e55296819a9beea3ea9fbf6689d9857c12.png" } }
Job data for a failing job. Multiple jobs failed within the pipeline though.
{ "id": 747403, "status": "failed", "stage": "prepare", "name": "check:Dependencies", "ref": "master", "tag": false, "coverage": null, "allow_failure": false, "created_at": "2020-07-02T06:30:43.259Z", "started_at": "2020-07-02T06:30:45.836Z", "finished_at": "2020-07-02T06:31:01.618Z", "duration": 15.781998, "user": { "id": 2, "name": "...", "username": "...", "state": "active", "avatar_url": "...", "web_url": "...", "created_at": "2016-08-29T12:18:53.375Z", "bio": "", "location": "...", "public_email": "...", "skype": "", "linkedin": "", "twitter": "", "website_url": "", "organization": "", "job_title": "", "work_information": null }, "commit": { "id": "3d188341afb9e8ccf6f24bc8b4679b3d7d92f626", "short_id": "3d188341", "created_at": "2020-07-02T08:25:51.000+02:00", "parent_ids": [ "2392f4b177b6c227433709ef629304677215b450" ], "title": "Fixed deps", "message": "Fixed deps\n", "author_name": "...", "author_email": "...", "authored_date": "2020-07-02T08:25:51.000+02:00", "committer_name": "...", "committer_email": "...", "committed_date": "2020-07-02T08:25:51.000+02:00", "web_url": "..." }, "pipeline": { "id": 164618, "sha": "3d188341afb9e8ccf6f24bc8b4679b3d7d92f626", "ref": "master", "status": "failed", "created_at": "2020-07-02T06:27:48.924Z", "updated_at": "2020-07-02T06:31:03.293Z", "web_url": "..." }, "web_url": "...", "artifacts_file": { "filename": "artifacts.zip", "size": 421 }, "artifacts": [ { "file_type": "archive", "size": 421, "filename": "artifacts.zip", "file_format": "zip" }, { "file_type": "metadata", "size": 185, "filename": "metadata.gz", "file_format": "gzip" }, { "file_type": "trace", "size": 5389, "filename": "job.log", "file_format": null } ], "runner": { "id": 125, "description": "docker-runner", "ip_address": "...", "active": true, "is_shared": true, "name": "gitlab-runner", "online": true, "status": "online" }, "artifacts_expire_at": "2020-07-16T06:31:00.728Z" }
This project worked, had all variables available
{ "id": 3, "description": "Simple interface to send data to Splat", "name": "EntityUpdateLib", "name_with_namespace": "SimulatedScenario / RadioSimulation / EntityUpdateLib", "path": "EntityUpdateLib", "path_with_namespace": "SimulatedScenario/RadioSimulation/EntityUpdateLib", "created_at": "2016-08-29T14:38:50.377Z", "default_branch": "master", "tag_list": [], "ssh_url_to_repo": "...", "http_url_to_repo": "...", "web_url": "...", "readme_url": "...", "avatar_url": "...", "star_count": 0, "forks_count": 0, "last_activity_at": "2020-07-02T06:12:30.109Z", "namespace": { "id": 93, "name": "RadioSimulation", "path": "RadioSimulation", "kind": "group", "full_path": "SimulatedScenario/RadioSimulation", "parent_id": 66, "avatar_url": null, "web_url": "..." }, "_links": { "self": "...", "issues": "..", "merge_requests": "...", "repo_branches": "...", "labels": "...", "events": "...", "members": "..." }, "empty_repo": false, "archived": false, "visibility": "internal", "resolve_outdated_diff_discussions": null, "container_registry_enabled": true, "issues_enabled": true, "merge_requests_enabled": true, "wiki_enabled": true, "jobs_enabled": true, "snippets_enabled": false, "can_create_merge_request_in": true, "issues_access_level": "enabled", "repository_access_level": "enabled", "merge_requests_access_level": "enabled", "forking_access_level": "enabled", "wiki_access_level": "enabled", "builds_access_level": "enabled", "snippets_access_level": "disabled", "pages_access_level": "public", "emails_disabled": null, "shared_runners_enabled": true, "lfs_enabled": true, "creator_id": 2, "import_status": "none", "import_error": null, "open_issues_count": 0, "runners_token": "EzA9pnz1Yf2LEd2FgwbC", "ci_default_git_depth": null, "public_jobs": true, "build_git_strategy": "fetch", "build_timeout": 3600, "auto_cancel_pending_pipelines": "enabled", "build_coverage_regex": null, "ci_config_path": null, "shared_with_groups": [ { "group_id": 128, "group_name": "VsTaskerPlugins", "group_full_path": "Team/.../VsTaskerPlugins", "group_access_level": 40, "expires_at": null } ], "only_allow_merge_if_pipeline_succeeds": false, "allow_merge_on_skipped_pipeline": null, "request_access_enabled": true, "only_allow_merge_if_all_discussions_are_resolved": null, "remove_source_branch_after_merge": null, "printing_merge_request_link_enabled": true, "merge_method": "merge", "suggestion_commit_message": null, "auto_devops_enabled": false, "auto_devops_deploy_strategy": "continuous", "autoclose_referenced_issues": true, "repository_storage": "default", "permissions": { "project_access": null, "group_access": { "access_level": 30, "notification_level": 3 } } }
Pipeline that passed
{ "id": 164620, "sha": "401facaf597260bfce2fbc3ce9c8d78ddeebb07d", "ref": "master", "status": "success", "created_at": "2020-07-02T06:32:34.139Z", "updated_at": "2020-07-02T06:34:15.984Z", "web_url": "...", "before_sha": "0000000000000000000000000000000000000000", "tag": false, "yaml_errors": null, "user": { "id": 2, "name": "...", "username": "...", "state": "active", "avatar_url": "...", "web_url": "..." }, "started_at": "2020-07-02T06:32:36.915Z", "finished_at": "2020-07-02T06:34:15.978Z", "committed_at": null, "duration": 96, "coverage": null, "detailed_status": { "icon": "status_success", "text": "passed", "label": "passed", "group": "success", "tooltip": "passed", "has_details": false, "details_path": "/SimulatedScenario/RadioSimulation/EntityUpdateLib/-/pipelines/164620", "illustration": null, "favicon": "/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png" } }
Job data that ran smoothly
{ "id": 747350, "status": "success", "stage": "prepare", "name": "check:Dependencies", "ref": "master", "tag": false, "coverage": null, "allow_failure": false, "created_at": "2020-07-02T06:23:12.761Z", "started_at": "2020-07-02T06:23:14.858Z", "finished_at": "2020-07-02T06:23:33.892Z", "duration": 19.034286, "user": { "id": 2, "name": "...", "username": "...", "state": "active", "avatar_url": "...", "web_url": "...", "created_at": "2016-08-29T12:18:53.375Z", "bio": "", "location": "...", "public_email": "...", "skype": "", "linkedin": "", "twitter": "", "website_url": "", "organization": "", "job_title": "", "work_information": null }, "commit": { "id": "401facaf597260bfce2fbc3ce9c8d78ddeebb07d", "short_id": "401facaf", "created_at": "2020-07-02T08:12:29.000+02:00", "parent_ids": [ "49c32ff49daa122d4a2e7568ef6fdbc87f7e666c" ], "title": "Changed: Small change for testing purposes", "message": "Changed: Small change for testing purposes", "author_name": "...", "author_email": "...", "authored_date": "2020-07-02T08:12:29.000+02:00", "committer_name": "...", "committer_email": "...", "committed_date": "2020-07-02T08:12:29.000+02:00", "web_url": "..." }, "pipeline": { "id": 164615, "sha": "401facaf597260bfce2fbc3ce9c8d78ddeebb07d", "ref": "master", "status": "success", "created_at": "2020-07-02T06:23:12.750Z", "updated_at": "2020-07-02T06:24:56.460Z", "web_url": "..." }, "web_url": "...", "artifacts_file": { "filename": "artifacts.zip", "size": 685 }, "artifacts": [ { "file_type": "archive", "size": 685, "filename": "artifacts.zip", "file_format": "zip" }, { "file_type": "metadata", "size": 210, "filename": "metadata.gz", "file_format": "gzip" }, { "file_type": "trace", "size": 5517, "filename": "job.log", "file_format": null } ], "runner": { "id": 125, "description": "docker-runner", "ip_address": "...", "active": true, "is_shared": true, "name": "gitlab-runner", "online": true, "status": "online" }, "artifacts_expire_at": "2020-07-16T06:23:32.944Z" }
As discussed here with @reprazent and @mbobin.
Steps to reproduce
Not able to create a repro of this yet, I've added as much info about our environment as I could within the summary.
Example Project
Not possible to create an example on GitLab.com, as this is depending on a local GitLab instance.
What is the current bug behavior?
Some pipelines seem not to have the variables available that were created via instance-wide variables. Other jobs had them available. All jobs had the same attributes:
- Running on our local docker-runner
- Same Docker image being used for the jobs
- Same
.gitlab-ci.yml
content, including their jobs and stages from a central repo - All groups (main and subgroups) had their variables removed, I double-checked
Some differences I found so far:
- Projects were in different groups
- Projects had different "content", as in containing different sourcecode, different age etc
- The failing project had no locked branches
- The working project has a locked master branch
What is the expected correct behavior?
All pipelines have the instance-wide variables available.
Relevant logs and/or screenshots
Just the error message from our build system that states, that there is no such variable available.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.27.0 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 13.1.1 Revision: bdb9883705a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: ... HTTP Clone URL: ... SSH Clone URL: ... Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 13.3.0 Repository storage paths:
- default: /repo/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 94/1 ... yes 94/2 ... yes 93/3 ... yes 6/4 ... yes 35/5 ... yes 94/9 ... yes 35/11 ... yes 11/12 ... yes 85/16 ... yes 85/17 ... yes 85/20 ... yes 85/21 ... yes 85/22 ... yes 85/23 ... yes 85/24 ... yes 85/25 ... yes 85/26 ... yes 85/27 ... yes 85/28 ... yes 85/29 ... yes 85/30 ... yes 85/31 ... yes 85/32 ... yes 85/33 ... yes 85/34 ... yes 85/35 ... yes 85/36 ... yes 85/37 ... yes 35/38 ... yes 85/39 ... yes 35/40 ... yes 11/41 ... yes 11/42 ... yes 346/43 ... yes 11/44 ... yes 346/45 ... yes 346/46 ... yes 346/47 ... yes 35/48 ... yes 85/49 ... yes 85/50 ... yes 85/51 ... yes 85/52 ... yes 76/53 ... yes 85/54 ... yes 11/55 ... yes 11/56 ... yes 85/57 ... yes 85/58 ... yes 85/59 ... yes 85/60 ... yes 85/61 ... yes 85/62 ... yes 85/63 ... yes 85/64 ... yes 85/65 ... yes 85/66 ... yes 85/67 ... yes 85/68 ... yes 85/69 ... yes 85/70 ... yes 85/71 ... yes 85/72 ... yes 85/73 ... yes 85/74 ... yes 85/75 ... yes 85/76 ... yes 85/77 ... yes 85/78 ... yes 85/79 ... yes 85/80 ... yes 85/81 ... yes 85/82 ... yes 85/83 ... yes 85/84 ... yes 85/85 ... yes 85/86 ... yes 85/87 ... yes 85/88 ... yes 85/89 ... yes 85/90 ... yes 85/91 ... yes 85/92 ... yes 85/93 ... yes 85/94 ... yes 85/95 ... yes 85/96 ... yes 85/97 ... yes 85/98 ... yes 85/99 ... yes 85/100 ... yes 85/101 ... yes 85/102 ... yes 85/103 ... yes 85/104 ... yes 85/105 ... yes 85/106 ... yes 85/107 ... yes 76/108 ... yes 85/109 ... yes 85/110 ... yes 76/111 ... yes 85/112 ... yes 76/113 ... yes 76/114 ... yes 85/115 ... yes 76/116 ... yes 76/117 ... yes 85/118 ... yes 76/119 ... yes 76/120 ... yes 76/121 ... yes 85/122 ... yes 85/123 ... yes 76/124 ... yes 76/125 ... yes 4/126 ... yes 6/127 ... yes 85/128 ... yes 85/129 ... yes 85/130 ... yes 85/131 ... yes 11/132 ... yes 11/133 ... yes 85/134 ... yes 93/135 ... yes 76/136 ... yes 93/137 ... yes 77/138 ... yes 11/139 ... yes 11/140 ... yes 6/141 ... yes 76/142 ... yes 11/143 ... yes 11/144 ... yes 11/145 ... yes 11/146 ... yes 11/147 ... yes 11/148 ... yes 11/149 ... yes 77/150 ... yes 76/151 ... yes 11/152 ... yes 76/153 ... yes 11/154 ... yes 11/155 ... yes 11/156 ... yes 11/157 ... yes 11/158 ... yes 94/159 ... yes 93/160 ... yes 11/161 ... yes 76/162 ... yes 76/163 ... yes 76/164 ... yes 29/165 ... yes 31/167 ... yes 31/168 ... yes 31/169 ... yes 31/170 ... yes 11/172 ... yes 11/173 ... yes 85/174 ... yes 11/176 ... yes 102/177 ... yes 6/178 ... yes 31/179 ... yes 34/186 ... yes 6/203 ... yes 85/204 ... yes 29/205 ... yes 35/207 ... yes 35/208 ... yes 35/209 ... yes 35/210 ... yes 11/212 ... yes 11/213 ... yes 11/215 ... yes 11/216 ... yes 36/217 ... yes 11/221 ... yes 35/222 ... yes 90/231 ... yes 276/234 ... yes 220/235 ... yes 276/236 ... yes 11/237 ... yes 29/240 ... yes 29/241 ... yes 29/242 ... yes 90/245 ... yes 29/246 ... yes 14/247 ... yes 36/293 ... yes 36/294 ... yes 220/295 ... yes 93/296 ... yes 93/297 ... yes 93/298 ... yes 93/299 ... yes 93/300 ... yes 93/301 ... yes 35/302 ... yes 85/303 ... yes 36/304 ... yes 34/307 ... yes 36/308 ... yes 36/309 ... yes 36/310 ... yes 36/311 ... yes 35/312 ... yes 35/313 ... yes 36/314 ... yes 36/315 ... yes 77/316 ... yes 11/317 ... yes 11/318 ... yes 29/320 ... yes 77/322 ... yes 29/344 ... yes 11/345 ... yes 11/365 ... yes 133/368 ... yes 133/369 ... yes 96/370 ... yes 11/371 ... yes 29/372 ... yes 51/373 ... yes 11/379 ... yes 11/380 ... yes 93/381 ... yes 36/382 ... yes 109/384 ... yes 85/386 ... yes 35/388 ... yes 35/389 ... yes 11/390 ... yes 76/392 ... yes 35/393 ... yes 54/394 ... yes 35/395 ... yes 35/396 ... yes 35/397 ... yes 35/398 ... yes 35/399 ... yes 62/401 ... yes 62/402 ... yes 62/403 ... yes 35/404 ... yes 35/405 ... yes 35/406 ... yes 35/408 ... yes 85/409 ... yes 35/410 ... yes 63/413 ... yes 48/414 ... yes 11/415 ... yes 11/416 ... yes 35/417 ... yes 35/418 ... yes 35/419 ... yes 35/420 ... yes 137/422 ... yes 35/423 ... yes 35/424 ... yes 35/425 ... yes 35/426 ... yes 35/427 ... yes 76/428 ... yes 35/434 ... yes 11/437 ... yes 11/438 ... yes 35/440 ... yes 67/441 ... yes 35/442 ... yes 35/443 ... yes 67/444 ... yes 35/450 ... yes 67/452 ... yes 85/453 ... yes 76/460 ... yes 67/461 ... yes 85/462 ... yes 76/508 ... yes 35/512 ... yes 31/524 ... yes 11/569 ... yes 35/587 ... yes 11/589 ... yes 11/640 ... yes 11/641 ... yes 67/650 ... yes 72/651 ... yes 11/652 ... yes 90/653 ... yes 11/654 ... yes 35/655 ... yes 11/656 ... yes 35/657 ... yes 76/658 ... yes 35/659 ... yes 35/660 ... yes 77/665 ... yes 11/666 ... yes 76/668 ... yes 11/669 ... yes 11/670 ... yes 11/671 ... yes 11/672 ... yes 11/673 ... yes 11/674 ... yes 85/675 ... yes 85/689 ... yes 80/691 ... yes 80/692 ... yes 81/694 ... yes 80/695 ... yes 11/696 ... yes 35/715 ... yes 35/716 ... yes 85/717 ... yes 35/718 ... yes 6/719 ... yes 85/720 ... yes 85/721 ... yes 85/722 ... yes 85/723 ... yes 96/727 ... yes 4/735 ... yes 85/736 ... yes 85/737 ... yes 107/738 ... yes 35/739 ... yes 35/741 ... yes 35/742 ... yes 109/743 ... yes 6/744 ... yes 101/745 ... yes 76/746 ... yes 76/747 ... yes 85/750 ... yes 77/751 ... yes 67/752 ... yes 67/753 ... yes 80/755 ... yes 11/757 ... yes 35/759 ... yes 35/760 ... yes 35/761 ... yes 35/762 ... yes 77/763 ... yes 35/769 ... yes 93/770 ... yes 11/771 ... yes 35/772 ... yes 35/774 ... yes 35/775 ... yes 11/776 ... yes 35/777 ... yes 35/778 ... yes 35/779 ... yes 76/780 ... yes 76/781 ... yes 35/782 ... yes 54/783 ... yes 35/784 ... yes 92/785 ... yes 85/786 ... yes 35/787 ... yes 35/788 ... yes 35/789 ... yes 35/790 ... yes 35/791 ... yes 35/792 ... yes 35/793 ... yes 11/794 ... yes 76/795 ... yes 77/796 ... yes 35/797 ... yes 29/798 ... yes 35/799 ... yes 76/802 ... yes 108/803 ... yes 29/804 ... yes 121/805 ... yes 121/806 ... yes 121/807 ... yes 121/808 ... yes 121/809 ... yes 81/813 ... yes 29/814 ... yes 108/815 ... yes 108/816 ... yes 108/817 ... yes 108/818 ... yes 108/819 ... yes 7/820 ... yes 108/821 ... yes 108/825 ... yes 29/831 ... yes 93/834 ... yes 101/838 ... yes 35/842 ... yes 85/844 ... yes 11/847 ... yes 141/849 ... yes 108/850 ... yes 93/852 ... yes 11/853 ... yes 93/854 ... yes 35/855 ... yes 93/857 ... yes 36/859 ... yes 4/861 ... yes 7/862 ... yes 76/865 ... yes 11/866 ... yes 11/868 ... yes 11/869 ... yes 67/872 ... yes 11/873 ... yes 11/874 ... yes 11/875 ... yes 11/876 ... yes 11/877 ... yes 11/878 ... yes 11/879 ... yes 11/880 ... yes 11/881 ... yes 11/882 ... yes 4/883 ... yes 6/884 ... yes 4/888 ... yes 6/889 ... yes 6/890 ... yes 6/891 ... yes 81/893 ... yes 11/897 ... yes 6/898 ... yes 6/899 ... yes 96/903 ... yes 96/904 ... yes 11/905 ... yes 102/906 ... yes 150/907 ... yes 76/908 ... yes 76/909 ... yes 76/910 ... yes 91/911 ... yes 76/912 ... yes 7/913 ... yes 77/914 ... yes 77/915 ... yes 67/916 ... yes 35/917 ... yes 76/918 ... yes 76/919 ... yes 76/920 ... yes 76/921 ... yes 76/922 ... yes 11/923 ... yes 11/924 ... yes 11/925 ... yes 11/926 ... yes 11/927 ... yes 11/928 ... yes 11/929 ... yes 35/930 ... yes 76/931 ... yes 11/932 ... yes 90/933 ... yes 11/935 ... yes 76/936 ... yes 85/938 ... yes 160/939 ... yes 160/940 ... yes 160/941 ... yes 7/942 ... yes 77/943 ... yes 35/944 ... yes 85/945 ... yes 85/946 ... yes 96/948 ... yes 36/949 ... yes 108/950 ... yes 6/951 ... yes 76/952 ... yes 76/954 ... yes 85/955 ... yes 85/956 ... yes 11/957 ... yes 85/958 ... yes 106/959 ... yes 11/960 ... yes 76/961 ... yes 76/962 ... yes 76/963 ... yes 4/964 ... yes 11/965 ... yes 35/966 ... yes 108/967 ... yes 76/969 ... yes 135/970 ... yes 6/971 ... yes 276/977 ... yes 276/978 ... yes 276/979 ... yes 166/983 ... yes 76/993 ... yes 76/994 ... yes 76/995 ... yes 85/996 ... yes 11/997 ... yes 84/998 ... yes 167/1000 ... yes 167/1001 ... yes 167/1002 ... yes 85/1004 ... yes 167/1005 ... yes 133/1010 ... yes 76/1011 ... yes 35/1012 ... yes 76/1013 ... yes 6/1014 ... yes 141/1017 ... yes 6/1018 ... yes 6/1019 ... yes 76/1020 ... yes 80/1021 ... yes 80/1022 ... yes 85/1023 ... yes 76/1025 ... yes 11/1026 ... yes 76/1027 ... yes 80/1028 ... yes 35/1029 ... yes 35/1030 ... yes 85/1036 ... yes 63/1037 ... yes 11/1038 ... yes 76/1039 ... yes 141/1050 ... yes 176/1051 ... yes 176/1052 ... yes 176/1053 ... yes 176/1054 ... yes 176/1055 ... yes 176/1056 ... yes 76/1057 ... yes 76/1058 ... yes 76/1059 ... yes 29/1060 ... yes 35/1061 ... yes 87/1062 ... yes 76/1063 ... yes 76/1064 ... yes 76/1065 ... yes 76/1066 ... yes 76/1067 ... yes 76/1068 ... yes 76/1069 ... yes 76/1070 ... yes 35/1072 ... yes 155/1073 ... yes 176/1074 ... yes 85/1077 ... yes 29/1078 ... yes 89/1080 ... yes 89/1081 ... yes 89/1087 ... yes 80/1088 ... yes 89/1090 ... yes 35/1091 ... yes 35/1093 ... yes 35/1094 ... yes 35/1095 ... yes 133/1096 ... yes 133/1097 ... yes 133/1098 ... yes 92/1099 ... yes 35/1100 ... yes 141/1101 ... yes 67/1103 ... yes 67/1104 ... yes 141/1105 ... yes 11/1107 ... yes 77/1108 ... yes 11/1109 ... yes 11/1110 ... yes 91/1111 ... yes 93/1112 ... yes 154/1113 ... yes 179/1115 ... yes 11/1116 ... yes 11/1117 ... yes 11/1118 ... yes 11/1119 ... yes 11/1120 ... yes 35/1122 ... yes 76/1130 ... yes 67/1131 ... yes 8/1132 ... yes 166/1133 ... yes 96/1134 ... yes 77/1136 ... yes 67/1137 ... yes 77/1138 ... yes 76/1140 ... yes 76/1141 ... yes 91/1142 ... yes 76/1143 ... yes 35/1144 ... yes 96/1145 ... yes 99/1147 ... yes 67/1148 ... yes 89/1149 ... yes 35/1150 ... yes 76/1151 ... yes 76/1152 ... yes 77/1153 ... yes 96/1154 ... yes 35/1155 ... yes 91/1156 ... yes 77/1157 ... yes 108/1158 ... yes 96/1159 ... yes 35/1164 ... yes 97/1165 ... yes 150/1166 ... yes 76/1167 ... yes 11/1168 ... yes 67/1169 ... yes 35/1170 ... yes 108/1171 ... yes 92/1172 ... yes 184/1173 ... yes 77/1174 ... yes 72/1175 ... yes 90/1177 ... yes 108/1178 ... yes 6/1179 ... yes 67/1180 ... yes 77/1181 ... yes 76/1182 ... yes 35/1186 ... yes 35/1187 ... yes 35/1188 ... yes 35/1189 ... yes 11/1190 ... yes 98/1191 ... yes 98/1192 ... yes 35/1193 ... yes 91/1194 ... yes 98/1195 ... yes 76/1196 ... yes 95/1197 ... yes 89/1198 ... yes 89/1199 ... yes 35/1200 ... yes 88/1201 ... yes 67/1202 ... yes 96/1203 ... yes 96/1204 ... yes 96/1205 ... yes 77/1206 ... yes 76/1207 ... yes 77/1208 ... yes 77/1209 ... yes 91/1210 ... yes 76/1211 ... yes 35/1212 ... yes 35/1213 ... yes 93/1214 ... yes 67/1215 ... yes 67/1216 ... yes 76/1217 ... yes 76/1218 ... yes 77/1219 ... yes 35/1220 ... yes 93/1221 ... yes 35/1222 ... yes 96/1223 ... yes 35/1224 ... yes 35/1225 ... yes 107/1226 ... yes 11/1229 ... yes 88/1230 ... yes 35/1232 ... yes 35/1233 ... yes 35/1234 ... yes 35/1235 ... yes 220/1236 ... yes 97/1237 ... yes 77/1238 ... yes 76/1239 ... yes 89/1240 ... yes 96/1241 ... yes 93/1242 ... yes 96/1243 ... yes 76/1244 ... yes 94/1245 ... yes 77/1246 ... yes 93/1247 ... yes 89/1248 ... yes 96/1249 ... yes 77/1250 ... yes 76/1251 ... yes 96/1252 ... yes 76/1253 ... yes 96/1254 ... yes 96/1255 ... yes 93/1256 ... yes 35/1257 ... yes 76/1258 ... yes 35/1259 ... yes 35/1260 ... yes 11/1261 ... yes 66/1262 ... yes 96/1263 ... yes 76/1264 ... yes 35/1266 ... yes 35/1268 ... yes 35/1269 ... yes 96/1270 ... yes 97/1271 ... yes 77/1273 ... yes 35/1274 ... yes 98/1275 ... yes 35/1277 ... yes 77/1278 ... yes 96/1279 ... yes 76/1280 ... yes 96/1281 ... yes 96/1282 ... yes 108/1283 ... yes 96/1284 ... yes 91/1285 ... yes 35/1286 ... yes 29/1287 ... yes 7/1288 ... yes 92/1289 ... yes 76/1290 ... yes 85/1291 ... yes 76/1292 ... yes 191/1298 ... yes 191/1301 ... yes 11/1302 ... yes 130/1303 ... yes 204/1304 ... yes 76/1305 ... yes 11/1306 ... yes 11/1307 ... yes 194/1308 ... yes 85/1309 ... yes 29/1311 ... yes 96/1312 ... yes 85/1316 ... yes 198/1317 ... yes 85/1318 ... yes 11/1319 ... yes 11/1320 ... yes 7/1321 ... yes 7/1322 ... yes 199/1323 ... yes 36/1324 ... yes 76/1325 ... yes 76/1326 ... yes 35/1327 ... yes 91/1328 ... yes 88/1329 ... yes 77/1330 ... yes 96/1331 ... yes 77/1332 ... yes 67/1333 ... yes 76/1334 ... yes 88/1335 ... yes 89/1336 ... yes 35/1337 ... yes 91/1338 ... yes 77/1339 ... yes 96/1340 ... yes 93/1341 ... yes 96/1342 ... yes 81/1343 ... yes 89/1344 ... yes 97/1345 ... yes 89/1346 ... yes 67/1347 ... yes 96/1348 ... yes 89/1349 ... yes 76/1350 ... yes 96/1351 ... yes 91/1352 ... yes 76/1353 ... yes 97/1354 ... yes 89/1355 ... yes 76/1357 ... yes 96/1358 ... yes 96/1359 ... yes 76/1360 ... yes 35/1361 ... yes 67/1362 ... yes 35/1363 ... yes 35/1364 ... yes 35/1365 ... yes 35/1366 ... yes 35/1367 ... yes 35/1368 ... yes 35/1369 ... yes 35/1370 ... yes 96/1371 ... yes 96/1372 ... yes 96/1373 ... yes 96/1374 ... yes 35/1377 ... yes 35/1378 ... yes 35/1379 ... yes 166/1380 ... yes 35/1381 ... yes 35/1382 ... yes 35/1383 ... yes 35/1384 ... yes 35/1385 ... yes 35/1386 ... yes 35/1387 ... yes 35/1388 ... yes 76/1389 ... yes 76/1390 ... yes 76/1391 ... yes 29/1392 ... yes 204/1393 ... yes 97/1394 ... yes 35/1395 ... yes 35/1396 ... yes 35/1397 ... yes 35/1398 ... yes 67/1399 ... yes 35/1400 ... yes 35/1401 ... yes 96/1402 ... yes 35/1403 ... yes 35/1404 ... yes 35/1405 ... yes 77/1406 ... yes 35/1407 ... yes 35/1408 ... yes 35/1409 ... yes 35/1410 ... yes 35/1411 ... yes 76/1412 ... yes 35/1413 ... yes 35/1414 ... yes 35/1415 ... yes 35/1416 ... yes 97/1417 ... yes 77/1418 ... yes 35/1419 ... yes 35/1420 ... yes 35/1421 ... yes 35/1422 ... yes 89/1423 ... yes 97/1424 ... yes 11/1426 ... yes 35/1427 ... yes 85/1429 ... yes 67/1430 ... yes 98/1436 ... yes 98/1437 ... yes 98/1438 ... yes 96/1439 ... yes 35/1440 ... yes 76/1441 ... yes 76/1442 ... yes 35/1443 ... yes 85/1446 ... yes 92/1447 ... yes 93/1448 ... yes 93/1449 ... yes 93/1450 ... yes 93/1451 ... yes 93/1452 ... yes 93/1453 ... yes 93/1454 ... yes 141/1455 ... yes 76/1456 ... yes 35/1457 ... yes 67/1458 ... yes 35/1459 ... yes 35/1460 ... yes 35/1461 ... yes 35/1462 ... yes 76/1463 ... yes 76/1464 ... yes 11/1465 ... yes 35/1466 ... yes 76/1467 ... yes 76/1468 ... yes 130/1470 ... yes 96/1471 ... yes 76/1472 ... yes 85/1473 ... yes 76/1474 ... yes 93/1475 ... yes 35/1476 ... yes 35/1477 ... yes 93/1478 ... yes 67/1479 ... yes 76/1480 ... yes 76/1481 ... yes 76/1482 ... yes 81/1483 ... yes 85/1484 ... yes 6/1485 ... yes 7/1486 ... yes 77/1487 ... yes 85/1488 ... yes 85/1489 ... yes 11/1490 ... yes 150/1491 ... yes 76/1492 ... yes 35/1493 ... yes 89/1494 ... yes 35/1495 ... yes 89/1496 ... yes 85/1498 ... yes 216/1500 ... yes 216/1501 ... yes 217/1502 ... yes 217/1503 ... yes 217/1504 ... yes 6/1505 ... yes 76/1506 ... yes 76/1507 ... yes 76/1508 ... yes 76/1509 ... yes 122/1511 ... yes 93/1512 ... yes 220/1513 ... yes 220/1514 ... yes 99/1522 ... yes 89/1523 ... yes 89/1524 ... yes 85/1526 ... yes 225/1527 ... yes 225/1529 ... yes 11/1530 ... yes 76/1531 ... yes 11/1532 ... yes 11/1533 ... yes 11/1534 ... yes 85/1535 ... yes 92/1536 ... yes 76/1537 ... yes 85/1538 ... yes 6/1539 ... yes 76/1541 ... yes 76/1542 ... yes 76/1543 ... yes 87/1544 ... yes 6/1545 ... yes 6/1546 ... yes 85/1547 ... yes 85/1548 ... yes 6/1549 ... yes 35/1550 ... yes 87/1551 ... yes 11/1552 ... yes 96/1553 ... yes 89/1555 ... yes 6/1556 ... yes 76/1559 ... yes 255/1560 ... yes 76/1561 ... yes 35/1562 ... yes 29/1563 ... yes 29/1564 ... yes 29/1565 ... yes 11/1566 ... yes 76/1567 ... yes 11/1568 ... yes 76/1570 ... yes 76/1571 ... yes 157/1572 ... yes 257/1573 ... yes 220/1574 ... yes 31/1575 ... yes 76/1576 ... yes 85/1580 ... yes 6/1581 ... yes 35/1582 ... yes 35/1583 ... yes 11/1584 ... yes 91/1585 ... yes 89/1586 ... yes 249/1587 ... yes 85/1588 ... yes 204/1589 ... yes 225/1590 ... yes 11/1592 ... yes 11/1593 ... yes 249/1594 ... yes 89/1595 ... yes 11/1596 ... yes 89/1597 ... yes 255/1598 ... yes 256/1599 ... yes 36/1601 ... yes 204/1602 ... yes 85/1603 ... yes 80/1604 ... yes 76/1605 ... yes 76/1606 ... yes 84/1607 ... yes 85/1608 ... yes 87/1609 ... yes 76/1610 ... yes 76/1611 ... yes 87/1612 ... yes 76/1613 ... yes 276/1614 ... yes 264/1615 ... yes 76/1617 ... yes 96/1618 ... yes 96/1620 ... yes 96/1621 ... yes 76/1622 ... yes 257/1623 ... yes 77/1624 ... yes 257/1625 ... yes 96/1626 ... yes 94/1627 ... yes 89/1628 ... yes 257/1629 ... yes 190/1632 ... yes 35/1634 ... yes 35/1635 ... yes 2/1636 ... yes 76/1637 ... yes 255/1638 ... yes 255/1639 ... yes 85/1642 ... yes 85/1643 ... yes 265/1645 ... yes 265/1647 ... yes 81/1648 ... yes 225/1649 ... yes 85/1650 ... yes 284/1652 ... yes 284/1653 ... yes 284/1654 ... yes 284/1655 ... yes 284/1656 ... yes 66/1657 ... yes 285/1658 ... yes 284/1659 ... yes 225/1660 ... yes 76/1661 ... yes 35/1662 ... yes 81/1663 ... yes 220/1664 ... yes 220/1665 ... yes 220/1666 ... yes 220/1667 ... yes 81/1668 ... yes 66/1669 ... yes 249/1670 ... yes 35/1671 ... yes 249/1672 ... yes 249/1673 ... yes 81/1674 ... yes 220/1675 ... yes 220/1676 ... yes 288/1677 ... yes 31/1678 ... yes 31/1680 ... yes 282/1681 ... yes 31/1682 ... yes 257/1683 ... yes 76/1685 ... yes 31/1686 ... yes 256/1687 ... yes 256/1688 ... yes 67/1689 ... yes 256/1690 ... yes 76/1693 ... yes 257/1694 ... yes 249/1695 ... yes 36/1696 ... yes 11/1697 ... yes 6/1698 ... yes 257/1699 ... yes 11/1700 ... yes 76/1701 ... yes 76/1702 ... yes 265/1703 ... yes 220/1704 ... yes 108/1705 ... yes 211/1706 ... yes 108/1707 ... yes 35/1709 ... yes 76/1710 ... yes 90/1712 ... yes 72/1713 ... yes 72/1714 ... yes 76/1715 ... yes 76/1717 ... yes 341/1718 ... yes 284/1719 ... yes 85/1720 ... yes 256/1721 ... yes 256/1722 ... yes 256/1723 ... yes 211/1725 ... yes 211/1726 ... yes 108/1728 ... yes 76/1729 ... yes 35/1734 ... yes 108/1736 ... yes 249/1737 ... yes 256/1738 ... yes 85/1739 ... yes 249/1740 ... yes 276/1743 ... yes 256/1744 ... yes 220/1745 ... yes 35/1746 ... yes 256/1747 ... yes 249/1748 ... yes 76/1750 ... yes 308/1751 ... yes 76/1752 ... yes 229/1753 ... yes 309/1754 ... yes 288/1755 ... yes 229/1756 ... yes 220/1757 ... yes 229/1758 ... yes 76/1761 ... yes 66/1762 ... yes 66/1763 ... yes 66/1764 ... yes 308/1765 ... yes 108/1767 ... yes 35/1768 ... yes 66/1769 ... yes 229/1770 ... yes 93/1771 ... yes 249/1772 ... yes 36/1773 ... yes 97/1774 ... yes 89/1775 ... yes 256/1776 ... yes 256/1777 ... yes 35/1778 ... yes 263/1779 ... yes 149/1780 ... yes 257/1781 ... yes 157/1782 ... yes 265/1783 ... yes 157/1784 ... yes 76/1787 ... yes 157/1789 ... yes 76/1790 ... yes 284/1791 ... yes 256/1792 ... yes 11/1793 ... yes 11/1795 ... yes 249/1796 ... yes 11/1798 ... yes 11/1799 ... yes 11/1800 ... yes 11/1801 ... yes 11/1802 ... yes 11/1803 ... yes 256/1804 ... yes 11/1806 ... yes 308/1807 ... yes 257/1808 ... yes 284/1809 ... yes 11/1810 ... yes 11/1811 ... yes 256/1812 ... yes 11/1814 ... yes 288/1815 ... yes 249/1816 ... yes 11/1817 ... yes 11/1818 ... yes 308/1819 ... yes 11/1820 ... yes 249/1821 ... yes 256/1822 ... yes 76/1824 ... yes 341/1825 ... yes 76/1826 ... yes 11/1827 ... yes 76/1828 ... yes 225/1829 ... yes 66/1830 ... yes 265/1831 ... yes 35/1832 ... yes 225/1833 ... yes 225/1834 ... yes 76/1835 ... yes 76/1836 ... yes 35/1837 ... yes 93/1838 ... yes 36/1839 ... yes 11/1840 ... yes 85/1841 ... yes 345/1844 ... yes 11/1848 ... yes 11/1849 ... yes 227/1850 ... yes 220/1851 ... yes 85/1852 ... yes 249/1853 ... yes 309/1854 ... yes 229/1855 ... yes 257/1856 ... yes 343/1857 ... yes 345/1859 ... yes 345/1860 ... yes 345/1861 ... yes 343/1862 ... yes 345/1863 ... yes 76/1865 ... yes 76/1866 ... yes 308/1867 ... yes 76/1868 ... yes 110/1869 ... yes 256/1871 ... yes 256/1872 ... yes 220/1873 ... yes 157/1874 ... yes 256/1876 ... yes 345/1877 ... yes 96/1878 ... yes 90/1882 ... yes 249/1883 ... yes 76/1884 ... yes 76/1885 ... yes 249/1886 ... yes 345/1887 ... yes 184/1888 ... yes 80/1889 ... yes 76/1890 ... yes 6/1891 ... yes 249/1892 ... yes 334/1893 ... yes 249/1894 ... yes 93/1895 ... yes 2/1896 ... yes 76/1897 ... yes 257/1898 ... yes 309/1899 ... yes 357/1900 ... yes 76/1901 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.27.0) Git user has default SSH configuration? ... yes Active users: ... 214 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished