Route not loaded when using `find_by_full_path` making it impossible to update any resources
Summary
When trying to update any user/group/project we get the following error Route can't be blank
. This is reliably the same on every single resource that implements the routable interface.
This currently prevents any form of updating any projects/ci.
This was caused after upgrading to 13.1 OR restoring a backup (within a day or 2 so it's hard to tell the exact cause)
Any advice to help fix this with data or any steps I can take to help isolate the issue further would be gladly welcomed.
Steps to reproduce
This is reliably reproduced for every single user/project/namespace but only with my current psql db. This doesn't appear with a new fresh install. So this leads me to a data issue or bug that is trigger by some configuration.
I exported the psql to an entirely new gitlab setup (not a full backup) and the issue remained.
The issue only appears when using the UI. Updating a project on the rails console
works. I isolated it to the find_by_full_path
method from the routable
concern which doesn't appear to load the dependant route.
I have tried to isolate it as much as possible and I will try to document those here.
working cases
# where syntax
p = Project.where(path: 'only-except-upgrade').first
=> #<Project id:603 david/only-except-upgrade>>
irb(main):005:0> p.route
=> #<Route id: 731, source_id: 603, source_type: "Project", path: "david/only-except-upgrade", created_at: "2020-05-04 14:57:44", updated_at: "2020-05-04 14:57:44", name: "David Rubin / only-except-upgrade">
# find syntax
p = Project.find(603)
=> #<Project id:603 david/only-except-upgrade>>
irb(main):012:0> p.route
=> #<Route id: 731, source_id: 603, source_type: "Project", path: "david/only-except-upgrade", created_at: "2020-05-04 14:57:44", updated_at: "2020-05-04 14:57:44", name: "David Rubin / only-except-upgrade">
# Find + includes to load dependant object
p = Project.includes(:route).find(603)
=> #<Project id:603 david/only-except-upgrade>>
irb(main):017:0> p.route
=> #<Route id: 731, source_id: 603, source_type: "Project", path: "david/only-except-upgrade", created_at: "2020-05-04 14:57:44", updated_at: "2020-05-04 14:57:44", name: "David Rubin / only-except-upgrade">
failure case
p = Project.find_by_full_path('david/only-except-upgrade')
=> #<Project id:603 david/only-except-upgrade>>
irb(main):019:0> p.route
=> nil
This reliably fails for every Project/group / User or anything that implements the routable module.
Example Project
na
What is the current bug behavior?
Can't update any projects/users/data.
What is the expected correct behavior?
Be able to update projects
Relevant logs and/or screenshots
** UI Error on the frontend **
Since this is a validation error its a client error 4xx
and thus doesn't generate any error logs
Sql for p = Project.find_by_full_path('david/only-except-upgrade')
.
SELECT
"projects"."id" AS t0_r0,
"projects"."name" AS t0_r1,
"projects"."path" AS t0_r2,
"projects"."description" AS t0_r3,
"projects"."created_at" AS t0_r4,
"projects"."updated_at" AS t0_r5,
"projects"."creator_id" AS t0_r6,
"projects"."namespace_id" AS t0_r7,
"projects"."last_activity_at" AS t0_r8,
"projects"."import_url" AS t0_r9,
"projects"."visibility_level" AS t0_r10,
"projects"."archived" AS t0_r11,
"projects"."avatar" AS t0_r12,
"projects"."star_count" AS t0_r13,
"projects"."import_type" AS t0_r14,
"projects"."import_source" AS t0_r15,
"projects"."shared_runners_enabled" AS t0_r16,
"projects"."runners_token" AS t0_r17,
"projects"."build_coverage_regex" AS t0_r18,
"projects"."build_allow_git_fetch" AS t0_r19,
"projects"."build_timeout" AS t0_r20,
"projects"."pending_delete" AS t0_r21,
"projects"."public_builds" AS t0_r22,
"projects"."last_repository_check_failed" AS t0_r23,
"projects"."last_repository_check_at" AS t0_r24,
"projects"."container_registry_enabled" AS t0_r25,
"projects"."only_allow_merge_if_pipeline_succeeds" AS t0_r26,
"projects"."has_external_issue_tracker" AS t0_r27,
"projects"."repository_storage" AS t0_r28,
"projects"."request_access_enabled" AS t0_r29,
"projects"."has_external_wiki" AS t0_r30,
"projects"."lfs_enabled" AS t0_r31,
"projects"."description_html" AS t0_r32,
"projects"."only_allow_merge_if_all_discussions_are_resolved" AS t0_r33,
"projects"."printing_merge_request_link_enabled" AS t0_r34,
"projects"."auto_cancel_pending_pipelines" AS t0_r35,
"projects"."cached_markdown_version" AS t0_r36,
"projects"."last_repository_updated_at" AS t0_r37,
"projects"."ci_config_path" AS t0_r38,
"projects"."delete_error" AS t0_r39,
"projects"."storage_version" AS t0_r40,
"projects"."resolve_outdated_diff_discussions" AS t0_r41,
"projects"."merge_requests_rebase_enabled" AS t0_r42,
"projects"."merge_requests_ff_only_enabled" AS t0_r43,
"projects"."repository_read_only" AS t0_r44,
"projects"."jobs_cache_index" AS t0_r45,
"projects"."pages_https_only" AS t0_r46,
"projects"."remote_mirror_available_overridden" AS t0_r47,
"projects"."pool_repository_id" AS t0_r48,
"projects"."runners_token_encrypted" AS t0_r49,
"projects"."bfg_object_map" AS t0_r50,
"projects"."detected_repository_languages" AS t0_r51,
"projects"."external_authorization_classification_label" AS t0_r52,
"projects"."disable_overriding_approvers_per_merge_request" AS t0_r53,
"projects"."external_webhook_token" AS t0_r54,
"projects"."issues_template" AS t0_r55,
"projects"."merge_requests_author_approval" AS t0_r56,
"projects"."merge_requests_disable_committers_approval" AS t0_r57,
"projects"."merge_requests_template" AS t0_r58,
"projects"."mirror_last_successful_update_at" AS t0_r59,
"projects"."mirror_last_update_at" AS t0_r60,
"projects"."mirror_overwrites_diverged_branches" AS t0_r61,
"projects"."mirror_user_id" AS t0_r62,
"projects"."only_mirror_protected_branches" AS t0_r63,
"projects"."packages_enabled" AS t0_r64,
"projects"."pull_mirror_available_overridden" AS t0_r65,
"projects"."repository_size_limit" AS t0_r66,
"projects"."require_password_to_approve" AS t0_r67,
"projects"."mirror" AS t0_r68,
"projects"."mirror_trigger_builds" AS t0_r69,
"projects"."reset_approvals_on_push" AS t0_r70,
"projects"."service_desk_enabled" AS t0_r71,
"projects"."approvals_before_merge" AS t0_r72,
"projects"."emails_disabled" AS t0_r73,
"projects"."max_pages_size" AS t0_r74,
"projects"."max_artifacts_size" AS t0_r75,
"projects"."pull_mirror_branch_prefix" AS t0_r76,
"projects"."marked_for_deletion_at" AS t0_r77,
"projects"."marked_for_deletion_by_user_id" AS t0_r78,
"projects"."remove_source_branch_after_merge" AS t0_r79,
"projects"."suggestion_commit_message" AS t0_r80,
"projects"."autoclose_referenced_issues" AS t0_r81,
"routes"."id" AS t1_r0,
"routes"."source_id" AS t1_r1,
"routes"."source_type" AS t1_r2,
"routes"."path" AS t1_r3,
"routes"."created_at" AS t1_r4,
"routes"."updated_at" AS t1_r5,
"routes"."name" AS t1_r6
FROM "projects"
LEFT OUTER JOIN "routes" ON "routes"."source_type" = 'Project' AND
"routes"."source_id" = "projects"."id"
WHERE "routes"."path" = 'david/only-except-upgrade' LIMIT 1
Sql for Project.includes(:route).find(603)
.
SELECT "projects".* FROM "projects" WHERE "projects"."id" = 603 LIMIT 1
SELECT "routes".* FROM "routes" WHERE "routes"."source_type" = 'Project' AND "routes"."source_id" = 603
So it is potentially an issue with the LEFT OUTER JOIN "routes"
Please note that the when running this against the raw psql db it returns route info
Output of checks
Results of GitLab environment info
I am aware there are new patch releases out but none of the changelogs highlight this issue and I am worried about making the problem worse by upgrading mid issue.
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.27.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 13.1.0 Revision: d62ee60b669 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.2 URL: https://gitlab. HTTP Clone URL: https://gitlab.System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.27.0 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 13.1.0 Revision: d62ee60b669 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.2 URL: https://gitlab. HTTP Clone URL: https://gitlab./some-group/some-project.git SSH Clone URL: git@gitlab.:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git/some-group/some-project.git SSH Clone URL: git@gitlab.:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 5/2 ... yes 5/3 ... yes 4/4 ... yes 6/5 ... yes 6/7 ... yes 6/8 ... yes 118/9 ... yes 5/10 ... yes 2/12 ... yes 5/13 ... yes 5/14 ... yes 6/15 ... yes 6/16 ... yes 15/18 ... yes 4/19 ... yes 4/20 ... yes 4/21 ... yes 4/22 ... yes 10/24 ... yes 16/25 ... yes 2/26 ... yes 5/27 ... yes 4/29 ... yes 5/30 ... yes 4/31 ... yes 20/32 ... yes 16/33 ... yes 11/34 ... yes 10/35 ... yes 4/36 ... yes 5/37 ... yes 5/38 ... yes 8/39 ... yes 2/41 ... yes 2/42 ... yes 8/43 ... yes 5/44 ... yes 8/45 ... yes 5/46 ... yes 8/47 ... yes 8/48 ... yes 8/49 ... yes 4/50 ... yes 8/52 ... yes 28/53 ... yes 4/54 ... yes 8/55 ... yes 23/56 ... yes 2/58 ... yes 4/60 ... yes 31/61 ... yes 4/62 ... yes 5/64 ... yes 30/65 ... yes 32/66 ... yes 32/67 ... yes 32/68 ... yes 32/69 ... yes 35/70 ... yes 32/71 ... yes 32/72 ... yes 35/73 ... yes 32/74 ... yes 37/75 ... yes 37/76 ... yes 4/77 ... yes 4/78 ... yes 32/79 ... yes 5/80 ... yes 38/81 ... yes 32/82 ... yes 4/83 ... yes 5/85 ... yes 4/87 ... yes 4/89 ... yes 4/91 ... yes 4/110 ... yes 4/111 ... yes 11/112 ... yes 4/115 ... yes 25/117 ... yes 11/118 ... yes 25/119 ... yes 25/120 ... yes 25/121 ... yes 32/122 ... yes 43/123 ... yes 32/124 ... yes 43/125 ... yes 32/127 ... yes 32/128 ... yes 47/130 ... yes 47/131 ... yes 47/132 ... yes 47/133 ... yes 47/134 ... yes 47/135 ... yes 47/136 ... yes 47/137 ... yes 45/138 ... yes 47/139 ... yes 47/140 ... yes 47/141 ... yes 47/142 ... yes 47/143 ... yes 47/144 ... yes 47/145 ... yes 47/146 ... yes 49/147 ... yes 49/148 ... yes 49/149 ... yes 49/150 ... yes 49/151 ... yes 49/152 ... yes 49/153 ... yes 49/154 ... yes 49/155 ... yes 49/156 ... yes 46/157 ... yes 47/158 ... yes 49/160 ... yes 47/163 ... yes 32/164 ... yes 46/165 ... yes 5/166 ... yes 5/171 ... yes 8/172 ... yes 4/173 ... yes 4/174 ... yes 47/175 ... yes 45/176 ... yes 4/177 ... yes 4/178 ... yes 32/180 ... yes 32/181 ... yes 47/182 ... yes 47/183 ... yes 45/184 ... yes 32/185 ... yes 47/186 ... yes 3/187 ... yes 3/188 ... yes 45/189 ... yes 4/190 ... yes 32/193 ... yes 47/195 ... yes 2/197 ... yes 52/200 ... yes 52/201 ... yes 47/203 ... yes 45/204 ... yes 32/205 ... yes 4/206 ... yes 32/208 ... yes 32/209 ... yes 32/210 ... yes 32/211 ... yes 32/212 ... yes 45/213 ... yes 47/214 ... yes 49/215 ... yes 47/216 ... yes 56/218 ... yes 47/219 ... yes 32/221 ... yes 52/222 ... yes 45/223 ... yes 52/224 ... yes 32/225 ... yes 4/226 ... yes 32/227 ... yes 5/229 ... yes 5/230 ... yes 32/231 ... yes 33/232 ... yes 5/233 ... yes 52/234 ... yes 56/235 ... yes 52/236 ... yes 45/237 ... yes 56/238 ... yes 52/239 ... yes 60/240 ... yes 52/241 ... yes 57/242 ... yes 32/243 ... yes 3/244 ... yes 32/245 ... yes 3/246 ... yes 52/247 ... yes 57/249 ... yes 45/250 ... yes 56/251 ... yes 52/252 ... yes 45/254 ... yes 66/255 ... yes 66/256 ... yes 32/257 ... yes 60/260 ... yes 66/261 ... yes 66/262 ... yes 4/263 ... yes 3/264 ... yes 4/265 ... yes 52/266 ... yes 32/269 ... yes 4/270 ... yes 45/271 ... yes 71/272 ... yes 103/273 ... yes 4/275 ... yes 4/276 ... yes 103/277 ... yes 32/279 ... yes 60/281 ... yes 60/282 ... yes 66/283 ... yes 52/285 ... yes 52/287 ... yes 52/288 ... yes 47/289 ... yes 45/290 ... yes 52/291 ... yes 25/292 ... yes 56/293 ... yes 56/294 ... yes 52/295 ... yes 66/296 ... yes 47/298 ... yes 52/299 ... yes 4/300 ... yes 45/301 ... yes 33/303 ... yes 52/304 ... yes 59/305 ... yes 65/306 ... yes 8/307 ... yes 66/308 ... yes 60/309 ... yes 3/310 ... yes 56/312 ... yes 65/314 ... yes 4/315 ... yes 56/316 ... yes 3/317 ... yes 60/320 ... yes 65/322 ... yes 65/323 ... yes 4/324 ... yes 65/327 ... yes 60/328 ... yes 60/329 ... yes 60/330 ... yes 5/331 ... yes 52/332 ... yes 66/333 ... yes 4/334 ... yes 32/335 ... yes 47/336 ... yes 4/337 ... yes 32/338 ... yes 45/339 ... yes 47/340 ... yes 5/341 ... yes 5/342 ... yes 45/343 ... yes 5/344 ... yes 66/347 ... yes 45/348 ... yes 45/352 ... yes 3/353 ... yes 60/354 ... yes 4/355 ... yes 60/356 ... yes 25/357 ... yes 5/358 ... yes 32/359 ... yes 45/360 ... yes 56/361 ... yes 56/362 ... yes 118/363 ... yes 59/364 ... yes 56/365 ... yes 32/366 ... yes 45/367 ... yes 4/369 ... yes 32/374 ... yes 45/375 ... yes 32/376 ... yes 45/377 ... yes 4/378 ... yes 45/379 ... yes 52/381 ... yes 92/383 ... yes 67/385 ... yes 32/387 ... yes 4/388 ... yes 32/389 ... yes 93/391 ... yes 4/393 ... yes 45/394 ... yes 45/395 ... yes 4/396 ... yes 32/397 ... yes 4/398 ... yes 77/399 ... yes 60/401 ... yes 5/403 ... yes 45/404 ... yes 99/405 ... yes 99/406 ... yes 99/408 ... yes 5/409 ... yes 45/410 ... yes 99/413 ... yes 99/415 ... yes 56/416 ... yes 99/417 ... yes 99/418 ... yes 77/421 ... yes 77/424 ... yes 103/425 ... yes 102/426 ... yes 56/427 ... yes 77/428 ... yes 45/429 ... yes 99/430 ... yes 100/431 ... yes 4/432 ... yes 100/433 ... yes 5/435 ... yes 47/436 ... yes 99/437 ... yes 5/439 ... yes 60/440 ... yes 99/441 ... yes 47/443 ... yes 99/444 ... yes 99/445 ... yes 99/446 ... yes 99/447 ... yes 102/448 ... yes 47/449 ... yes 56/450 ... yes 56/451 ... yes 100/453 ... yes 56/454 ... yes 32/455 ... yes 105/456 ... yes 118/458 ... yes 56/459 ... yes 99/464 ... yes 32/465 ... yes 32/466 ... yes 5/467 ... yes 99/468 ... yes 99/469 ... yes 99/470 ... yes 93/471 ... yes 116/472 ... yes 52/473 ... yes 47/474 ... yes 52/475 ... yes 56/476 ... yes 120/477 ... yes 104/480 ... yes 52/481 ... yes 74/482 ... yes 121/483 ... yes 102/484 ... yes 102/485 ... yes 72/486 ... yes 47/488 ... yes 5/489 ... yes 56/490 ... yes 56/492 ... yes 99/493 ... yes 25/494 ... yes 47/496 ... yes 56/500 ... yes 130/501 ... yes 130/502 ... yes 130/503 ... yes 32/506 ... yes 47/507 ... yes 47/508 ... yes 47/509 ... yes 116/510 ... yes 99/511 ... yes 99/513 ... yes 47/514 ... yes 130/515 ... yes 130/516 ... yes 135/517 ... yes 52/518 ... yes 60/519 ... yes 45/520 ... yes 127/521 ... yes 45/522 ... yes 32/523 ... yes 60/524 ... yes 62/525 ... yes 56/526 ... yes 47/527 ... yes 131/529 ... yes 47/530 ... yes 52/531 ... yes 72/532 ... yes 93/533 ... yes 52/534 ... yes 134/537 ... yes 135/539 ... yes 99/540 ... yes 52/541 ... yes 39/542 ... yes 59/543 ... yes 72/544 ... yes 39/545 ... yes 134/546 ... yes 4/548 ... yes 72/549 ... yes 141/550 ... yes 135/551 ... yes 134/552 ... yes 143/553 ... yes 99/554 ... yes 129/555 ... yes 134/556 ... yes 134/557 ... yes 134/559 ... yes 134/560 ... yes 140/562 ... yes 47/563 ... yes 134/564 ... yes 140/565 ... yes 4/567 ... yes 143/568 ... yes 47/569 ... yes 47/570 ... yes 130/572 ... yes 32/573 ... yes 72/574 ... yes 4/575 ... yes 4/577 ... yes 141/578 ... yes 4/579 ... yes 141/580 ... yes 32/582 ... yes 5/583 ... yes 4/585 ... yes 134/588 ... yes 140/589 ... yes 56/590 ... yes 135/591 ... yes 134/592 ... yes 145/593 ... yes 130/594 ... yes 134/595 ... yes 4/597 ... yes 4/598 ... yes 47/599 ... yes 47/600 ... yes 99/601 ... yes 145/602 ... yes 52/603 ... yes 140/604 ... yes 103/605 ... yes 153/606 ... yes 59/607 ... yes 72/608 ... yes 142/609 ... yes 5/610 ... yes 93/611 ... yes 62/612 ... yes 4/613 ... yes 56/614 ... yes 134/615 ... yes 47/616 ... yes 152/617 ... yes 56/618 ... yes 106/619 ... yes 142/620 ... yes 4/621 ... yes 145/622 ... yes 145/623 ... yes 4/624 ... yes 5/625 ... yes 49/626 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.27.0) Git user has default SSH configuration? ... yes Active users: ... 65 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
find_by_full_path
in routeable. https://gitlab.com/gitlab-org/gitlab/-/blob/6ac20f415c1f4ed553426ef7580d4d4649105cce/app/models/concerns/routable.rb#L32-43
Downgrading to SQL 11? Is 12 supported, the docs listed minimum, but we have been using Psql 12 since GitLab 13.0 without issues.
A WIP MR that solves the issue(for us) !36711 (closed)
Thanks for getting this far in the bug report, sorry its so verbose my hope is that someone can help me isolate and fix this with as much info as possible