BackfillUserDetailsFields: users - error 15.4.6 => 15.9.0
Summary
Following the upgrade path from 15.4.6
-> 15.9.0
succeeds but the background migration BackfillUserDetailsFields: users
fails with the following error:
Exception Class | Exception Message |
---|---|
ActiveRecord::StatementInvalid | PG::UndefinedColumn: ERROR: column "linkedin" does not exist LINE 1: ...ERE "users"."id" BETWEEN 1 AND 287 AND ((COALESCE(linkedin, ... ^ |
Steps to reproduce
- update to
15.4.6
with following command:
sudo apt-get clean && sudo apt-get update && sudo apt-get -y install gitlab-ee=15.4.6-ee.0 && sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart && sleep 40
- wait for all background migrations to complete
- upgrade to
15.9.0
with the following command:
sudo apt-get clean && sudo apt-get update && sudo apt-get -y install gitlab-ee=15.9.0-ee.0 && sudo gitlab-ctl reconfigure && sudo gitlab-ctl restart && sleep 40
What is the current bug behavior?
Migration Fails
What is the expected correct behavior?
Migration Completes
Relevant logs and/or screenshots
Exception Class | Exception Message |
---|---|
ActiveRecord::StatementInvalid | PG::UndefinedColumn: ERROR: column "linkedin" does not exist LINE 1: ...ERE "users"."id" BETWEEN 1 AND 287 AND ((COALESCE(linkedin, ... ^ |
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.7p221 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.8 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.9.0-ee Revision: b107c0bae08 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.12 URL: https://**** HTTP Clone URL: https://****/some-group/some-project.git SSH Clone URL: git@****:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.17.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.17.0 ? ... OK (14.17.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 10/2 ... yes 10/3 ... yes 13/4 ... yes 10/5 ... yes 10/6 ... yes 9/8 ... yes 9/10 ... yes 9/11 ... yes 18/12 ... yes 13/13 ... yes 18/14 ... yes 13/15 ... yes 10/18 ... yes 10/19 ... yes 232/20 ... yes 10/21 ... yes 10/22 ... yes 232/23 ... yes 10/24 ... yes 10/25 ... yes 232/26 ... yes 232/29 ... yes 232/30 ... yes 10/31 ... yes 10/32 ... yes 232/33 ... yes 10/34 ... yes 10/35 ... yes 232/36 ... yes 232/38 ... yes 10/39 ... yes 10/40 ... yes 10/41 ... yes 39/43 ... yes 13/45 ... yes 13/46 ... yes 39/47 ... yes 39/48 ... yes 39/49 ... yes 39/50 ... yes 39/51 ... yes 42/52 ... yes 42/53 ... yes 42/54 ... yes 44/55 ... yes 10/56 ... yes 10/57 ... yes 10/58 ... yes 10/59 ... yes 48/60 ... yes 48/61 ... yes 38/62 ... yes 48/63 ... yes 48/64 ... yes 48/65 ... yes 48/66 ... yes 38/67 ... yes 48/68 ... yes 48/69 ... yes 38/70 ... yes 48/71 ... yes 48/74 ... yes 38/75 ... yes 48/76 ... yes 38/79 ... yes 38/80 ... yes 38/81 ... yes 38/82 ... yes 38/84 ... yes 38/87 ... yes 38/88 ... yes 38/89 ... yes 193/91 ... yes 38/93 ... yes 48/94 ... yes 18/95 ... yes 52/97 ... yes 52/98 ... yes 38/99 ... yes 38/100 ... yes 38/101 ... yes 38/102 ... yes 38/103 ... yes 10/105 ... yes 9/106 ... yes 13/107 ... yes 9/108 ... yes 39/109 ... yes 39/110 ... yes 9/111 ... yes 9/112 ... yes 9/113 ... yes 9/114 ... yes 9/115 ... yes 9/116 ... yes 9/117 ... yes 9/118 ... yes 9/119 ... yes 9/120 ... yes 9/121 ... yes 9/122 ... yes 9/123 ... yes 9/124 ... yes 9/125 ... yes 9/126 ... yes 18/127 ... yes 53/128 ... yes 18/129 ... yes 18/130 ... yes 53/131 ... yes 53/132 ... yes 13/133 ... yes 38/134 ... yes 38/135 ... yes 55/136 ... yes 54/137 ... yes 48/138 ... yes 55/140 ... yes 9/141 ... yes 9/142 ... yes 53/143 ... yes 9/144 ... yes 9/145 ... yes 9/146 ... yes 9/147 ... yes 9/148 ... yes 9/149 ... yes 9/150 ... yes 9/151 ... yes 9/152 ... yes 9/153 ... yes 9/154 ... yes 9/155 ... yes 9/156 ... yes 9/157 ... yes 9/158 ... yes 9/159 ... yes 9/160 ... yes 9/161 ... yes 54/164 ... yes 193/165 ... yes 14/166 ... yes 17/167 ... yes 20/168 ... yes 13/169 ... yes 13/170 ... yes 60/172 ... yes 60/173 ... yes 54/174 ... yes 13/175 ... yes 59/176 ... yes 59/178 ... yes 67/179 ... yes 69/180 ... yes 68/183 ... yes 66/184 ... yes 70/185 ... yes 70/188 ... yes 67/189 ... yes 70/192 ... yes 53/194 ... yes 66/195 ... yes 66/196 ... yes 67/197 ... yes 65/198 ... yes 69/199 ... yes 65/200 ... yes 66/201 ... yes 67/202 ... yes 54/203 ... yes 68/204 ... yes 54/205 ... yes 69/206 ... yes 70/207 ... yes 70/208 ... yes 68/210 ... yes 65/211 ... yes 67/212 ... yes 66/214 ... yes 54/215 ... yes 68/216 ... yes 59/217 ... yes 67/218 ... yes 13/219 ... yes 232/220 ... yes 67/222 ... yes 13/224 ... yes 9/226 ... yes 78/227 ... yes 10/228 ... yes 59/229 ... yes 10/230 ... yes 38/234 ... yes 13/236 ... yes 13/237 ... yes 82/239 ... yes 82/240 ... yes 82/243 ... yes 14/244 ... yes 23/245 ... yes 85/247 ... yes 12/248 ... yes 13/249 ... yes 13/252 ... yes 79/253 ... yes 79/255 ... yes 13/256 ... yes 13/258 ... yes 87/260 ... yes 10/261 ... yes 69/262 ... yes 87/264 ... yes 13/265 ... yes 13/266 ... yes 38/268 ... yes 50/269 ... yes 47/271 ... yes 13/273 ... yes 9/274 ... yes 90/275 ... yes 91/276 ... yes 93/279 ... yes 89/280 ... yes 93/281 ... yes 10/282 ... yes 90/284 ... yes 90/285 ... yes 94/287 ... yes 93/289 ... yes 95/290 ... yes 48/291 ... yes 97/293 ... yes 59/295 ... yes 97/296 ... yes 69/297 ... yes 18/298 ... yes 59/299 ... yes 97/300 ... yes 92/301 ... yes 95/302 ... yes 14/304 ... yes 69/305 ... yes 14/306 ... yes 100/307 ... yes 90/308 ... yes 99/309 ... yes 93/310 ... yes 13/311 ... yes 47/312 ... yes 99/313 ... yes 89/317 ... yes 13/318 ... yes 47/319 ... yes 47/320 ... yes 10/321 ... yes 92/323 ... yes 13/325 ... yes 47/326 ... yes 94/327 ... yes 82/329 ... yes 232/330 ... yes 47/332 ... yes 47/333 ... yes 95/334 ... yes 59/335 ... yes 59/337 ... yes 89/338 ... yes 92/339 ... yes 95/340 ... yes 104/341 ... yes 59/342 ... yes 106/343 ... yes 90/344 ... yes 87/345 ... yes 90/346 ... yes 47/347 ... yes 47/348 ... yes 107/352 ... yes 108/353 ... yes 108/354 ... yes 108/355 ... yes 106/356 ... yes 10/357 ... yes 59/358 ... yes 232/359 ... yes 113/362 ... yes 232/363 ... yes 13/365 ... yes 113/366 ... yes 113/367 ... yes 13/368 ... yes 113/369 ... yes 95/370 ... yes 13/371 ... yes 13/372 ... yes 9/373 ... yes 47/374 ... yes 113/375 ... yes 82/376 ... yes 108/377 ... yes 108/378 ... yes 13/379 ... yes 13/380 ... yes 95/382 ... yes 53/383 ... yes 118/384 ... yes 10/385 ... yes 118/386 ... yes 47/387 ... yes 95/388 ... yes 117/389 ... yes 89/390 ... yes 10/391 ... yes 13/392 ... yes 10/393 ... yes 10/394 ... yes 95/395 ... yes 10/397 ... yes 10/399 ... yes 13/400 ... yes 10/401 ... yes 9/402 ... yes 47/403 ... yes 10/404 ... yes 232/405 ... yes 95/406 ... yes 95/407 ... yes 95/408 ... yes 10/409 ... yes 13/410 ... yes 235/411 ... yes 88/412 ... yes 95/413 ... yes 87/414 ... yes 10/415 ... yes 89/416 ... yes 10/417 ... yes 95/418 ... yes 281/419 ... yes 89/420 ... yes 47/421 ... yes 10/422 ... yes 95/423 ... yes 10/424 ... yes 13/425 ... yes 110/426 ... yes 131/427 ... yes 310/428 ... yes 10/429 ... yes 12/430 ... yes 113/431 ... yes 12/432 ... yes 63/434 ... yes 10/435 ... yes 10/436 ... yes 10/437 ... yes 89/438 ... yes 93/439 ... yes 12/440 ... yes 13/441 ... yes 95/442 ... yes 10/443 ... yes 95/444 ... yes 95/445 ... yes 49/446 ... yes 49/447 ... yes 13/448 ... yes 13/449 ... yes 13/450 ... yes 10/451 ... yes 95/452 ... yes 47/453 ... yes 47/455 ... yes 151/456 ... yes 154/458 ... yes 10/459 ... yes 79/460 ... yes 47/461 ... yes 10/463 ... yes 95/464 ... yes 154/465 ... yes 167/467 ... yes 95/468 ... yes 166/470 ... yes 10/471 ... yes 193/472 ... yes 10/473 ... yes 154/474 ... yes 232/475 ... yes 95/476 ... yes 77/477 ... yes 48/479 ... yes 48/480 ... yes 95/481 ... yes 13/482 ... yes 13/483 ... yes 95/484 ... yes 95/485 ... yes 13/486 ... yes 47/487 ... yes 13/489 ... yes 13/490 ... yes 113/491 ... yes 160/492 ... yes 114/493 ... yes 172/494 ... yes 172/495 ... yes 154/496 ... yes 154/497 ... yes 165/498 ... yes 164/499 ... yes 164/500 ... yes 166/501 ... yes 13/502 ... yes 13/503 ... yes 174/504 ... yes 13/505 ... yes 13/506 ... yes 166/507 ... yes 166/508 ... yes 166/509 ... yes 95/510 ... yes 166/511 ... yes 232/512 ... yes 166/514 ... yes 10/515 ... yes 33/516 ... yes 79/517 ... yes 38/518 ... yes 167/519 ... yes 13/520 ... yes 10/521 ... yes 154/522 ... yes 166/523 ... yes 164/524 ... yes 10/525 ... yes 195/526 ... yes 13/527 ... yes 10/528 ... yes 18/529 ... yes 13/530 ... yes 48/532 ... yes 98/533 ... yes 9/534 ... yes 166/535 ... yes 38/536 ... yes 144/537 ... yes 193/538 ... yes 13/539 ... yes 232/540 ... yes 173/541 ... yes 10/542 ... yes 166/543 ... yes 281/545 ... yes 180/546 ... yes 48/547 ... yes 13/548 ... yes 10/549 ... yes 10/550 ... yes 13/551 ... yes 185/552 ... yes 38/553 ... yes 185/554 ... yes 82/555 ... yes 154/556 ... yes 185/557 ... yes 12/558 ... yes 154/559 ... yes 195/560 ... yes 108/561 ... yes 185/563 ... yes 10/564 ... yes 193/565 ... yes 185/566 ... yes 185/567 ... yes 185/569 ... yes 189/570 ... yes 192/572 ... yes 154/575 ... yes 154/576 ... yes 150/577 ... yes 281/578 ... yes 217/579 ... yes 217/580 ... yes 179/581 ... yes 169/582 ... yes 189/583 ... yes 195/584 ... yes 189/585 ... yes 189/586 ... yes 194/587 ... yes 192/588 ... yes 192/589 ... yes 10/590 ... yes 195/591 ... yes 194/592 ... yes 189/593 ... yes 189/594 ... yes 192/595 ... yes 195/596 ... yes 10/597 ... yes 174/598 ... yes 10/599 ... yes 10/600 ... yes 192/601 ... yes 194/602 ... yes 106/604 ... yes 108/605 ... yes 199/606 ... yes 201/607 ... yes 201/608 ... yes 13/609 ... yes 13/611 ... yes 192/612 ... yes 203/613 ... yes 179/615 ... yes 192/616 ... yes 195/617 ... yes 189/618 ... yes 13/619 ... yes 12/620 ... yes 208/622 ... yes 12/623 ... yes 12/624 ... yes 77/626 ... yes 194/627 ... yes 232/628 ... yes 10/629 ... yes 232/630 ... yes 10/631 ... yes 10/632 ... yes 13/633 ... yes 13/635 ... yes 217/636 ... yes 217/637 ... yes 117/638 ... yes 13/639 ... yes 181/640 ... yes 192/641 ... yes 38/642 ... yes 12/643 ... yes 13/644 ... yes 10/645 ... yes 223/646 ... yes 201/647 ... yes 189/648 ... yes 10/649 ... yes 10/650 ... yes 232/651 ... yes 13/652 ... yes 227/653 ... yes 232/654 ... yes 114/655 ... yes 237/656 ... yes 235/657 ... yes 192/658 ... yes 232/659 ... yes 236/660 ... yes 10/661 ... yes 237/662 ... yes 10/663 ... yes 185/664 ... yes 237/665 ... yes 220/666 ... yes 220/667 ... yes 220/668 ... yes 220/669 ... yes 220/670 ... yes 282/671 ... yes 222/672 ... yes 223/674 ... yes 13/675 ... yes 93/677 ... yes 237/679 ... yes 237/682 ... yes 232/683 ... yes 237/684 ... yes 237/685 ... yes 10/686 ... yes 10/688 ... yes 114/689 ... yes 237/690 ... yes 220/691 ... yes 13/693 ... yes 186/696 ... yes 189/699 ... yes 114/701 ... yes 10/702 ... yes 10/703 ... yes 237/704 ... yes 237/706 ... yes 38/707 ... yes 220/708 ... yes 237/709 ... yes 189/710 ... yes 10/711 ... yes 232/712 ... yes 10/713 ... yes 235/715 ... yes 10/716 ... yes 237/717 ... yes 10/719 ... yes 192/720 ... yes 10/721 ... yes 237/722 ... yes 255/724 ... yes 10/725 ... yes 10/726 ... yes 10/727 ... yes 143/728 ... yes 13/730 ... yes 255/733 ... yes 10/734 ... yes 10/735 ... yes 220/736 ... yes 220/737 ... yes 232/738 ... yes 93/739 ... yes 189/740 ... yes 282/741 ... yes 232/742 ... yes 232/743 ... yes 13/744 ... yes 232/746 ... yes 237/747 ... yes 232/748 ... yes 10/749 ... yes 282/750 ... yes 282/751 ... yes 234/752 ... yes 236/753 ... yes 232/754 ... yes 277/755 ... yes 277/756 ... yes 277/757 ... yes 236/759 ... yes 232/760 ... yes 189/761 ... yes 232/762 ... yes 232/763 ... yes 272/764 ... yes 278/765 ... yes 232/767 ... yes 237/768 ... yes 281/770 ... yes 282/771 ... yes 189/772 ... yes 282/773 ... yes 237/774 ... yes 93/775 ... yes 282/776 ... yes 282/778 ... yes 278/780 ... yes 277/781 ... yes 232/782 ... yes 38/783 ... yes 48/785 ... yes 93/786 ... yes 232/788 ... yes 28/790 ... yes 232/791 ... yes 93/792 ... yes 232/793 ... yes 10/794 ... yes 232/795 ... yes 293/796 ... yes 237/797 ... yes 281/798 ... yes 237/799 ... yes 282/800 ... yes 281/801 ... yes 85/802 ... yes 237/803 ... yes 239/804 ... yes 281/805 ... yes 13/806 ... yes 209/807 ... yes 223/809 ... yes 10/810 ... yes 239/811 ... yes 38/812 ... yes 13/813 ... yes 93/814 ... yes 281/815 ... yes 236/816 ... yes 179/817 ... yes 293/818 ... yes 293/820 ... yes 93/821 ... yes 59/822 ... yes 10/823 ... yes 282/825 ... yes 277/826 ... yes 281/827 ... yes 237/828 ... yes 237/829 ... yes 10/830 ... yes 281/831 ... yes 237/832 ... yes 220/833 ... yes 195/835 ... yes 195/836 ... yes 195/837 ... yes 277/838 ... yes 10/839 ... yes 189/840 ... yes 270/842 ... yes 10/843 ... yes 270/845 ... yes 189/846 ... yes 220/847 ... yes 13/848 ... yes 10/849 ... yes 10/850 ... yes 189/851 ... yes 270/853 ... yes 281/854 ... yes 42/855 ... yes 93/856 ... yes 10/857 ... yes 189/858 ... yes 281/859 ... yes 353/860 ... yes 353/861 ... yes 353/862 ... yes 353/863 ... yes 353/864 ... yes 353/865 ... yes 353/866 ... yes 353/867 ... yes 353/868 ... yes 353/869 ... yes 353/870 ... yes 353/871 ... yes 353/872 ... yes 353/873 ... yes 353/874 ... yes 353/875 ... yes 353/876 ... yes 353/877 ... yes 353/878 ... yes 353/879 ... yes 10/880 ... yes 220/881 ... yes 277/882 ... yes 189/883 ... yes 189/885 ... yes 189/886 ... yes 189/887 ... yes 189/888 ... yes 189/889 ... yes 322/890 ... yes 389/891 ... yes 189/892 ... yes 392/893 ... yes 232/894 ... yes 316/895 ... yes 189/896 ... yes 10/897 ... yes 9/898 ... yes 13/899 ... yes 108/900 ... yes 10/901 ... yes 189/903 ... yes 280/904 ... yes 237/905 ... yes 10/906 ... yes 169/907 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.7) Git user has default SSH configuration? ... yes Active users: ... 70 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... yes (elasticsearch 7.10.2) All migrations must be finished before doing a major upgrade ... no (You have 21 pending migrations.) For more information see: https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#all-migrations-must-be-finished-before-doing-a-major-upgrade Try fixing it: Wait for all advanced search migrations to complete. To list pending migrations, run
sudo gitlab-rake gitlab:elastic:list_pending_migrations
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
$ sudo gitlab-rake gitlab:elastic:list_pending_migrations Pending migrations: BackfillNamespaceAncestryForIssues AddNamespaceAncestryIdsToIssuesMapping RedoBackfillNamespaceAncestryIdsForIssues DeleteOrphanedCommits PopulateCommitPermissionsInMainIndex PauseIndexingForUnsupportedEsVersions MigrateCommitsToSeparateIndex DeleteCommitsFromOriginalIndex AddLabelIdsAndSchemaVersionToIssuesMapping BackfillLabelIdsForIssues CreateUserIndex BackfillUsers AddNamespaceAncestryIdsToOriginalIndexMapping AddTraversalIdsToOriginalIndexMapping BackfillTraversalIdsToBlobsAndWikiBlobs AddInternalToNotes AddSchemaVersionToMainIndexMapping BackfillTraversalIdsForProjects BackfillInternalOnNotes DeleteRemovedFieldsFromUsers ReindexUsers
Note: i see none of those pending migrations in the UI