Backgroup migrations fails for BackfillAdminModeScopeForPersonalAccessTokens: personal_access_tokens

Summary

After upgrading our self-hosted Gitlab from 15.7.5 to 15.8.0 (chart deployment), the Backgroup migrations has a failed job: BackfillAdminModeScopeForPersonalAccessTokens: personal_access_tokens

Steps to reproduce

I'm not sure it's reproductible

1. install helm chart 6.7.5 (gitlab 15.7.5)
2. upgrade the helm chart to 6.8.0 (gitlab 15.8.0) using the following command

   helm -n gitlab upgrade gitlab gitlab/gitlab --description "Upgrade to version 15.8.0" --version 6.8.0 -f values.yaml --set gitlab.migrations.enabled=true

Example Project

What is the current bug behavior?

In the Admin Area > Backgroup Migrations the job is noted failed

Batched Job (Id: 137)

Id	Min Value	Max Value	Batch size	Started at	Finished at	Attempts	Pause time (ms)
137	9	248	1000	2023-01-23 12:19:07 UTC	2023-01-23 12:19:08 UTC	3	100

Transition logs:

Id	Created At	Previous Status	Next Status	Exception Class	Exception Message
196	2023-01-23 12:15:35 UTC	pending	running
197	2023-01-23 12:15:35 UTC	running	failed	Psych::DisallowedClass	Tried to load unspecified class: Symbol
198	2023-01-23 12:18:02 UTC	failed	running
199	2023-01-23 12:18:03 UTC	running	failed	Psych::DisallowedClass	Tried to load unspecified class: Symbol
200	2023-01-23 12:19:07 UTC	failed	running
201	2023-01-23 12:19:08 UTC	running	failed	Psych::DisallowedClass	Tried to load unspecified class: Symbol

What is the expected correct behavior?

The migration job should be successful

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

$ kubectl -n gitlab exec -it deployments/gitlab-toolbox -c toolbox -- gitlab-rake gitlab:env:info

System information
System:		
Proxy:		GITLAB_GITLAB_PAGES_SERVICE_PORT_HTTP_GITLAB_PAGES_PROXY: 8090
Current User:	git
Using RVM:	no
Ruby Version:	2.7.7p221
Gem Version:	3.2.33
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.0.16
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	15.8.0-ee
Revision:	1d89c23c9f9
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	14.6
URL:		https://gitlab.mim-libre.fr
HTTP Clone URL:	https://gitlab.mim-libre.fr/some-group/some-project.git
SSH Clone URL:	git@gitlab.mim-libre.fr:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: oauth2_generic

GitLab Shell
Version:	14.15.0
Repository storages:
- default: 	tcp://gitlab-gitaly-0.gitlab-gitaly.gitlab.svc:8075
GitLab Shell path:		/home/git/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

$ kubectl -n gitlab exec -it deployments/gitlab-toolbox -c toolbox -- gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.15.0 ? ... OK (14.15.0)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Systemd unit files or init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
5/9 ... yes
13/10 ... yes
14/11 ... yes
13/13 ... yes
11/14 ... yes
10/15 ... yes
13/16 ... yes
13/17 ... yes
13/18 ... yes
13/19 ... yes
13/20 ... yes
13/21 ... yes
48/22 ... yes
48/23 ... yes
48/24 ... yes
48/25 ... yes
48/26 ... yes
13/27 ... yes
48/28 ... yes
48/29 ... yes
48/30 ... yes
48/31 ... yes
48/32 ... yes
48/33 ... yes
48/34 ... yes
48/35 ... yes
48/36 ... yes
48/37 ... yes
48/39 ... yes
48/40 ... yes
48/41 ... yes
30/42 ... yes
30/43 ... yes
13/44 ... yes
68/47 ... yes
71/48 ... yes
13/49 ... yes
71/50 ... yes
40/51 ... yes
23/52 ... yes
40/53 ... yes
71/54 ... yes
5/55 ... yes
82/56 ... yes
89/57 ... yes
235/58 ... yes
95/59 ... yes
52/60 ... yes
98/61 ... yes
298/62 ... yes
103/63 ... yes
103/64 ... yes
103/65 ... yes
103/66 ... yes
103/67 ... yes
103/68 ... yes
113/69 ... yes
5/74 ... yes
119/75 ... yes
119/76 ... yes
122/77 ... yes
122/78 ... yes
114/79 ... yes
5/81 ... yes
14/82 ... yes
43/83 ... yes
118/84 ... yes
102/86 ... yes
103/87 ... yes
98/88 ... yes
71/89 ... yes
104/90 ... yes
103/91 ... yes
104/92 ... yes
104/93 ... yes
108/94 ... yes
104/95 ... yes
82/98 ... yes
139/99 ... yes
42/100 ... yes
42/101 ... yes
148/102 ... yes
148/103 ... yes
99/104 ... yes
148/105 ... yes
173/106 ... yes
177/107 ... yes
148/108 ... yes
99/109 ... yes
40/110 ... yes
175/111 ... yes
175/112 ... yes
71/115 ... yes
71/116 ... yes
71/118 ... yes
71/119 ... yes
175/120 ... yes
171/121 ... yes
223/122 ... yes
103/123 ... yes
71/124 ... yes
148/125 ... yes
227/126 ... yes
71/127 ... yes
231/128 ... yes
175/129 ... yes
99/130 ... yes
175/131 ... yes
233/132 ... yes
234/133 ... yes
234/134 ... yes
234/135 ... yes
5/136 ... yes
71/137 ... yes
53/138 ... yes
77/140 ... yes
175/141 ... yes
175/142 ... yes
239/143 ... yes
241/144 ... yes
245/145 ... yes
5/146 ... yes
247/147 ... yes
247/148 ... yes
247/149 ... yes
247/150 ... yes
247/151 ... yes
247/152 ... yes
247/153 ... yes
247/154 ... yes
247/155 ... yes
247/156 ... yes
247/157 ... yes
247/158 ... yes
247/159 ... yes
247/160 ... yes
247/161 ... yes
136/162 ... yes
175/163 ... yes
53/164 ... yes
53/165 ... yes
71/166 ... yes
235/167 ... yes
261/168 ... yes
43/169 ... yes
384/170 ... yes
384/171 ... yes
53/173 ... yes
384/174 ... yes
384/175 ... yes
263/176 ... yes
261/177 ... yes
384/178 ... yes
384/179 ... yes
261/180 ... yes
99/182 ... yes
261/183 ... yes
71/184 ... yes
175/185 ... yes
71/186 ... yes
175/187 ... yes
273/188 ... yes
277/190 ... yes
175/191 ... yes
175/192 ... yes
84/193 ... yes
71/194 ... yes
227/196 ... yes
71/197 ... yes
286/198 ... yes
288/199 ... yes
84/200 ... yes
71/201 ... yes
71/202 ... yes
299/203 ... yes
310/204 ... yes
310/205 ... yes
310/206 ... yes
311/207 ... yes
311/208 ... yes
318/209 ... yes
312/210 ... yes
312/211 ... yes
311/212 ... yes
71/213 ... yes
314/214 ... yes
314/215 ... yes
314/216 ... yes
314/217 ... yes
314/218 ... yes
227/219 ... yes
191/220 ... yes
318/221 ... yes
99/222 ... yes
321/223 ... yes
175/224 ... yes
431/226 ... yes
312/227 ... yes
175/228 ... yes
294/229 ... yes
294/230 ... yes
294/231 ... yes
431/232 ... yes
430/233 ... yes
329/234 ... yes
334/236 ... yes
334/237 ... yes
334/238 ... yes
334/239 ... yes
329/241 ... yes
318/242 ... yes
336/243 ... yes
335/245 ... yes
338/247 ... yes
431/248 ... yes
318/249 ... yes
318/250 ... yes
175/251 ... yes
175/252 ... yes
318/253 ... yes
341/254 ... yes
341/255 ... yes
341/256 ... yes
341/257 ... yes
341/258 ... yes
341/259 ... yes
341/260 ... yes
341/261 ... yes
341/262 ... yes
341/263 ... yes
340/264 ... yes
341/265 ... yes
341/266 ... yes
1361/267 ... yes
329/268 ... yes
430/269 ... yes
71/270 ... yes
1361/271 ... yes
342/272 ... yes
335/273 ... yes
335/274 ... yes
335/275 ... yes
288/277 ... yes
349/278 ... yes
346/279 ... yes
348/280 ... yes
348/281 ... yes
246/282 ... yes
14/283 ... yes
335/284 ... yes
351/285 ... yes
352/286 ... yes
344/287 ... yes
356/288 ... yes
71/289 ... yes
360/290 ... yes
431/291 ... yes
301/292 ... yes
301/293 ... yes
296/294 ... yes
5/295 ... yes
364/296 ... yes
282/298 ... yes
364/299 ... yes
364/300 ... yes
365/301 ... yes
364/302 ... yes
364/303 ... yes
366/304 ... yes
341/305 ... yes
366/306 ... yes
342/308 ... yes
1361/309 ... yes
341/310 ... yes
335/311 ... yes
301/312 ... yes
383/313 ... yes
364/314 ... yes
300/316 ... yes
341/317 ... yes
346/318 ... yes
360/319 ... yes
360/320 ... yes
361/321 ... yes
360/322 ... yes
382/323 ... yes
282/324 ... yes
71/325 ... yes
383/327 ... yes
383/328 ... yes
383/329 ... yes
383/332 ... yes
399/334 ... yes
360/335 ... yes
385/336 ... yes
360/338 ... yes
342/339 ... yes
322/340 ... yes
384/341 ... yes
349/342 ... yes
349/343 ... yes
397/344 ... yes
235/345 ... yes
349/346 ... yes
349/347 ... yes
342/348 ... yes
431/349 ... yes
342/350 ... yes
400/351 ... yes
402/352 ... yes
227/353 ... yes
402/354 ... yes
405/355 ... yes
384/356 ... yes
384/357 ... yes
322/358 ... yes
175/359 ... yes
383/362 ... yes
383/363 ... yes
342/364 ... yes
397/365 ... yes
431/366 ... yes
5/367 ... yes
416/368 ... yes
175/369 ... yes
382/370 ... yes
495/371 ... yes
321/372 ... yes
342/373 ... yes
1361/374 ... yes
322/375 ... yes
399/376 ... yes
382/377 ... yes
318/378 ... yes
71/380 ... yes
382/381 ... yes
175/382 ... yes
382/383 ... yes
382/384 ... yes
43/387 ... yes
282/388 ... yes
399/389 ... yes
342/390 ... yes
43/393 ... yes
43/394 ... yes
430/395 ... yes
322/396 ... yes
1361/397 ... yes
349/398 ... yes
430/399 ... yes
342/400 ... yes
342/401 ... yes
1361/402 ... yes
322/403 ... yes
430/404 ... yes
312/406 ... yes
438/407 ... yes
23/408 ... yes
340/409 ... yes
442/410 ... yes
402/411 ... yes
402/412 ... yes
402/413 ... yes
402/414 ... yes
402/415 ... yes
382/416 ... yes
312/417 ... yes
399/419 ... yes
5/420 ... yes
423/422 ... yes
40/423 ... yes
415/425 ... yes
175/427 ... yes
382/429 ... yes
461/432 ... yes
43/433 ... yes
461/434 ... yes
376/436 ... yes
383/437 ... yes
383/438 ... yes
175/440 ... yes
441/441 ... yes
246/443 ... yes
342/444 ... yes
322/445 ... yes
43/446 ... yes
382/447 ... yes
384/448 ... yes
43/449 ... yes
382/450 ... yes
43/451 ... yes
430/452 ... yes
1361/453 ... yes
430/454 ... yes
430/455 ... yes
43/456 ... yes
1361/458 ... yes
495/459 ... yes
1361/460 ... yes
71/461 ... yes
402/462 ... yes
383/463 ... yes
490/464 ... yes
490/465 ... yes
43/466 ... yes
402/467 ... yes
341/468 ... yes
322/469 ... yes
341/470 ... yes
472/471 ... yes
322/472 ... yes
342/473 ... yes
318/474 ... yes
382/475 ... yes
342/476 ... yes
136/477 ... yes
382/478 ... yes
349/479 ... yes
341/480 ... yes
17/481 ... yes
346/482 ... yes
377/483 ... yes
375/484 ... yes
402/486 ... yes
402/487 ... yes
382/488 ... yes
495/489 ... yes
382/490 ... yes
500/491 ... yes
501/492 ... yes
175/493 ... yes
246/494 ... yes
495/495 ... yes
503/496 ... yes
84/497 ... yes
84/498 ... yes
416/499 ... yes
416/500 ... yes
399/501 ... yes
430/502 ... yes
1361/503 ... yes
416/504 ... yes
136/507 ... yes
508/508 ... yes
509/509 ... yes
508/511 ... yes
382/512 ... yes
71/513 ... yes
227/514 ... yes
296/515 ... yes
43/516 ... yes
1513/517 ... yes
43/518 ... yes
383/521 ... yes
294/523 ... yes
318/524 ... yes
52/525 ... yes
503/526 ... yes
71/527 ... yes
349/528 ... yes
498/529 ... yes
322/530 ... yes
322/531 ... yes
384/532 ... yes
322/533 ... yes
342/534 ... yes
296/535 ... yes
535/536 ... yes
342/537 ... yes
344/538 ... yes
342/539 ... yes
175/540 ... yes
382/541 ... yes
318/542 ... yes
318/543 ... yes
175/544 ... yes
318/545 ... yes
318/546 ... yes
318/547 ... yes
501/548 ... yes
366/549 ... yes
175/550 ... yes
500/551 ... yes
344/552 ... yes
342/553 ... yes
383/554 ... yes
382/555 ... yes
548/557 ... yes
431/558 ... yes
1361/559 ... yes
366/560 ... yes
366/562 ... yes
318/563 ... yes
430/566 ... yes
1361/567 ... yes
338/568 ... yes
437/569 ... yes
558/570 ... yes
382/571 ... yes
383/572 ... yes
382/573 ... yes
383/574 ... yes
342/576 ... yes
344/577 ... yes
43/578 ... yes
431/579 ... yes
318/580 ... yes
71/583 ... yes
1361/584 ... yes
416/585 ... yes
383/586 ... yes
53/587 ... yes
318/588 ... yes
43/590 ... yes
569/591 ... yes
570/593 ... yes
23/594 ... yes
558/596 ... yes
382/597 ... yes
573/598 ... yes
43/599 ... yes
43/601 ... yes
322/602 ... yes
312/603 ... yes
227/604 ... yes
43/605 ... yes
136/606 ... yes
416/607 ... yes
342/608 ... yes
579/609 ... yes
318/610 ... yes
322/611 ... yes
71/612 ... yes
342/613 ... yes
382/614 ... yes
582/616 ... yes
71/617 ... yes
322/618 ... yes
342/619 ... yes
430/620 ... yes
1361/621 ... yes
382/622 ... yes
318/623 ... yes
341/624 ... yes
341/625 ... yes
341/626 ... yes
341/627 ... yes
364/628 ... yes
416/629 ... yes
501/630 ... yes
322/631 ... yes
586/632 ... yes
341/633 ... yes
341/634 ... yes
364/635 ... yes
341/637 ... yes
364/639 ... yes
341/640 ... yes
342/641 ... yes
335/642 ... yes
322/643 ... yes
342/644 ... yes
296/645 ... yes
592/647 ... yes
341/648 ... yes
342/649 ... yes
416/650 ... yes
296/651 ... yes
587/652 ... yes
382/653 ... yes
346/658 ... yes
1181/660 ... yes
1181/661 ... yes
1181/662 ... yes
1181/663 ... yes
1181/664 ... yes
1181/665 ... yes
1189/666 ... yes
1189/667 ... yes
1189/668 ... yes
1190/669 ... yes
1190/670 ... yes
1190/671 ... yes
416/672 ... yes
71/673 ... yes
11/674 ... yes
71/675 ... yes
1181/676 ... yes
227/677 ... yes
438/678 ... yes
438/679 ... yes
5/680 ... yes
1220/681 ... yes
1220/682 ... yes
382/683 ... yes
296/684 ... yes
382/685 ... yes
416/686 ... yes
587/687 ... yes
382/688 ... yes
382/689 ... yes
438/690 ... yes
231/691 ... yes
1189/692 ... yes
501/693 ... yes
501/694 ... yes
501/695 ... yes
553/696 ... yes
341/697 ... yes
1246/698 ... yes
501/699 ... yes
501/701 ... yes
501/702 ... yes
1246/703 ... yes
531/704 ... yes
548/705 ... yes
1245/706 ... yes
1261/707 ... yes
344/708 ... yes
342/709 ... yes
1246/710 ... yes
416/711 ... yes
1246/712 ... yes
579/713 ... yes
231/714 ... yes
1246/715 ... yes
1246/716 ... yes
1246/717 ... yes
43/718 ... yes
1283/720 ... yes
327/721 ... yes
1287/722 ... yes
1287/723 ... yes
335/725 ... yes
366/726 ... yes
1286/727 ... yes
344/728 ... yes
366/729 ... yes
342/730 ... yes
503/731 ... yes
1246/732 ... yes
1274/733 ... yes
366/734 ... yes
1286/735 ... yes
1308/736 ... yes
1283/739 ... yes
1274/740 ... yes
1274/741 ... yes
1283/742 ... yes
383/743 ... yes
1246/744 ... yes
322/745 ... yes
342/746 ... yes
318/747 ... yes
587/748 ... yes
335/749 ... yes
1283/750 ... yes
1332/751 ... yes
1283/752 ... yes
382/753 ... yes
342/754 ... yes
1372/755 ... yes
1372/756 ... yes
1246/757 ... yes
231/758 ... yes
1274/759 ... yes
1372/760 ... yes
43/761 ... yes
1246/762 ... yes
383/763 ... yes
71/764 ... yes
1372/765 ... yes
1372/766 ... yes
1366/767 ... yes
1366/768 ... yes
1366/769 ... yes
574/770 ... yes
382/771 ... yes
1380/772 ... yes
1380/773 ... yes
1246/806 ... yes
1283/807 ... yes
1420/808 ... yes
573/809 ... yes
227/810 ... yes
1373/811 ... yes
1283/812 ... yes
384/813 ... yes
335/814 ... yes
136/815 ... yes
1431/816 ... yes
1433/817 ... yes
1435/818 ... yes
1438/819 ... yes
1438/820 ... yes
1441/822 ... yes
1444/823 ... yes
1444/824 ... yes
1447/825 ... yes
342/828 ... yes
1455/829 ... yes
1455/830 ... yes
1458/831 ... yes
1458/832 ... yes
1461/833 ... yes
1461/834 ... yes
1464/835 ... yes
1464/836 ... yes
1467/837 ... yes
1467/838 ... yes
1473/841 ... yes
1473/842 ... yes
322/843 ... yes
1293/844 ... yes
1293/845 ... yes
342/846 ... yes
383/847 ... yes
383/848 ... yes
1487/849 ... yes
342/850 ... yes
322/851 ... yes
579/852 ... yes
416/853 ... yes
43/854 ... yes
322/855 ... yes
342/856 ... yes
342/857 ... yes
1506/858 ... yes
1506/859 ... yes
53/860 ... yes
382/861 ... yes
1515/862 ... yes
1519/863 ... yes
351/864 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.7)
Git user has default SSH configuration? ... yes
Active users: ... 484
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes