Backup exits cleanly despite failing to backup repos
Summary
When creating a backup with rake it will exit with a clean exit code of 0 even if it fails to backup some repositories. This is terrible and very much unexpected behavior from a backup program.
Steps to reproduce
Kinda tricky to reproduce unless you have some way of making the backup for a certain repository fail?
What is the current bug behavior?
gitlab-rake gitlab:backup:create
exits with exit code 0 even if it hits errors dumping repositories.
What is the expected correct behavior?
When gitlab-rake gitlab:backup:create
encounters an error, any error affecting the integrity of the backup it should exit with an exit code greater than 0.
Relevant logs and/or screenshots
Excerpt from a run of gitlab-rake gitlab:backup:create
where there were errors dumping some repositories:
$ gitlab-rake gitlab:backup:create
(...)
* bjorn/arboxing ... [WARNING] Executing 4:Deadline Exceeded
Ignoring error on bjorn/arboxing - Failed to backup repo
[DONE]
(...)
$ echo $?
0
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.17.1 Sidekiq Version:5.1.3 Go Version: unknown
GitLab information Version: 11.1.2 Revision: 35936b0 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.bublar.com HTTP Clone URL: https://gitlab.bublar.com/some-group/some-project.git SSH Clone URL: git@gitlab.bublar:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 7.1.4 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 7.1.4 ? ... OK (7.1.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/2 ... ok 2/4 ... ok 5/5 ... ok 4/6 ... ok 6/7 ... ok 3/8 ... ok 7/9 ... ok 3/10 ... ok 2/12 ... ok 5/13 ... ok 3/14 ... ok 4/15 ... ok 7/16 ... ok 6/17 ... ok 9/19 ... ok 10/20 ... ok 5/21 ... ok 14/22 ... ok 18/24 ... ok 2/26 ... ok 3/27 ... ok 17/28 ... ok 18/29 ... ok 19/30 ... ok 22/32 ... ok 35/33 ... ok 18/34 ... ok 24/35 ... ok 16/36 ... ok 15/38 ... ok 16/39 ... ok 2/40 ... ok 24/41 ... ok 17/42 ... ok 17/43 ... ok 15/44 ... ok 23/45 ... ok 13/46 ... ok 26/47 ... ok 35/48 ... ok 2/49 ... ok 10/50 ... ok 2/54 ... ok 2/56 ... ok 4/57 ... ok 35/58 ... ok 17/59 ... ok 17/61 ... ok 16/62 ... ok 3/63 ... ok 14/65 ... ok 16/66 ... ok 2/67 ... ok 16/68 ... ok 15/69 ... ok 35/70 ... ok 35/71 ... ok 15/72 ... ok 35/73 ... ok 2/75 ... ok 4/76 ... ok 2/77 ... ok 14/78 ... ok 3/79 ... ok 17/80 ... ok 16/81 ... ok 13/83 ... ok 24/84 ... ok 23/85 ... ok 3/86 ... ok 2/87 ... ok 35/88 ... ok 35/89 ... ok 2/90 ... ok 23/91 ... ok 2/93 ... ok 2/94 ... ok 4/95 ... ok 14/96 ... ok 3/97 ... ok 16/98 ... ok 10/99 ... ok 15/100 ... ok 16/101 ... ok 2/102 ... ok 3/103 ... ok 14/104 ... ok 35/105 ... ok 32/106 ... ok 4/107 ... ok 17/108 ... ok 16/109 ... ok 16/110 ... ok 25/111 ... ok 15/112 ... ok 17/113 ... ok 2/114 ... ok 16/115 ... ok 35/116 ... ok 15/118 ... ok 13/119 ... ok 35/120 ... ok 16/121 ... ok 13/122 ... ok 2/123 ... ok 10/124 ... ok 4/125 ... ok 25/126 ... ok 15/127 ... ok 31/128 ... ok 34/129 ... ok 23/130 ... ok 4/131 ... ok 2/132 ... ok 2/133 ... ok 24/134 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) DN: uid=admin,cn=users,cn=accounts,dc=ipa,dc=bublar uid: admin DN: uid=joinadmin,cn=users,cn=accounts,dc=ipa,dc=bublar uid: joinadmin DN: uid=emilf,cn=users,cn=accounts,dc=ipa,dc=bublar uid: emilf DN: uid=perg,cn=users,cn=accounts,dc=ipa,dc=bublar uid: perg DN: uid=andreask,cn=users,cn=accounts,dc=ipa,dc=bublar uid: andreask DN: uid=adrianoo,cn=users,cn=accounts,dc=ipa,dc=bublar uid: adrianoo DN: uid=axell,cn=users,cn=accounts,dc=ipa,dc=bublar uid: axell DN: uid=bjorng,cn=users,cn=accounts,dc=ipa,dc=bublar uid: bjorng DN: uid=isabelleg,cn=users,cn=accounts,dc=ipa,dc=bublar uid: isabelleg DN: uid=josefineb,cn=users,cn=accounts,dc=ipa,dc=bublar uid: josefineb DN: uid=kennethh,cn=users,cn=accounts,dc=ipa,dc=bublar uid: kennethh DN: uid=noorar,cn=users,cn=accounts,dc=ipa,dc=bublar uid: noorar DN: uid=rasmusb,cn=users,cn=accounts,dc=ipa,dc=bublar uid: rasmusb DN: uid=sarab,cn=users,cn=accounts,dc=ipa,dc=bublar uid: sarab DN: uid=pers,cn=users,cn=accounts,dc=ipa,dc=bublar uid: pers DN: uid=mattiass,cn=users,cn=accounts,dc=ipa,dc=bublar uid: mattiass DN: uid=magnusg,cn=users,cn=accounts,dc=ipa,dc=bublar uid: magnusg DN: uid=christopherw,cn=users,cn=accounts,dc=ipa,dc=bublar uid: christopherw DN: uid=johne,cn=users,cn=accounts,dc=ipa,dc=bublar uid: johne DN: uid=apan,cn=users,cn=accounts,dc=ipa,dc=bublar uid: apan DN: uid=christopherp,cn=users,cn=accounts,dc=ipa,dc=bublar uid: christopherp DN: uid=wictorh,cn=users,cn=accounts,dc=ipa,dc=bublar uid: wictorh DN: uid=gitlab,cn=users,cn=accounts,dc=ipa,dc=bublar uid: gitlab DN: uid=antonq,cn=users,cn=accounts,dc=ipa,dc=bublar uid: antonq DN: uid=filippan,cn=users,cn=accounts,dc=ipa,dc=bublar uid: filippan DN: uid=moal,cn=users,cn=accounts,dc=ipa,dc=bublar uid: moal DN: uid=kaos,cn=users,cn=accounts,dc=ipa,dc=bublar uid: kaos DN: uid=josej,cn=users,cn=accounts,dc=ipa,dc=bublar uid: josej DN: uid=joelw,cn=users,cn=accounts,dc=ipa,dc=bublar uid: joelw DN: uid=ludvige,cn=users,cn=accounts,dc=ipa,dc=bublar uid: ludvige DN: uid=michaelt,cn=users,cn=accounts,dc=ipa,dc=bublar uid: michaelt DN: uid=joshw,cn=users,cn=accounts,dc=ipa,dc=bublar uid: joshw DN: uid=saram,cn=users,cn=accounts,dc=ipa,dc=bublar uid: saram
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/2 ... yes 2/4 ... yes 5/5 ... yes 4/6 ... yes 6/7 ... yes 3/8 ... yes 7/9 ... yes 3/10 ... yes 2/12 ... yes 5/13 ... yes 3/14 ... yes 4/15 ... yes 7/16 ... yes 6/17 ... yes 9/19 ... yes 10/20 ... yes 5/21 ... yes 14/22 ... yes 18/24 ... yes 2/26 ... yes 3/27 ... yes 17/28 ... yes 18/29 ... yes 19/30 ... yes 22/32 ... yes 35/33 ... yes 18/34 ... yes 24/35 ... yes 16/36 ... yes 15/38 ... yes 16/39 ... yes 2/40 ... yes 24/41 ... yes 17/42 ... yes 17/43 ... yes 15/44 ... yes 23/45 ... yes 13/46 ... yes 26/47 ... yes 35/48 ... yes 2/49 ... yes 10/50 ... yes 2/54 ... yes 2/56 ... yes 4/57 ... yes 35/58 ... yes 17/59 ... yes 17/61 ... yes 16/62 ... yes 3/63 ... yes 14/65 ... yes 16/66 ... yes 2/67 ... yes 16/68 ... yes 15/69 ... yes 35/70 ... yes 35/71 ... yes 15/72 ... yes 35/73 ... yes 2/75 ... yes 4/76 ... yes 2/77 ... yes 14/78 ... yes 3/79 ... yes 17/80 ... yes 16/81 ... yes 13/83 ... yes 24/84 ... yes 23/85 ... yes 3/86 ... yes 2/87 ... yes 35/88 ... yes 35/89 ... yes 2/90 ... yes 23/91 ... yes 2/93 ... yes 2/94 ... yes 4/95 ... yes 14/96 ... yes 3/97 ... yes 16/98 ... yes 10/99 ... yes 15/100 ... yes 16/101 ... yes 2/102 ... yes 3/103 ... yes 14/104 ... yes 35/105 ... yes 32/106 ... yes 4/107 ... yes 17/108 ... yes 16/109 ... yes 16/110 ... yes 25/111 ... yes 15/112 ... yes 17/113 ... yes 2/114 ... yes 16/115 ... yes 35/116 ... yes 15/118 ... yes 13/119 ... yes 35/120 ... yes 16/121 ... yes 13/122 ... yes 2/123 ... yes 10/124 ... yes 4/125 ... yes 25/126 ... yes 15/127 ... yes 31/128 ... yes 34/129 ... yes 23/130 ... yes 4/131 ... yes 2/132 ... yes 2/133 ... yes 24/134 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.4) Git version >= 2.9.5 ? ... yes (2.17.1) Git user has default SSH configuration? ... yes Active users: ... 23
Checking GitLab ... Finished