backup_keep_time is not respected

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Sometimes backups are not rotated, which leads to backup failures due to free space reaching zero.

A side issue is that it's tedious to clean the mess as temporary files are right there along with the backup tar files. An enhancement would be to create the temp files in a temporary subdirectory of backup_path.

I've pushed the thing to the limit and keep only 24h, which should leave in most cases only one (but certainly at most two) backup files:

gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/var/archives/gitlab"
gitlab_rails['backup_keep_time'] = 86400

Here you can see there are two backup files standing there while a third one is attempted to be created (and failed):

# ls -ld /var/archives/gitlab
drwx------ 4 git root 4096 Jun 19 02:29 /var/archives/gitlab
# ls -l /var/archives/gitlab/
total 185167588
-rw-------  1 git git 71219701760 Jun 17 02:41 1497659206_2017_06_17_9.2.5-ee_gitlab_backup.tar
-rw-------  1 git git 48013590528 Jun 18 02:40 1497745825_2017_06_18_9.2.5-ee_gitlab_backup.tar
-rw-------  1 git git   118616064 Jun 19 02:29 1497832182_2017_06_19_9.2.5-ee_gitlab_backup.tar
-rw-------  1 git git   346960493 Jun 19 01:26 artifacts.tar.gz
-rw-r--r--  1 git git         159 Jun 19 02:29 backup_information.yml
-rw-------  1 git git   182103374 Jun 19 01:26 builds.tar.gz
drwxr-xr-x  2 git git        4096 Jun 19 01:17 db
-rw-------  1 git git    72063097 Jun 19 01:27 lfs.tar.gz
-rw-------  1 git git         274 Jun 19 01:26 pages.tar.gz
-rw-------  1 git git 65682194654 Jun 19 02:29 registry.tar.gz
drwx------ 24 git git        4096 Jun 19 01:19 repositories
-rw-------  1 git git  3976320174 Jun 19 01:25 uploads.tar.gz

FYI here are the mounts, just in case:

# mount | grep var
/dev/sda on /var/archives type ext4 (rw,relatime,discard,data=ordered)
/dev/sdb on /var/opt/gitlab type ext4 (rw,relatime,discard,data=ordered)

Steps to reproduce

No idea. Do a backup?

What is the current bug behavior?

Backups are not rotated.

What is the expected correct behavior?

Backups should be rotated.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
root@gitlab:~# gitlab-rake gitlab:env:info


System information
System:		Debian 8.8
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.11.1
Sidekiq Version:5.0.0


GitLab information
Version:	9.2.5-ee
Revision:	915a0aa
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.1
URL:		https://gitlab.adhoc-gti.com
HTTP Clone URL:	https://gitlab.adhoc-gti.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.adhoc-gti.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	no


GitLab Shell
Version:	5.0.4
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
root@gitlab:~# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...


GitLab Shell version >= 5.0.4 ? ... OK (5.0.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
14/1 ... ok
14/2 ... ok
14/3 ... ok
18/4 ... ok
14/5 ... ok
14/6 ... ok
14/7 ... ok
14/8 ... ok
14/9 ... ok
14/10 ... ok
14/11 ... ok
14/12 ... ok
14/13 ... ok
14/14 ... ok
14/15 ... ok
14/16 ... ok
14/17 ... ok
14/18 ... ok
14/19 ... ok
14/20 ... ok
14/21 ... ok
14/22 ... ok
14/23 ... ok
14/25 ... ok
14/26 ... ok
14/27 ... ok
14/28 ... ok
14/29 ... ok
14/30 ... ok
14/31 ... ok
14/32 ... ok
14/33 ... ok
14/34 ... ok
14/35 ... ok
14/36 ... ok
14/40 ... ok
14/42 ... ok
14/44 ... ok
14/49 ... ok
15/55 ... repository is empty
15/56 ... ok
18/57 ... ok
19/59 ... ok
14/78 ... ok
14/79 ... ok
14/82 ... ok
14/92 ... ok
14/94 ... ok
15/106 ... ok
15/109 ... ok
15/110 ... ok
9/111 ... ok
15/113 ... ok
15/114 ... ok
15/115 ... ok
15/117 ... ok
15/119 ... ok
14/120 ... ok
61/121 ... ok
22/127 ... ok
47/128 ... ok
15/129 ... ok
15/130 ... ok
14/132 ... ok
14/133 ... ok
14/134 ... ok
22/135 ... ok
33/136 ... ok
14/137 ... ok
14/138 ... ok
47/139 ... ok
15/142 ... ok
47/143 ... repository is empty
15/144 ... ok
14/145 ... ok
14/146 ... ok
14/149 ... ok
14/150 ... ok
23/151 ... ok
14/153 ... ok
22/156 ... ok
42/157 ... ok
14/158 ... ok
14/159 ... ok
14/160 ... ok
14/161 ... ok
14/162 ... ok
14/165 ... ok
15/166 ... ok
19/167 ... ok
9/168 ... ok
42/169 ... ok
6/171 ... ok
14/172 ... ok
18/174 ... ok
37/177 ... repository is empty
18/178 ... ok
18/179 ... ok
18/180 ... ok
14/181 ... ok
14/183 ... ok
14/184 ... ok
9/185 ... ok
23/187 ... ok
14/189 ... ok
14/190 ... ok
14/191 ... ok
14/192 ... ok
14/193 ... ok
14/195 ... ok
14/196 ... ok
14/197 ... ok
14/198 ... ok
14/199 ... ok
47/200 ... repository is empty
15/202 ... ok
14/203 ... ok
19/204 ... ok
14/205 ... ok
14/206 ... ok
14/208 ... ok
15/209 ... ok
6/210 ... ok
18/212 ... ok
6/213 ... ok
19/214 ... ok
14/216 ... ok
49/217 ... ok
11/226 ... ok
14/227 ... ok
47/228 ... ok
24/229 ... ok
51/231 ... ok
14/232 ... ok
47/233 ... ok
6/235 ... ok
11/236 ... ok
6/237 ... ok
6/238 ... ok
19/239 ... ok
6/240 ... ok
55/241 ... ok
50/246 ... ok
14/248 ... ok
15/249 ... ok
42/250 ... ok
14/251 ... ok
14/253 ... ok
15/254 ... ok
55/255 ... ok
14/256 ... ok
14/257 ... ok
14/258 ... repository is empty
18/259 ... ok
49/260 ... ok
15/261 ... ok
2/262 ... ok
63/270 ... ok
55/272 ... ok
6/273 ... ok
6/274 ... ok
6/275 ... ok
2/276 ... repository is empty
43/277 ... repository is empty
55/278 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped (omnibus-gitlab has no init script)
MailRoom running? ... can't check because of previous errors


Checking Reply by email ... Finished


Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
14/1 ... yes
14/2 ... yes
14/3 ... yes
18/4 ... yes
14/5 ... yes
14/6 ... yes
14/7 ... yes
14/8 ... yes
14/9 ... yes
14/10 ... yes
14/11 ... yes
14/12 ... yes
14/13 ... yes
14/14 ... yes
14/15 ... yes
14/16 ... yes
14/17 ... yes
14/18 ... yes
14/19 ... yes
14/20 ... yes
14/21 ... yes
14/22 ... yes
14/23 ... yes
14/25 ... yes
14/26 ... yes
14/27 ... yes
14/28 ... yes
14/29 ... yes
14/30 ... yes
14/31 ... yes
14/32 ... yes
14/33 ... yes
14/34 ... yes
14/35 ... yes
14/36 ... yes
14/40 ... yes
14/42 ... yes
14/44 ... yes
14/49 ... yes
15/55 ... yes
15/56 ... yes
18/57 ... yes
19/59 ... yes
14/78 ... yes
14/79 ... yes
14/82 ... yes
14/92 ... yes
14/94 ... yes
15/106 ... yes
15/109 ... yes
15/110 ... yes
9/111 ... yes
15/113 ... yes
15/114 ... yes
15/115 ... yes
15/117 ... yes
15/119 ... yes
14/120 ... yes
61/121 ... yes
22/127 ... yes
47/128 ... yes
15/129 ... yes
15/130 ... yes
14/132 ... yes
14/133 ... yes
14/134 ... yes
22/135 ... yes
33/136 ... yes
14/137 ... yes
14/138 ... yes
47/139 ... yes
15/142 ... yes
47/143 ... yes
15/144 ... yes
14/145 ... yes
14/146 ... yes
14/149 ... yes
14/150 ... yes
23/151 ... yes
14/153 ... yes
22/156 ... yes
42/157 ... yes
14/158 ... yes
14/159 ... yes
14/160 ... yes
14/161 ... yes
14/162 ... yes
14/165 ... yes
15/166 ... yes
19/167 ... yes
9/168 ... yes
42/169 ... yes
6/171 ... yes
14/172 ... yes
18/174 ... yes
37/177 ... yes
18/178 ... yes
18/179 ... yes
18/180 ... yes
14/181 ... yes
14/183 ... yes
14/184 ... yes
9/185 ... yes
23/187 ... yes
14/189 ... yes
14/190 ... yes
14/191 ... yes
14/192 ... yes
14/193 ... yes
14/195 ... yes
14/196 ... yes
14/197 ... yes
14/198 ... yes
14/199 ... yes
47/200 ... yes
15/202 ... yes
14/203 ... yes
19/204 ... yes
14/205 ... yes
14/206 ... yes
14/208 ... yes
15/209 ... yes
6/210 ... yes
18/212 ... yes
6/213 ... yes
19/214 ... yes
14/216 ... yes
49/217 ... yes
11/226 ... yes
14/227 ... yes
47/228 ... yes
24/229 ... yes
51/231 ... yes
14/232 ... yes
47/233 ... yes
6/235 ... yes
11/236 ... yes
6/237 ... yes
6/238 ... yes
19/239 ... yes
6/240 ... yes
55/241 ... yes
50/246 ... yes
14/248 ... yes
15/249 ... yes
42/250 ... yes
14/251 ... yes
14/253 ... yes
15/254 ... yes
55/255 ... yes
14/256 ... yes
14/257 ... yes
14/258 ... yes
18/259 ... yes
49/260 ... yes
15/261 ... yes
2/262 ... yes
63/270 ... yes
55/272 ... yes
6/273 ... yes
6/274 ... yes
6/275 ... yes
2/276 ... yes
43/277 ... yes
55/278 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.1)
Active users: 36


Checking GitLab ... Finished
Edited by 🤖 GitLab Bot 🤖