Skip to content

Port repository backup/restore to use GlRepository

James Fargher requested to merge port_repo_backup_to_gl_repo into master

What does this MR do?

  • Ports rake gitlab:backup:create and rake gitlab:backup:restore to use GlRepository
  • GlRepository allows us to generically interface with git repositories without needing to have any special logic to handle different repository types (project, wiki, design, snippet, etc)
  • Design repos will be handled in #233430 (closed)
  • Snippet repos will be handled in #233626 (closed)

Manual Testing

Create backup with this MR

$ time be rake gitlab:backup:create
2020-09-18 09:50:28 +1200 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_development ... [DONE]
2020-09-18 09:50:29 +1200 -- done
2020-09-18 09:50:29 +1200 -- Dumping repositories ...
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... 
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... [DONE]
 * gitlab-org/gitlab-test.wiki (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki) ... 
 * gitlab-org/gitlab-test.wiki (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki) ... [SKIPPED]
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... 
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [DONE]
 * gitlab-org/gitlab-shell.wiki (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki) ... 
 * gitlab-org/gitlab-shell.wiki (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki) ... [SKIPPED]
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... 
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... [DONE]
 * gnuwget/wget2.wiki (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki) ... 
 * gnuwget/wget2.wiki (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki) ... [SKIPPED]
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... 
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [DONE]
 * Commit451/lab-coat.wiki (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki) ... 
 * Commit451/lab-coat.wiki (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki) ... [SKIPPED]
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... 
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... [DONE]
 * jashkenas/underscore.wiki (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki) ... 
 * jashkenas/underscore.wiki (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki) ... [SKIPPED]
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... 
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [DONE]
 * flightjs/flight.wiki (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.wiki) ... 
 * flightjs/flight.wiki (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.wiki) ... [SKIPPED]
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... 
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [DONE]
 * twitter/typeahead-js.wiki (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.wiki) ... 
 * twitter/typeahead-js.wiki (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.wiki) ... [SKIPPED]
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... 
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [DONE]
 * h5bp/html5-boilerplate.wiki (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki) ... 
 * h5bp/html5-boilerplate.wiki (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki) ... [SKIPPED]
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... 
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [SKIPPED]
 * lakenya/gitlab-shell.wiki (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki) ... 
 * lakenya/gitlab-shell.wiki (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki) ... [SKIPPED]
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... 
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [SKIPPED]
 * reported_user_6/gitlab-shell.wiki (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.wiki) ... 
 * reported_user_6/gitlab-shell.wiki (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.wiki) ... [SKIPPED]
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... 
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [SKIPPED]
 * melvin_botsford/flight.wiki (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.wiki) ... 
 * melvin_botsford/flight.wiki (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.wiki) ... [SKIPPED]
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... 
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... [SKIPPED]
 * darrel.schinner/gitlab-shell.wiki (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918.wiki) ... 
 * darrel.schinner/gitlab-shell.wiki (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918.wiki) ... [SKIPPED]
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... 
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... [SKIPPED]
 * shelley_walker/underscore.wiki (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki) ... 
 * shelley_walker/underscore.wiki (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki) ... [SKIPPED]
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... 
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... [SKIPPED]
 * jesusa/gitlab-shell.wiki (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61.wiki) ... 
 * jesusa/gitlab-shell.wiki (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61.wiki) ... [SKIPPED]
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... 
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... [SKIPPED]
 * jannet/underscore.wiki (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb.wiki) ... 
 * jannet/underscore.wiki (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb.wiki) ... [SKIPPED]
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... 
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... [SKIPPED]
 * reported_user_18/underscore.wiki (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.wiki) ... 
 * reported_user_18/underscore.wiki (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.wiki) ... [SKIPPED]
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... 
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [SKIPPED]
 * reported_user_3/gitlab-shell.wiki (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki) ... 
 * reported_user_3/gitlab-shell.wiki (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki) ... [SKIPPED]
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... 
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... [SKIPPED]
 * reported_user_19/underscore.wiki (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.wiki) ... 
 * reported_user_19/underscore.wiki (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.wiki) ... [SKIPPED]
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... 
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... [DONE]
 * root/gitaly.wiki (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.wiki) ... 
 * root/gitaly.wiki (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.wiki) ... [DONE]
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... 
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... [DONE]
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring.wiki (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.wiki) ... 
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring.wiki (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.wiki) ... [SKIPPED]
2020-09-18 09:50:32 +1200 -- done
2020-09-18 09:50:32 +1200 -- Dumping uploads ... 
2020-09-18 09:50:32 +1200 -- done
2020-09-18 09:50:32 +1200 -- Dumping builds ... 
2020-09-18 09:50:33 +1200 -- done
2020-09-18 09:50:33 +1200 -- Dumping artifacts ... 
2020-09-18 09:50:33 +1200 -- done
2020-09-18 09:50:33 +1200 -- Dumping pages ... 
2020-09-18 09:50:33 +1200 -- done
2020-09-18 09:50:33 +1200 -- Dumping lfs objects ... 
2020-09-18 09:50:33 +1200 -- done
2020-09-18 09:50:33 +1200 -- Dumping container registry images ... 
2020-09-18 09:50:33 +1200 -- [DISABLED]
Creating backup archive: 1600379433_2020_09_18_13.4.0-pre_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

real	0m15.009s
user	0m9.291s
sys	0m2.469s

Create backup on master

$ time be rake gitlab:backup:create
2020-09-18 09:53:07 +1200 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_development ... [DONE]
2020-09-18 09:53:09 +1200 -- done
2020-09-18 09:53:09 +1200 -- Dumping repositories ...
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... 
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... [DONE]
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... [SKIPPED] Wiki
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... 
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [DONE]
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [SKIPPED] Wiki
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... 
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... [DONE]
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... [SKIPPED] Wiki
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... 
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [DONE]
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [SKIPPED] Wiki
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... 
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... [DONE]
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... [SKIPPED] Wiki
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... 
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [DONE]
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [SKIPPED] Wiki
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... 
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [DONE]
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [SKIPPED] Wiki
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... 
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [DONE]
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [SKIPPED] Wiki
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... 
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [SKIPPED]
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [SKIPPED] Wiki
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... 
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [SKIPPED]
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [SKIPPED] Wiki
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... 
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [SKIPPED]
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [SKIPPED] Wiki
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... 
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... [SKIPPED]
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... [SKIPPED] Wiki
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... 
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... [SKIPPED]
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... [SKIPPED] Wiki
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... 
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... [SKIPPED]
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... [SKIPPED] Wiki
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... 
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... [SKIPPED]
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... [SKIPPED] Wiki
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... 
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... [SKIPPED]
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... [SKIPPED] Wiki
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... 
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [SKIPPED]
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [SKIPPED] Wiki
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... 
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... [SKIPPED]
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... [SKIPPED] Wiki
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... 
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... [DONE]
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... [DONE] Wiki
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... 
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... [DONE]
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... [SKIPPED] Wiki
2020-09-18 09:53:11 +1200 -- done
2020-09-18 09:53:11 +1200 -- Dumping uploads ... 
2020-09-18 09:53:11 +1200 -- done
2020-09-18 09:53:11 +1200 -- Dumping builds ... 
2020-09-18 09:53:11 +1200 -- done
2020-09-18 09:53:11 +1200 -- Dumping artifacts ... 
2020-09-18 09:53:12 +1200 -- done
2020-09-18 09:53:12 +1200 -- Dumping pages ... 
2020-09-18 09:53:12 +1200 -- done
2020-09-18 09:53:12 +1200 -- Dumping lfs objects ... 
2020-09-18 09:53:12 +1200 -- done
2020-09-18 09:53:12 +1200 -- Dumping container registry images ... 
2020-09-18 09:53:12 +1200 -- [DISABLED]
Creating backup archive: 1600379592_2020_09_18_13.4.0-pre_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

real	0m13.915s
user	0m9.152s
sys	0m2.320s

Verify repository backups

Remove all the empty directories as the logic was not consistent previously and it will clutter up the diff:

$ find 1600379592_2020_09_18_13.4.0-pre_gitlab_backup/repositories -type d -empty -delete
$ find 1600379433_2020_09_18_13.4.0-pre_gitlab_backup/repositories -type d -empty -delete

Now diff repos:

$ diff -qr 1600379433_2020_09_18_13.4.0-pre_gitlab_backup/repositories 1600379592_2020_09_18_13.4.0-pre_gitlab_backup/repositories

No output == success 🎉

Restore backup with this MR

Now that we know the backups taken are identical we only need to restore once using this MR to verify it works. I setup a fresh GDK to restore the backup into.

$ cp ../../gdk/gitlab/tmp/backups/1600379433_2020_09_18_13.4.0-pre_gitlab_backup.tar tmp/backups/
$ be rake gitlab:backup:restore
Unpacking backup ... done
Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations

Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? yes
...
2020-09-18 10:51:47 +1200 -- Restoring repositories ...
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... 
 * gitlab-org/gitlab-test (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... [DONE]
 * gitlab-org/gitlab-test.wiki (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki) ... 
 * gitlab-org/gitlab-test.wiki (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki) ... [DONE]
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... 
 * gitlab-org/gitlab-shell (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [DONE]
 * gitlab-org/gitlab-shell.wiki (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki) ... 
 * gitlab-org/gitlab-shell.wiki (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki) ... [DONE]
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... 
 * gnuwget/wget2 (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce) ... [DONE]
 * gnuwget/wget2.wiki (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki) ... 
 * gnuwget/wget2.wiki (@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.wiki) ... [DONE]
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... 
 * Commit451/lab-coat (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [DONE]
 * Commit451/lab-coat.wiki (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki) ... 
 * Commit451/lab-coat.wiki (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki) ... [DONE]
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... 
 * jashkenas/underscore (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) ... [DONE]
 * jashkenas/underscore.wiki (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki) ... 
 * jashkenas/underscore.wiki (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.wiki) ... [DONE]
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... 
 * flightjs/flight (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [DONE]
 * flightjs/flight.wiki (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.wiki) ... 
 * flightjs/flight.wiki (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.wiki) ... [DONE]
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... 
 * twitter/typeahead-js (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [DONE]
 * twitter/typeahead-js.wiki (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.wiki) ... 
 * twitter/typeahead-js.wiki (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451.wiki) ... [DONE]
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... 
 * h5bp/html5-boilerplate (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [DONE]
 * h5bp/html5-boilerplate.wiki (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki) ... 
 * h5bp/html5-boilerplate.wiki (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki) ... [DONE]
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... 
 * lakenya/gitlab-shell (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [DONE]
 * lakenya/gitlab-shell.wiki (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki) ... 
 * lakenya/gitlab-shell.wiki (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki) ... [DONE]
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... 
 * reported_user_6/gitlab-shell (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [DONE]
 * reported_user_6/gitlab-shell.wiki (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.wiki) ... 
 * reported_user_6/gitlab-shell.wiki (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5.wiki) ... [DONE]
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... 
 * melvin_botsford/flight (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [DONE]
 * melvin_botsford/flight.wiki (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.wiki) ... 
 * melvin_botsford/flight.wiki (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.wiki) ... [DONE]
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... 
 * darrel.schinner/gitlab-shell (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918) ... [DONE]
 * darrel.schinner/gitlab-shell.wiki (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918.wiki) ... 
 * darrel.schinner/gitlab-shell.wiki (@hashed/6b/51/6b51d431df5d7f141cbececcf79edf3dd861c3b4069f0b11661a3eefacbba918.wiki) ... [DONE]
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... 
 * shelley_walker/underscore (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278) ... [DONE]
 * shelley_walker/underscore.wiki (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki) ... 
 * shelley_walker/underscore.wiki (@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki) ... [DONE]
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... 
 * jesusa/gitlab-shell (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61) ... [DONE]
 * jesusa/gitlab-shell.wiki (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61.wiki) ... 
 * jesusa/gitlab-shell.wiki (@hashed/85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61.wiki) ... [DONE]
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... 
 * jannet/underscore (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb) ... [DONE]
 * jannet/underscore.wiki (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb.wiki) ... 
 * jannet/underscore.wiki (@hashed/e6/29/e629fa6598d732768f7c726b4b621285f9c3b85303900aa912017db7617d8bdb.wiki) ... [DONE]
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... 
 * reported_user_18/underscore (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9) ... [DONE]
 * reported_user_18/underscore.wiki (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.wiki) ... 
 * reported_user_18/underscore.wiki (@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.wiki) ... [DONE]
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... 
 * reported_user_3/gitlab-shell (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [DONE]
 * reported_user_3/gitlab-shell.wiki (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki) ... 
 * reported_user_3/gitlab-shell.wiki (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3.wiki) ... [DONE]
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... 
 * reported_user_19/underscore (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a) ... [DONE]
 * reported_user_19/underscore.wiki (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.wiki) ... 
 * reported_user_19/underscore.wiki (@hashed/4e/c9/4ec9599fc203d176a301536c2e091a19bc852759b255bd6818810a42c5fed14a.wiki) ... [DONE]
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... 
 * root/gitaly (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767) ... [DONE]
 * root/gitaly.wiki (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.wiki) ... 
 * root/gitaly.wiki (@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.wiki) ... [DONE]
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... 
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b) ... [DONE]
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring.wiki (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.wiki) ... 
 * gitlab-instance-administrators-35c066fc/gitlab-self-monitoring.wiki (@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.wiki) ... [DONE]
 - Object pool @pools/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b...
 - Object pool @pools/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35...
 - Object pool @pools/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce...
2020-09-18 10:51:53 +1200 -- done
...

🎉

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by James Fargher

Merge request reports