Backup fails due to missing PostgreSQL directory
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=27418)
</details>
<!--IssueSummary end-->
[A customer](https://gitlab.zendesk.com/agent/tickets/116634) (internal link) is attempting to create a backup of his installation and getting the following error:
```bash
root@gitlab:/mnt/backups/configs/gitlab# sudo gitlab-rake gitlab:backup:create --trace
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:backup:create
** Invoke gitlab:backup:db:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:db:create
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: [archiver (db)] query failed: ERROR: could not open file "base/16400/2660": No such file or directory
pg_dump: [archiver (db)] query was: SELECT p.tableoid, p.oid, p.proname, p.prolang, p.pronargs, p.proargtypes, p.prorettype, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(p.proacl,pg_catalog.acldefault('f',p.proowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('f',p.proowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) AS proacl, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('f',p.proowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(p.proacl,pg_catalog.acldefault('f',p.proowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) AS rproacl, NULL AS initproacl, NULL AS initrproacl, p.pronamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = p.proowner) AS rolname FROM pg_proc p LEFT JOIN pg_init_privs pip ON (p.oid = pip.objoid AND pip.classoid = 'pg_proc'::regclass AND pip.objsubid = 0) WHERE NOT proisagg
AND NOT EXISTS (SELECT 1 FROM pg_depend WHERE classid = 'pg_proc'::regclass AND objid = p.oid AND deptype = 'i')
AND (
pronamespace != (SELECT oid FROM pg_namespace WHERE nspname = 'pg_catalog')
OR EXISTS (SELECT 1 FROM pg_cast
WHERE pg_cast.oid > 16383
AND p.oid = pg_cast.castfunc)
OR EXISTS (SELECT 1 FROM pg_transform
WHERE pg_transform.oid > 16383 AND
(p.oid = pg_transform.trffromsql
OR p.oid = pg_transform.trftosql))
OR p.proacl IS DISTINCT FROM pip.initprivs)
[FAILED]
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:47:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:99:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:db:create
```
It looks like the `base/16400/2660` directory is missing:
```bash
# This dir exists
root@gitlab:/home/xxxxx# ls -al /var/opt/gitlab/postgresql/data/base/16400
# This does not
root@gitlab:/home/xxxxx# ls -al /var/opt/gitlab/postgresql/data/base/16400/2660
ls: cannot access '/var/opt/gitlab/postgresql/data/base/16400/2660': No such file or directory
```
The customer also says that their last backup is over a year old and nobody noticed the problem for a year, so restoring from backup is not an option. They don't know how it could have gotten to this state since it happened so long ago.
The customer is also running 11.1.4-ee. Here is the section of their `gitlab.rb` that relates to either Postgres or backups:
```ruby
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/mnt/backups/gitlab"
gitlab_rails['backup_pg_schema'] = 'public'
gitlab_rails['backup_keep_time'] = 604800
git_data_dirs({
"default" => {
"path" => "/mnt/gitlab-nfs/git-data"
}
})
gitlab_rails['shared_path'] = '/mnt/gitlab-nfs'
gitlab_rails['uploads_directory'] = "/mnt/gitlab-nfs/uploads"
```
See [the ticket](https://gitlab.zendesk.com/agent/tickets/116634) for the full `gitlab.rb` they shared.
issue