Rake task gitlab:backup:create failed at gitlab:backup:db:create with error missing chunk number 0 for toast value 409107 in pg_toast_380566

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

I am facing this error after upgrade from 8.14 to 10.4.8, We followed the path defined for the upgrade, like 8.13.4 -> 8.17.7 -> 9.5.8 -> 10.1.4 -> 10.8.4

Upgrade was successful and pg_upgrade was also successful. But We're facing issue while taking backup.

Here is the error:

# sudo gitlab-rake gitlab:backup:create --trace
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:backup:create
** Invoke gitlab:backup:db:create (first_time)
** Invoke gitlab_environment 
** Execute gitlab:backup:db:create
Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... pg_dump: Dumping the contents of table "merge_request_diff_files" failed: PQgetResult() failed.
pg_dump: Error message from server: ERROR:  missing chunk number 0 for toast value 409107 in pg_toast_380566
pg_dump: The command was: COPY public.merge_request_diff_files (merge_request_diff_id, relative_order, new_file, renamed_file, deleted_file, too_large, a_mode, b_mode, new_path, old_path, diff, "binary") TO stdout;
[FAILED]
Backup failed

I tried to REINDEX and VACUUM the table merge_request_diff_files but didn't helped much.

Here is the information for the GitLab omni environment:

# sudo gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.3.7p456
Gem Version:	2.6.14
Bundler Version:1.13.7
Rake Version:	12.3.1
Redis Version:	3.2.11
Git Version:	2.16.4
Sidekiq Version:5.0.5
Go Version:	unknown

GitLab information
Version:	10.8.4
Revision:	2268d0c
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab.example.com
HTTP Clone URL:	https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.example.com:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: saml

GitLab Shell
Version:	7.1.2
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

Can anyone help me? Thanks