Backup Error [Postgresql]

sudo gitlab-ctl reconfigure i have executed and rebooted machine many times

Summary

In gitlab i was not able to take backup it is showing PostgreSQL error, my version of gitlab is 11.9.1

gitlab-rake gitlab:backup:create
2019-04-22 10:26:26 +0530 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: Dumping the contents of table "user_agent_details" failed: PQgetCopyData() failed.
pg_dump: Error message from server: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
pg_dump: The command was: COPY public.user_agent_details (id, user_agent, ip_address, subject_id, subject_type, submitted, created_at, updated_at) TO stdout;
[FAILED]
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:50:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:99:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:db:create
(See full trace by running task with --trace)

Steps to reproduce

(How one can reproduce the issue - this is very important)

Relevant logs

Relevant logs 
==> /var/log/gitlab/prometheus/state <==

==> /var/log/gitlab/prometheus/current <==
2019-04-21_23:00:00.83857 level=info ts=2019-04-21T23:00:00.838432968Z caller=compact.go:416 component=tsdb msg="write block" mint=1555876800000 maxt=1555884000000 ulid=01D9121CRK704TQ66VED4PEJMA
2019-04-21_23:00:00.86307 level=info ts=2019-04-21T23:00:00.862994203Z caller=head.go:520 component=tsdb msg="head GC completed" duration=6.147048ms
2019-04-22_01:00:00.79736 level=info ts=2019-04-22T01:00:00.797263682Z caller=compact.go:416 component=tsdb msg="write block" mint=1555884000000 maxt=1555891200000 ulid=01D918X40FVWHKH20KF5J0HNKC
2019-04-22_01:00:00.82407 level=info ts=2019-04-22T01:00:00.823999469Z caller=head.go:520 component=tsdb msg="head GC completed" duration=7.079407ms
2019-04-22_01:00:02.12296 level=info ts=2019-04-22T01:00:02.122905922Z caller=head.go:567 component=tsdb msg="WAL checkpoint complete" first=521 last=522 duration=1.29886095s
2019-04-22_03:00:00.78917 level=info ts=2019-04-22T03:00:00.789083631Z caller=compact.go:416 component=tsdb msg="write block" mint=1555891200000 maxt=1555898400000 ulid=01D91FRV8F4CA65B6WBWESVJMW
2019-04-22_03:00:00.81710 level=info ts=2019-04-22T03:00:00.817036104Z caller=head.go:520 component=tsdb msg="head GC completed" duration=7.023603ms
2019-04-22_03:00:01.25606 level=info ts=2019-04-22T03:00:01.255985091Z caller=compact.go:369 component=tsdb msg="compact blocks" count=3 mint=1555869600000 maxt=1555891200000 ulid=01D91FRVNXTC0RTFWZ2C4ZTRD9 sources="[01D90V5NGAREDSB8N6RFDZKSK5 01D9121CRK704TQ66VED4PEJMA 01D918X40FVWHKH20KF5J0HNKC]" duration=426.352507ms
2019-04-22_05:00:00.86625 level=info ts=2019-04-22T05:00:00.866132834Z caller=compact.go:416 component=tsdb msg="write block" mint=1555898400000 maxt=1555905600000 ulid=01D91PMJGGB6PZM30NWA0PHE0J
2019-04-22_05:00:00.88997 level=info ts=2019-04-22T05:00:00.889896694Z caller=head.go:520 component=tsdb msg="head GC completed" duration=6.099903ms


==> /var/log/gitlab/sidekiq/state <==


==> /var/log/gitlab/sidekiq/current <==
2019-04-22_05:05:06.34130 2019-04-22T05:05:06.341Z 9293 TID-oxmo6ym9h UpdateHeadPipelineForMergeRequestWorker JID-7027a1ff7bd1a77715f5338c INFO: done: 0.045 sec
2019-04-22_05:05:06.35510 2019-04-22T05:05:06.354Z 9293 TID-oxmo6ynh1 CreateGpgSignatureWorker JID-2679a89492b55602f07a5510 INFO: done: 0.027 sec
2019-04-22_05:05:06.35733 2019-04-22T05:05:06.357Z 9293 TID-oxmo61x95 PostReceive JID-a143374625cdf2a4adef7d08 INFO: done: 0.179 sec
2019-04-22_05:05:06.48927 2019-04-22T05:05:06.489Z 9293 TID-oxmlyzep9 UpdateHeadPipelineForMergeRequestWorker JID-3c96280276b2df517e1c5ff7 INFO: start
2019-04-22_05:05:06.49027 2019-04-22T05:05:06.490Z 9293 TID-oxmo6chod UpdateHeadPipelineForMergeRequestWorker JID-352088b08b41e3c64bb431e4 INFO: start
2019-04-22_05:05:06.52711 2019-04-22T05:05:06.527Z 9293 TID-oxmo6chod UpdateHeadPipelineForMergeRequestWorker JID-352088b08b41e3c64bb431e4 INFO: done: 0.037 sec
2019-04-22_05:05:06.53024 2019-04-22T05:05:06.529Z 9293 TID-oxmlyzep9 UpdateHeadPipelineForMergeRequestWorker JID-3c96280276b2df517e1c5ff7 INFO: done: 0.04 sec
2019-04-22_05:05:06.71434 2019-04-22T05:05:06.714Z 9293 TID-oxmo6l40x UpdateMergeRequestsWorker JID-13c8ba9fb5dd9d05b192fa95 INFO: done: 0.509 sec
2019-04-22_05:05:44.74772 2019-04-22T05:05:44.747Z 9293 TID-oxmo66sc5 ProjectDailyStatisticsWorker JID-f81cc53e212dfe54afd5dc16 INFO: start
2019-04-22_05:05:44.80284 2019-04-22T05:05:44.802Z 9293 TID-oxmo66sc5 ProjectDailyStatisticsWorker JID-f81cc53e212dfe54afd5dc16 INFO: done: 0.055 sec


==> /var/log/gitlab/alertmanager/state <==


==> /var/log/gitlab/alertmanager/current <==


==> /var/log/gitlab/postgresql/state <==


==> /var/log/gitlab/postgresql/current <==
2019-04-22_04:56:51.06784 LOG:  invalid record length at 362/F60E6060: wanted 24, got 0
2019-04-22_04:56:51.06785 LOG:  redo done at 362/F60E6038
2019-04-22_04:56:51.06785 LOG:  last completed transaction was at log time 2019-04-22 04:56:49.638251+00
2019-04-22_04:56:51.57126 FATAL:  the database system is in recovery mode
2019-04-22_04:56:51.57263 FATAL:  the database system is in recovery mode
2019-04-22_04:56:52.10954 FATAL:  the database system is in recovery mode
2019-04-22_04:56:52.11163 FATAL:  the database system is in recovery mode
2019-04-22_04:56:52.57428 LOG:  MultiXact member wraparound protections are now enabled
2019-04-22_04:56:52.57749 LOG:  database system is ready to accept connections
2019-04-22_04:56:52.57778 LOG:  autovacuum launcher started


==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
time="2019-04-22T10:34:07+05:30" level=info msg="finished HTTP request" duration=0.03396627 gitaly_embedded=true method=POST pid=13114 url="http://127.0.0.1:8080/api/v4/internal/post_receive"
time="2019-04-22T10:34:47+05:30" level=info msg="finished HTTP request" duration=0.031810798 gitaly_embedded=false method=POST pid=13557 url="http://127.0.0.1:8080/api/v4/internal/allowed"
time="2019-04-22T10:34:47+05:30" level=info msg="executing git command" command="gitaly-upload-pack unix:/var/opt/gitlab/gitaly/gitaly.socket {"repository":{"storage_name":"default","relative_path":"john/wofox.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-116","gl_project_path":"john/wofox"},"gl_repository":"project-116","gl_project_path":"john/wofox","gl_id":"key-830","gl_username":"jenifer.a.tr","git_config_options":[],"git_protocol":null}" pid=13557 user="user with id key-830"
time="2019-04-22T10:34:52+05:30" level=info msg="finished HTTP request" duration=0.032423296 gitaly_embedded=false method=POST pid=13619 url="http://127.0.0.1:8080/api/v4/internal/allowed"
time="2019-04-22T10:34:52+05:30" level=info msg="executing git command" command="gitaly-upload-pack unix:/var/opt/gitlab/gitaly/gitaly.socket {"repository":{"storage_name":"default","relative_path":"john/wofox.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-116","gl_project_path":"john/wofox"},"gl_repository":"project-116","gl_project_path":"john/wofox","gl_id":"key-466","gl_username":"jenishadalin.rh.tr","git_config_options":[],"git_protocol":null}" pid=13619 user="user with id key-466"
time="2019-04-22T10:35:01+05:30" level=info msg="finished HTTP request" duration=0.047372477 gitaly_embedded=false method=POST pid=13746 url="http://127.0.0.1:8080/api/v4/internal/allowed"
time="2019-04-22T10:35:01+05:30" level=info msg="executing git command" command="gitaly-receive-pack unix:/var/opt/gitlab/gitaly/gitaly.socket {"repository":{"storage_name":"default","relative_path":"john/wofox.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-116","gl_project_path":"john/wofox"},"gl_repository":"project-116","gl_project_path":"john/wofox","gl_id":"key-830","gl_username":"jenifer.a.tr","git_config_options":[],"git_protocol":null}" pid=13746 user="user with id key-830"
time="2019-04-22T10:35:05+05:30" level=info msg="finished HTTP request" duration=0.514734763 gitaly_embedded=true method=POST pid=13798 url="http://127.0.0.1:8080/api/v4/internal/allowed"
time="2019-04-22T10:35:05+05:30" level=info msg="finished HTTP request" duration=0.018887217 gitaly_embedded=true method=POST pid=13798 url="http://127.0.0.1:8080/api/v4/internal/pre_receive"
time="2019-04-22T10:35:06+05:30" level=info msg="finished HTTP request" duration=0.036128587 gitaly_embedded=true method=POST pid=13819 url="http://127.0.0.1:8080/api/v4/internal/post_receive"


==> /var/log/gitlab/redis/state <==


==> /var/log/gitlab/redis/current <==
2019-04-22_04:57:44.02898 9259:M 22 Apr 10:27:44.028 * 10 changes in 300 seconds. Saving...
2019-04-22_04:57:44.03466 9259:M 22 Apr 10:27:44.034 * Background saving started by pid 9561
2019-04-22_04:57:45.95764 9561:C 22 Apr 10:27:45.957 * DB saved on disk
2019-04-22_04:57:45.96080 9561:C 22 Apr 10:27:45.960 * RDB: 13 MB of memory used by copy-on-write
2019-04-22_04:57:46.04249 9259:M 22 Apr 10:27:46.042 * Background saving terminated with success
2019-04-22_05:02:47.01464 9259:M 22 Apr 10:32:47.014 * 10 changes in 300 seconds. Saving...
2019-04-22_05:02:47.02035 9259:M 22 Apr 10:32:47.020 * Background saving started by pid 12351
2019-04-22_05:02:48.74313 12351:C 22 Apr 10:32:48.743 * DB saved on disk
2019-04-22_05:02:48.74633 12351:C 22 Apr 10:32:48.746 * RDB: 14 MB of memory used by copy-on-write
2019-04-22_05:02:48.82469 9259:M 22 Apr 10:32:48.824 * Background saving terminated with success

Details of package version

Provide the package version installation details 
rpm -qa | grep 'gitlab'
gitlab-ci-multi-runner-9.5.1-1.x86_64
gitlab-ce-11.9.1-ce.0.el6.x86_64

Environment details

  • Operating System: CentOS release 6.10 (Final)

  • Installation Target, remove incorrect values:

    • Bare Metal Machine

  • Installation Type, remove incorrect values:

    • Other: No New changes has been made

  • Is there any other software running on the machine: No

  • Is this a single or multiple node installation? Single Node

  • Resources

    • CPU: 4
    • Memory total: free total used free shared buffers cached Mem: 16166296 14473024 1693272 254268 420248 8802992 -/+ buffers/cache: 5249784 10916512 Swap: 8447996 64344 8383652

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
external_url 'http://XXXXXX'
 gitlab_rails['time_zone'] = 'Asia/Calcutta'
 gitlab_rails['gitlab_shell_ssh_port'] = XXXXX
 postgresql['shared_buffers'] = "4000MB"