GitLab helm chart backup utility fails backing up repositories
Summary
A GitLab Backup is run on a cron job every 12 hours via GitLab Helm chart toolbox pod. Since the upgrade to GitLab 16.3.1, the backup cron job fails.
Steps to reproduce
Gitlab backup output
# gitlab-backup create --trace
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:backup:create
2023-09-03 16:12:44 -0400 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2023-09-03 16:13:00 -0400 -- Dumping database ... done
2023-09-03 16:13:00 -0400 -- Dumping repositories ...
{"command":"create","gl_project_path":"<redacted_project_name>.wiki","level":"info","msg":"started create","pid":29478,"relative_path":"@groups/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.wiki.git","storage_name":"default","time":"2023-09-03T20:13:01.135Z"}
{"level":"info","msg":"[core] [Channel #1] Channel created","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] original dial target is: \"unix:/var/opt/gitlab/gitaly/gitaly.socket\"","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] parsed dial target is: {URL:{Scheme:unix Opaque: User: Host: Path:/var/opt/gitlab/gitaly/gitaly.socket RawPath: OmitHost:true ForceQuery:false RawQuery: Fragment: RawFragment:}}","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] Channel authority set to \"localhost\"","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"/var/opt/gitlab/gitaly/gitaly.socket\",\n \"ServerName\": \"\",\n \"Attributes\": {\n \"\\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\\u003e\": \"unix\"\n },\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n }\n ],\n \"ServiceConfig\": null,\n \"Attributes\": null\n} (resolver returned new addresses)","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] ignoring service config from resolver (\u003cnil\u003e) and applying the default because service config is disabled","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"level":"info","msg":"[core] [Channel #1] Channel switches to new LB policy \"round_robin\"","pid":29478,"system":"system","time":"2023-09-03T20:13:01.136Z"}
{"command":"create","gl_project_path":"<redacted_second_project_name>.wiki","level":"info","msg":"started create","pid":29478,"relative_path":"@groups/73/47/73475cb40a568e8da8a045ced110137e159f890ac4da883b6b17dc651b3a8049.wiki.git","storage_name":"default","time":"2023-09-03T20:13:01.136Z"}
2023-09-03 16:13:01 -0400 -- Deleting tar staging files ...
2023-09-03 16:13:01 -0400 -- Cleaning up /var/opt/gitlab/backups/db
2023-09-03 16:13:01 -0400 -- Deleting tar staging files ... done
2023-09-03 16:13:01 -0400 -- Deleting backups/tmp ...
2023-09-03 16:13:01 -0400 -- Deleting backups/tmp ... done
2023-09-03 16:13:01 -0400 -- Deleting backup and restore PID file ... done
rake aborted!
Backup::Error: gitaly-backup exit status
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:49:in `finish!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:30:in `ensure in dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:30:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:71:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:227:in `block in run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in `each_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in `run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:47:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:13:in `block in create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:62:in `lock_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:25:in `load'
/opt/gitlab/embedded/bin/rake:25:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:492:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.18/exe/bundle:37:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Caused by:
Errno::EPIPE: Broken pipe
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:112:in `write'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:112:in `puts'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:112:in `schedule_backup_job'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:57:in `enqueue'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:24:in `enqueue_group'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:38:in `block (2 levels) in enqueue_consecutive_groups'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:19:in `block in find_groups_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:71:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:229:in `loop'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-7.0.6/lib/active_record/relation/batches.rb:70:in `find_each'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:18:in `find_groups_in_batches'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:37:in `block in enqueue_consecutive_groups'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:248:in `allow_cross_joins_across_databases'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:36:in `enqueue_consecutive_groups'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/backup/repositories.rb:29:in `enqueue_consecutive'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:27:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:71:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:227:in `block in run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in `each_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:226:in `run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:47:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:13:in `block in create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:62:in `lock_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:101:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:25:in `load'
/opt/gitlab/embedded/bin/rake:25:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:492:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.18/exe/bundle:37:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:backup:createA GitLabSOS included in the ticket shows a 15 minute gap in the Gitaly logs during this time. The customer notes that there was a coredump at the time and Gitaly segfaulted. Also, strangely, when gitlab-backup is run against the repos that seemed to consistently fail, gitaly-backup actually works. (Output included in the `gitaly-backup-success*.txt
The customer opened the coredump in mdb and shared the following:
gitaly-backup coredump
# mdb core.gitaly-backup.86715
Loading modules: [ libc.so.1 ld.so.1 libc.so.6 ]
> ::status
debugging core file of gitaly-backup (64-bit) from gitlab-lx.local
file: /zones/8a874cdd-31e1-4071-b0ef-9175acd2eafe/root/opt/gitlab/embedded/bin/gitaly-backup
initial argv: /opt/gitlab/embedded/bin/gitaly-backup create -path /var/opt/gitlab/backups/repositories -layout pointer -id 1693854264_2023_09_04_16.3.1-ee
threading model: native threads
status: process terminated by SIGSEGV (Segmentation Fault), addr=0
> $C
000000c000932d50 google.golang.org/grpc/attributes.(*Attributes).String+0xc1()
000000c000932fa0 fmt.(*pp).handleMethods+0x2cb()
000000c000933180 fmt.(*pp).printValue+0xe5()
000000c000933360 fmt.(*pp).printValue+0x120a()
000000c000933540 fmt.(*pp).printValue+0x120a()
000000c0009335e0 fmt.(*pp).printArg+0x756()
000000c000933650 fmt.(*pp).doPrintln+0x3e()
000000c000933698 fmt.Sprintln+0x45()
000000c0009336e0 github.com/sirupsen/logrus.(*Entry).Logln+0x3e()
000000c000933718 github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.(*logrusGrpcLoggerV2).Infoln+0x38()
000000c000933750 google.golang.org/grpc/internal/grpclog.InfoDepth+0x58()
000000c0009337b8 google.golang.org/grpc/grpclog.(*componentData).Info+0x12d()
000000c000933d68 google.golang.org/grpc/balancer/base.(*baseBalancer).UpdateClientConnState+0x111()
000000c000933db8 google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState+0x82()
000000c000933ee0 google.golang.org/grpc.(*ccBalancerWrapper).updateClientConnState.func1+0xfa()
000000c000933fa8 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x194()
000000c000933fd0 google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1+0x2e()
0000000000000000 runtime.goexit.abi0+1()
> ::regs
%rax = 0x0000000000000000 %r8 = 0x0000000000000001
%rbx = 0x0000000000000000 %r9 = 0x000000c0006c8570
%rcx = 0x0000000000000008 %r10 = 0x000000c0006c8570
%rdx = 0x000000c0006c8571 %r11 = 0x0000000000000000
%rsi = 0x0000000000000000 %r12 = 0x0000000000000057
%rdi = 0x000000c0006c8570 %r13 = 0xffffffffffffffff
%r14 = 0x000000c00031c9c0
%r15 = 0x000000c000700000
%cs = 0x0053 %fs = 0x0000 %gs = 0x0000
%ds = 0x004b %es = 0x004b %ss = 0x004b
%rip = 0x00000000006a4641 google.golang.org/grpc/attributes.(*Attributes).String+0xc1
%rbp = 0x000000c000932d50
%rsp = 0x000000c000932c30
%rflags = 0x00010206
id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0
status=<of,df,IF,tf,sf,zf,af,PF,cf>
%gsbase = 0x0000000000000000
%fsbase = 0x00007fffb6600700
%trapno = 0xe
%err = 0x4
> ::stacks
mdb: couldn't find type uberdata_t: unknown object file name
mdb: CTF data is missing for uberdata_t; using current platform's offset for uberdata.all_lwps: unknown object file name
mdb: couldn't read frame for thread 0x2 at 7fffc6000280: no mapping for address
mdb: couldn't read frame for thread 0x3 at 74706f2f7261762f: no mapping for address
mdb: couldn't read frame for thread 0x6 at 7fffbe800280: no mapping for address
mdb: couldn't read frame for thread 0xf at 24: no mapping for address
THREAD STATE SOBJ COUNT
1 UNPARKED <NONE> 7
runtime.notesleep+0x87
runtime.stopm+0x8c
runtime.findRunnable+0xabc
runtime.schedule+0xb1
runtime.park_m+0x12d
runtime.mcall+0x43
runtime.gcBgMarkWorker+0xf1
runtime.goexit.abi0+1
b UNPARKED <NONE> 1
fmt.(*pp).handleMethods+0x2cb
fmt.(*pp).printValue+0xe5
fmt.(*pp).printValue+0x120a
fmt.(*pp).printValue+0x120a
fmt.(*pp).printArg+0x756
fmt.(*pp).doPrintln+0x3e
fmt.Sprintln+0x45
github.com/sirupsen/logrus.(*Entry).Logln+0x3e
github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.(*logrusGrpcLoggerV2).Infoln+0x38
google.golang.org/grpc/internal/grpclog.InfoDepth+0x58
google.golang.org/grpc/grpclog.(*componentData).Info+0x12d
google.golang.org/grpc/balancer/base.(*baseBalancer).UpdateClientConnState+0x111
google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState+0x82
google.golang.org/grpc.(*ccBalancerWrapper).updateClientConnState.func1+0xfa
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x194
google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.func1+0x2e
runtime.goexit.abi0+1
5 UNPARKED <NONE> 1
runtime.netpoll+0xd9
runtime.findRunnable+0x765
runtime.schedule+0xb1
runtime.goexit0+0x28e
runtime.mcall+0x43
runtime.goexit.abi0+1
3 UNPARKED <NONE> 1
runtime.notesleep+0x87
runtime.stopm+0x8c
runtime.findRunnable+0xabc
runtime.schedule+0xb1
runtime.goschedImpl+0xca
runtime.gopreempt_m+0x55
runtime.newstack+0x3cb
runtime.morestack.abi0+0x8b
0x24
0x2f62616c7469672f
c UNPARKED <NONE> 1
runtime.notesleep+0x87
runtime.stopm+0x8c
runtime.findRunnable+0xabc
runtime.schedule+0xb1
runtime.park_m+0x12d
runtime.mcall+0x43
runtime.bgsweep+0xde
runtime.gcenable.func1+0x26
runtime.goexit.abi0+1
6 UNPARKED <NONE> 1
runtime.notesleep+0x87
runtime.templateThread+0x71
runtime.mstart1+0x93
runtime.mstart0+0x7a
runtime.mstart.abi0+5
0x7fffbf1ffe80
10 UNPARKED <NONE> 1
runtime.runqgrab+0x85
runtime.runqsteal+0x3d
runtime.stealWork+0x292
runtime.findRunnable+0x33d
runtime.schedule+0xb1
runtime.park_m+0x12d
runtime.mcall+0x43
runtime.gcBgMarkWorker+0xf1
runtime.goexit.abi0+1
f UNPARKED <NONE> 1
runtime.runqgrab+0x85
runtime.runqsteal+0x3d
runtime.stealWork+0x292
runtime.findRunnable+0x33d
runtime.schedule+0xb1
runtime.park_m+0x12d
runtime.mcall+0x43
0xc0004f5e68
2 UNPARKED <NONE> 1
runtime.sysmon+0xa5
runtime.mstart1+0x93
runtime.mstart0+0x7a
runtime.mstart.abi0+5
0x7fffc69ffe80
e UNPARKED <NONE> 1
syscall.Syscall+0x48
syscall.read+0x45
internal/poll.(*FD).Read+0x2ce
os.(*File).Read+0x5e
encoding/json.(*Decoder).refill+0x188
encoding/json.(*Decoder).readValue+0xbb
encoding/json.(*Decoder).Decode+0x78
main.(*createSubcommand).Run+0x5e5
main.main+0x458
runtime.main+0x207
runtime.goexit.abi0+1
> ::mappings
BASE LIMIT SIZE NAME
400000 143f000 103f000 /opt/gitlab/embedded/bin/gitaly-backup
143f000 271e000 12df000 /opt/gitlab/embedded/bin/gitaly-backup
271e000 27d8000 ba000 /opt/gitlab/embedded/bin/gitaly-backup
27d8000 2840000 68000 [ heap ]
c000000000 c000001000 1000 [ unknown ]
c000002000 c000009000 7000 [ unknown ]
c00000f000 c000011000 2000 [ unknown ]
c000012000 c00001d000 b000 [ unknown ]
c000024000 c00002a000 6000 [ unknown ]
c000036000 c000050000 1a000 [ unknown ]
c000052000 c000057000 5000 [ unknown ]
c000058000 c000059000 1000 [ unknown ]
c00005a000 c000065000 b000 [ unknown ]
c000066000 c000067000 1000 [ unknown ]
c000068000 c00006b000 3000 [ unknown ]
c00006c000 c00006d000 1000 [ unknown ]
c00006e000 c00006f000 1000 [ unknown ]
c000070000 c000073000 3000 [ unknown ]
c000074000 c0000d3000 5f000 [ unknown ]
c0000da000 c0000e5000 b000 [ unknown ]
c0000ec000 c0000ef000 3000 [ unknown ]
c0000f5000 c000101000 c000 [ unknown ]
c000107000 c00010d000 6000 [ unknown ]
c000114000 c000115000 1000 [ unknown ]
c000116000 c000125000 f000 [ unknown ]
c000126000 c000169000 43000 [ unknown ]
c00016a000 c000181000 17000 [ unknown ]
c000188000 c00018f000 7000 [ unknown ]
c000190000 c000193000 3000 [ unknown ]
c000194000 c0001b5000 21000 [ unknown ]
c0001b8000 c0001c3000 b000 [ unknown ]
c0001c4000 c0001c5000 1000 [ unknown ]
c0001c6000 c0001d7000 11000 [ unknown ]
c0001d8000 c0001dd000 5000 [ unknown ]
c0001de000 c000217000 39000 [ unknown ]
c000218000 c00021d000 5000 [ unknown ]
c00022a000 c00022e000 4000 [ unknown ]
c000238000 c00023b000 3000 [ unknown ]
c00023e000 c00025c000 1e000 [ unknown ]
c000260000 c000283000 23000 [ unknown ]
c00028c000 c0002aa000 1e000 [ unknown ]
c0002ab000 c0002be000 13000 [ unknown ]
c0002c0000 c0002df000 1f000 [ unknown ]
c0002e0000 c0002e3000 3000 [ unknown ]
c0002e4000 c0002f7000 13000 [ unknown ]
c0002f8000 c000307000 f000 [ unknown ]
c000308000 c000309000 1000 [ unknown ]
c00030a000 c00030b000 1000 [ unknown ]
c00030c000 c000315000 9000 [ unknown ]
c00031b000 c000383000 68000 [ unknown ]
c000384000 c000385000 1000 [ unknown ]
c000386000 c000391000 b000 [ unknown ]
c000392000 c000393000 1000 [ unknown ]
c000394000 c00039b000 7000 [ unknown ]
c00039c000 c0003af000 13000 [ unknown ]
c0003b0000 c0003b1000 1000 [ unknown ]
c0003b2000 c0003b3000 1000 [ unknown ]
c0003b4000 c0003c3000 f000 [ unknown ]
c0003c4000 c0003d9000 15000 [ unknown ]
c0003dc000 c0003dd000 1000 [ unknown ]
c0003de000 c0003f7000 19000 [ unknown ]
c0003f8000 c0003ff000 7000 [ unknown ]
c000400000 c000407000 7000 [ unknown ]
c00040a000 c00040b000 1000 [ unknown ]
c00040c000 c00040d000 1000 [ unknown ]
c00040e000 c000435000 27000 [ unknown ]
c000436000 c00043b000 5000 [ unknown ]
c00043c000 c000443000 7000 [ unknown ]
c000446000 c000458000 12000 [ unknown ]
c00045a000 c00045b000 1000 [ unknown ]
c00045c000 c000463000 7000 [ unknown ]
c000464000 c000465000 1000 [ unknown ]
c000466000 c000469000 3000 [ unknown ]
c00046a000 c00046b000 1000 [ unknown ]
c00046c000 c000492000 26000 [ unknown ]
c000494000 c0004c3000 2f000 [ unknown ]
c0004c4000 c000560000 9c000 [ unknown ]
c00058f000 c000641000 b2000 [ unknown ]
c000642000 c000643000 1000 [ unknown ]
c000649000 c000653000 a000 [ unknown ]
c00065a000 c00065b000 1000 [ unknown ]
c000680000 c000681000 1000 [ unknown ]
c000682000 c000683000 1000 [ unknown ]
c000684000 c000685000 1000 [ unknown ]
c00068b000 c00068d000 2000 [ unknown ]
c00068e000 c000691000 3000 [ unknown ]
c000692000 c0006a0000 e000 [ unknown ]
c0006aa000 c0006c1000 17000 [ unknown ]
c0006c4000 c0006c5000 1000 [ unknown ]
c0006c6000 c0006c7000 1000 [ unknown ]
c0006c8000 c0006c9000 1000 [ unknown ]
c0006ca000 c0006cb000 1000 [ unknown ]
c0006cc000 c0006d1000 5000 [ unknown ]
c0006d2000 c0006d3000 1000 [ unknown ]
c0006d4000 c0006d7000 3000 [ unknown ]
c0006d8000 c0006d9000 1000 [ unknown ]
c0006da000 c0006dd000 3000 [ unknown ]
c0006de000 c0006df000 1000 [ unknown ]
c0006e0000 c0006e1000 1000 [ unknown ]
c0006e2000 c0006e3000 1000 [ unknown ]
c0006e4000 c0006ed000 9000 [ unknown ]
c0006f6000 c0006f7000 1000 [ unknown ]
c0006f8000 c0006fb000 3000 [ unknown ]
c0006fc000 c0006ff000 3000 [ unknown ]
c000700000 c000701000 1000 [ unknown ]
c000702000 c000703000 1000 [ unknown ]
c000704000 c000705000 1000 [ unknown ]
c00070b000 c00070c000 1000 [ unknown ]
c00070e000 c00070f000 1000 [ unknown ]
c000780000 c00078b000 b000 [ unknown ]
c00078c000 c00078f000 3000 [ unknown ]
c000790000 c000791000 1000 [ unknown ]
c000792000 c00079b000 9000 [ unknown ]
c00079c000 c00079d000 1000 [ unknown ]
c00079e000 c00079f000 1000 [ unknown ]
c0007a0000 c0007a1000 1000 [ unknown ]
c0007a2000 c0007a5000 3000 [ unknown ]
c0007aa000 c0007b9000 f000 [ unknown ]
c0007ba000 c0007bd000 3000 [ unknown ]
c0007c0000 c0007c5000 5000 [ unknown ]
c0007c6000 c0007c7000 1000 [ unknown ]
c0007c8000 c0007ca000 2000 [ unknown ]
c000800000 c000801000 1000 [ unknown ]
c000807000 c000809000 2000 [ unknown ]
c00080f000 c000820000 11000 [ unknown ]
c000880000 c00088c000 c000 [ unknown ]
c000900000 c000907000 7000 [ unknown ]
c000908000 c00090d000 5000 [ unknown ]
c00090e000 c000911000 3000 [ unknown ]
c000912000 c000915000 3000 [ unknown ]
c000916000 c000919000 3000 [ unknown ]
c00091a000 c00091d000 3000 [ unknown ]
c00091e000 c000929000 b000 [ unknown ]
c00092a000 c00092b000 1000 [ unknown ]
c00092c000 c00092d000 1000 [ unknown ]
c000930000 c000931000 1000 [ unknown ]
c000932000 c000934000 2000 [ unknown ]
c000980000 c000981000 1000 [ unknown ]
c000982000 c00098d000 b000 [ unknown ]
c00098e000 c00098f000 1000 [ unknown ]
c000990000 c000991000 1000 [ unknown ]
c000992000 c000993000 1000 [ unknown ]
c000994000 c000995000 1000 [ unknown ]
c000996000 c000997000 1000 [ unknown ]
c000998000 c000999000 1000 [ unknown ]
c00099a000 c00099b000 1000 [ unknown ]
c00099c000 c0009a5000 9000 [ unknown ]
c000a00000 c000a01000 1000 [ unknown ]
c000a08000 c000a09000 1000 [ unknown ]
c000a0a000 c000a0b000 1000 [ unknown ]
c000a0c000 c000a0d000 1000 [ unknown ]
c000a0e000 c000a11000 3000 [ unknown ]
c000a80000 c000a81000 1000 [ unknown ]
c000a88000 c000a89000 1000 [ unknown ]
c000b00000 c000b01000 1000 [ unknown ]
c000b02000 c000b03000 1000 [ unknown ]
c000b04000 c000b05000 1000 [ unknown ]
c000b06000 c000b07000 1000 [ unknown ]
c000b08000 c000b09000 1000 [ unknown ]
c000b0a000 c000b0d000 3000 [ unknown ]
c000b0e000 c000b0f000 1000 [ unknown ]
c000b10000 c000b11000 1000 [ unknown ]
7fff50000000 7fff50001000 1000 [ unknown ]
7fff58000000 7fff58001000 1000 [ unknown ]
7fff60000000 7fff60001000 1000 [ unknown ]
7fff68000000 7fff68001000 1000 [ unknown ]
7fff70000000 7fff70001000 1000 [ unknown ]
7fff78000000 7fff78001000 1000 [ unknown ]
7fff80000000 7fff80001000 1000 [ unknown ]
7fff88000000 7fff88001000 1000 [ unknown ]
7fff90000000 7fff90001000 1000 [ unknown ]
7fff98000000 7fff98001000 1000 [ unknown ]
7fffa0000000 7fffa0001000 1000 [ unknown ]
7fffa8000000 7fffa8001000 1000 [ unknown ]
7fffadbfe000 7fffadbff000 1000 [ unknown ]
7fffae5ff000 7fffae601000 2000 [ unknown ]
7fffaf1ff000 7fffaf201000 2000 [ unknown ]
7fffafdff000 7fffafe01000 2000 [ unknown ]
7fffb0000000 7fffb0001000 1000 [ unknown ]
7fffb41ff000 7fffb4200000 1000 [ unknown ]
7fffb43fe000 7fffb43ff000 1000 [ unknown ]
7fffb4dff000 7fffb4e01000 2000 [ unknown ]
7fffb59ff000 7fffb5a01000 2000 [ unknown ]
7fffb65fd000 7fffb6601000 4000 [ unknown ]
7fffb71fd000 7fffb7201000 4000 [ unknown ]
7fffb7dff000 7fffb7e01000 2000 [ unknown ]
7fffb8000000 7fffb8001000 1000 [ unknown ]
7fffbc203000 7fffbc204000 1000 [ unknown ]
7fffbc402000 7fffbc403000 1000 [ unknown ]
7fffbc601000 7fffbc602000 1000 [ unknown ]
7fffbc800000 7fffbc801000 1000 [ unknown ]
7fffbc9ff000 7fffbca00000 1000 [ unknown ]
7fffbcbfe000 7fffbcbff000 1000 [ unknown ]
7fffbd5ff000 7fffbd601000 2000 [ unknown ]
7fffbd9fe000 7fffbd9ff000 1000 [ unknown ]
7fffbe3ff000 7fffbe401000 2000 [ unknown ]
7fffbe7fe000 7fffbe7ff000 1000 [ unknown ]
7fffbf1ff000 7fffbf201000 2000 [ unknown ]
7fffbfdff000 7fffbfe01000 2000 [ unknown ]
7fffc0000000 7fffc0001000 1000 [ unknown ]
7fffc41ff000 7fffc4200000 1000 [ unknown ]
7fffc43fe000 7fffc43ff000 1000 [ unknown ]
7fffc4dff000 7fffc4e01000 2000 [ unknown ]
7fffc51fe000 7fffc51ff000 1000 [ unknown ]
7fffc5bff000 7fffc5c01000 2000 [ unknown ]
7fffc5ffe000 7fffc5fff000 1000 [ unknown ]
7fffc69ff000 7fffc6a01000 2000 [ unknown ]
7fffc6b3b000 7fffc6b3c000 1000 [ unknown ]
7fffc6b3d000 7fffc6b3e000 1000 [ unknown ]
7fffc6b3f000 7fffc6b40000 1000 [ unknown ]
7fffc6b7e000 7fffc6b7f000 1000 [ unknown ]
7fffc6bfd000 7fffc6bff000 2000 [ unknown ]
7fffc6c00000 7fffc6c01000 1000 [ unknown ]
7fffc7c00000 7fffc7c01000 1000 [ unknown ]
7fffc7c18000 7fffc7c19000 1000 [ unknown ]
7fffc8bff000 7fffc8c00000 1000 [ unknown ]
7fffc8c39000 7fffc8c4a000 11000 [ unknown ]
7fffc8c7a000 7fffc8ca4000 2a000 [ unknown ]
7fffc8ca5000 7fffc8cb9000 14000 [ unknown ]
7fffc8cbb000 7fffc8cfb000 40000 [ unknown ]
7fffc8d3b000 7fffc8d3c000 1000 [ unknown ]
7fffc8d3d000 7fffc8d40000 3000 [ unknown ]
7fffc8d7e000 7fffc8d80000 2000 [ unknown ]
7fffc8dfd000 7fffc8dff000 2000 [ unknown ]
7fffc9206000 7fffc9207000 1000 [ unknown ]
7fffc9620000 7fffc9623000 3000 [ unknown ]
7fffc9677000 7fffc9679000 2000 [ unknown ]
7fffc967a000 7fffc96b0000 36000 [ unknown ]
7fffc96b1000 7fffc96b2000 1000 [ unknown ]
7fffc96b3000 7fffc96b8000 5000 [ unknown ]
7fffc96bb000 7fffc96fa000 3f000 [ unknown ]
7fffc96fc000 7fffc972a000 2e000 [ unknown ]
7fffc972b000 7fffc973b000 10000 [ unknown ]
7fffc973d000 7fffc973f000 2000 [ unknown ]
7fffc97bc000 7fffc97be000 2000 [ unknown ]
7fffc97fd000 7fffc97ff000 2000 [ unknown ]
7fffd9980000 7fffd9981000 1000 [ unknown ]
7fffe9810000 7fffe9811000 1000 [ unknown ]
7fffe9820000 7fffe9821000 1000 [ unknown ]
7fffe9830000 7fffe9831000 1000 [ unknown ]
7fffe9839000 7fffe983a000 1000 [ unknown ]
7fffe983b000 7fffe983d000 2000 [ unknown ]
7fffe98b8000 7fffe98ba000 2000 [ unknown ]
7fffe98f9000 7fffe98fb000 2000 [ unknown ]
7fffe98fc000 7fffe9900000 4000 [ unknown ]
7fffe997b000 7fffe997d000 2000 [ unknown ]
7fffe997e000 7fffe9980000 2000 [ unknown ]
7fffe9981000 7fffe9990000 f000 [ unknown ]
7fffe99bf000 7fffe99c0000 1000 [ unknown ]
7fffe99c2000 7fffe99fe000 3c000 [ unknown ]
7fffeba30000 7fffeba31000 1000 [ unknown ]
7fffeda10000 7fffeda11000 1000 [ unknown ]
7fffeda20000 7fffeda23000 3000 [ unknown ]
7fffeda7b000 7fffeda7c000 1000 [ unknown ]
7fffedabc000 7fffedabd000 1000 [ unknown ]
7fffedabe000 7fffedaec000 2e000 [ unknown ]
7fffedaed000 7fffedaf1000 4000 [ unknown ]
7fffedaff000 7fffedb00000 1000 [ unknown ]
7fffee006000 7fffee007000 1000 [ unknown ]
7fffee410000 7fffee411000 1000 [ unknown ]
7fffee420000 7fffee421000 1000 [ unknown ]
7fffee46c000 7fffee46e000 2000 [ unknown ]
7fffee4ad000 7fffee4af000 2000 [ unknown ]
7fffee4b0000 7fffee4c0000 10000 [ unknown ]
7fffee4d0000 7fffee4e0000 10000 [ unknown ]
7fffee4ea000 7fffee4ff000 15000 [ unknown ]
7fffee501000 7fffee507000 6000 [ unknown ]
7fffee508000 7fffee50b000 3000 [ unknown ]
7fffee50c000 7fffee50d000 1000 [ unknown ]
7fffee50e000 7fffee50f000 1000 [ unknown ]
7fffee510000 7fffee511000 1000 [ unknown ]
7fffee512000 7fffee513000 1000 [ unknown ]
7fffee514000 7fffee515000 1000 [ unknown ]
7fffee516000 7fffee517000 1000 [ unknown ]
7fffee5ea000 7fffee5eb000 1000 [ unknown ]
7fffee5ee000 7fffee5f1000 3000 [ unknown ]
7fffee5fe000 7fffee5ff000 1000 [ unknown ]
7fffee600000 7fffee7bb000 1bb000 /lib64/libc.so.6
7fffee7bb000 7fffee7ff000 44000 [ unknown ]
7fffee9bb000 7fffee9bf000 4000 /lib64/libc.so.6
7fffee9bf000 7fffee9c1000 2000 /lib64/libc.so.6
7fffee9c1000 7fffee9c3000 2000 /lib64/libc.so.6
7fffee9c4000 7fffee9c5000 1000 /lib64/libc.so.6
7fffee9df000 7fffee9ff000 20000 [ unknown ]
7fffeea00000 7fffeea1b000 1b000 /lib64/libpthread.so.0
7fffeea1b000 7fffeea25000 a000 [ unknown ]
7fffeec1a000 7fffeec1b000 1000 /lib64/libpthread.so.0
7fffeec1b000 7fffeec1c000 1000 /lib64/libpthread.so.0
7fffeec1f000 7fffeec20000 1000 /lib64/libpthread.so.0
7fffeec30000 7fffeec31000 1000 [ unknown ]
7fffeec7e000 7fffeec7f000 1000 [ unknown ]
7fffeec80000 7fffeec90000 10000 [ unknown ]
7fffeeca0000 7fffeeca3000 3000 [ unknown ]
7fffeed3e000 7fffeed3f000 1000 [ unknown ]
7fffeedbf000 7fffeedc4000 5000 [ unknown ]
7fffeedc6000 7fffeedd6000 10000 [ unknown ]
7fffeee00000 7fffeee14000 14000 /lib64/libresolv.so.2
7fffeee14000 7fffeee17000 3000 [ unknown ]
7fffef014000 7fffef015000 1000 /lib64/libresolv.so.2
7fffef015000 7fffef016000 1000 /lib64/libresolv.so.2
7fffef03f000 7fffef04f000 10000 [ unknown ]
7fffef050000 7fffef052000 2000 [ unknown ]
7fffef060000 7fffef1d7000 177000 LMfd`/native/lib/amd64/libc.so.1
7fffef1e7000 7fffef1f3000 c000 LMfd`/native/lib/amd64/libc.so.1
7fffef1f3000 7fffef1f7000 4000 LMfd`/native/lib/amd64/libc.so.1
7fffef200000 7fffef22e000 2e000 /lib64/ld-linux-x86-64.so.2
7fffef240000 7fffef242000 2000 [ unknown ]
7fffef29d000 7fffef29f000 2000 [ unknown ]
7fffef2a0000 7fffef2a1000 1000 [ unknown ]
7fffef2ad000 7fffef2ae000 1000 [ unknown ]
7fffef2b0000 7fffef2b6000 6000 [ unknown ]
7fffef2c0000 7fffef2c1000 1000 [ unknown ]
7fffef2d0000 7fffef35b000 8b000 LMfd`/native/lib/amd64/libnsl.so.1
7fffef36b000 7fffef36e000 3000 LMfd`/native/lib/amd64/libnsl.so.1
7fffef36e000 7fffef376000 8000 LMfd`/native/lib/amd64/libnsl.so.1
7fffef380000 7fffef381000 1000 [ unknown ]
7fffef390000 7fffef399000 9000 LMfd`/native/lib/amd64/librpcsvc.so.1
7fffef3a9000 7fffef3aa000 1000 LMfd`/native/lib/amd64/librpcsvc.so.1
7fffef3b0000 7fffef3b1000 1000 [ unknown ]
7fffef3c0000 7fffef3c1000 1000 [ unknown ]
7fffef3d0000 7fffef3d2000 2000 LMfd`/native/usr/lib/amd64/libmapmalloc.so.1
7fffef3e2000 7fffef3e3000 1000 LMfd`/native/usr/lib/amd64/libmapmalloc.so.1
7fffef3f0000 7fffef3f1000 1000 [ unknown ]
7fffef400000 7fffef401000 1000 [ unknown ]
7fffef410000 7fffef411000 1000 [ unknown ]
7fffef420000 7fffef421000 1000 [ unknown ]
7fffef42e000 7fffef42f000 1000 /lib64/ld-linux-x86-64.so.2
7fffef42f000 7fffef431000 2000 /lib64/ld-linux-x86-64.so.2
7fffef450000 7fffef451000 1000 [ unknown ]
7fffef460000 7fffef461000 1000 [ unknown ]
7fffef46c000 7fffef46e000 2000 linux-vdso.so.1
7fffef470000 7fffef471000 1000 [ unknown ]
7fffef477000 7fffef4c7000 50000 LMfd`/native/lib/amd64/ld.so.1
7fffef4c7000 7fffef4ca000 3000 LMfd`/native/lib/amd64/ld.so.1
7fffef4ca000 7fffef4cc000 2000 LMfd`/native/lib/amd64/ld.so.1
7fffef4cd000 7fffef4ea000 1d000 LMfd`/native/usr/lib/amd64/lx_brand.so.1
7fffef4fa000 7fffef4fe000 4000 LMfd`/native/usr/lib/amd64/lx_brand.so.1
7fffef4fe000 7fffef4ff000 1000 LMfd`/native/usr/lib/amd64/lx_brand.so.1
7fffffefb000 7ffffff00000 5000 [ stack ]
The coredump points to a problem with the logrus go library.
The customer investigated further in the debugger and shared the following (notes from the customer are included inline)
More coredump output
# mdb core.gitaly-backup.86715
// Find the address of the crashed function:
> ::nm ! fgrep 'google.golang.org/grpc/attributes.(*Attributes).String'
0x00000000006a4580|0x00000000000004c5|FUNC |GLOB |0x0 |1 |google.golang.org/grpc/attributes.(*Attributes).String
// Disassemble the crashed function:
> 0x00000000006a4580::dis
google.golang.org/grpc/attributes.(*Attributes).String: leaq -0xa8(%rsp),%r12
google.golang.org/grpc/attributes.(*Attributes).String+8: cmpq 0x10(%r14),%r12
google.golang.org/grpc/attributes.(*Attributes).String+0xc: jbe +0x49a <google.golang.org/grpc/attributes.(*Attributes).String+0x4ac>
google.golang.org/grpc/attributes.(*Attributes).String+0x12: subq $0x128,%rsp
google.golang.org/grpc/attributes.(*Attributes).String+0x19: movq %rbp,0x120(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x21: leaq 0x120(%rsp),%rbp
google.golang.org/grpc/attributes.(*Attributes).String+0x29: movups %xmm15,0x80(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x32: movups %xmm15,0x90(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x3b: nop
google.golang.org/grpc/attributes.(*Attributes).String+0x3c: leaq 0x80(%rsp),%rdx
google.golang.org/grpc/attributes.(*Attributes).String+0x44: movq %rdx,0x80(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x4c: movq 0x90(%rsp),%rbx
google.golang.org/grpc/attributes.(*Attributes).String+0x54: leaq 0x1(%rbx),%r8
google.golang.org/grpc/attributes.(*Attributes).String+0x58: movq 0x98(%rsp),%rcx
google.golang.org/grpc/attributes.(*Attributes).String+0x60: movq 0x88(%rsp),%r9
google.golang.org/grpc/attributes.(*Attributes).String+0x68: cmpq %r8,%rcx
google.golang.org/grpc/attributes.(*Attributes).String+0x6b: jae +0x37 <google.golang.org/grpc/attributes.(*Attributes).String+0xa4>
google.golang.org/grpc/attributes.(*Attributes).String+0x6d: movq %rax,0x130(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x75: movq %rbx,0x48(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0x7a: movq %r9,%rax
google.golang.org/grpc/attributes.(*Attributes).String+0x7d: movq %r8,%rbx
google.golang.org/grpc/attributes.(*Attributes).String+0x80: movl $0x1,%edi
google.golang.org/grpc/attributes.(*Attributes).String+0x85: leaq +0xe937b4(%rip),%rsi <0x1537dc0>
google.golang.org/grpc/attributes.(*Attributes).String+0x8c: call -0x253491 <runtime.growslice>
google.golang.org/grpc/attributes.(*Attributes).String+0x91: movq %rax,%r9
google.golang.org/grpc/attributes.(*Attributes).String+0x94: movq %rbx,%r8
google.golang.org/grpc/attributes.(*Attributes).String+0x97: movq 0x130(%rsp),%rax
google.golang.org/grpc/attributes.(*Attributes).String+0x9f: movq 0x48(%rsp),%rbx
google.golang.org/grpc/attributes.(*Attributes).String+0xa4: movb $0x7b,(%r9,%rbx)
google.golang.org/grpc/attributes.(*Attributes).String+0xa9: movq %r8,0x90(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0xb1: movq %rcx,0x98(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0xb9: movq %r9,0x88(%rsp)
google.golang.org/grpc/attributes.(*Attributes).String+0xc1: movq (%rax),%rbx
…
// The last instruction above is the one that crashed; it was attempting to read from the memory address stored in %rax.
// %rax = 0x0, so the crash was due to a null dereference.
// %rax is set by the `movq 0x130(%rsp),%rax` instruction a few lines above, which copies the 64 bits at (%rsp + 0x130) to %rax.
> (0x000000c000932c30 + 0x130)::dump
\/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef
c000932d60: 00000000 00000000 c0a36b01 00000000 ..........k.....
// Indeed, the first 64 bits are zero. Why are they zero? What should they be?
// https://github.com/grpc/grpc-go/blob/8eb4ac4c1514c190ee0b5d01a91c63218dac93c0/attributes/attributes.go#L107-L122 seems to be the Go source to the crashed function.
// https://go.googlesource.com/go/+/refs/heads/dev.regabi/src/cmd/compile/internal-abi.md#function-call-argument-and-result-passing describes the stack layout, but I haven't tried to work through it.One particular note from the customer stands out: %rax = 0x0, so the crash was due to a null dereference. The customer also noted a particular log that was output during the successful runs:
{"level":"info","msg":"[balancer] base.baseBalancer: got new ClientConn state: {{[{Addr: \"/var/opt/gitlab/gitaly/gitaly.socket\", ServerName: \"\", Attributes: {\"\u003c%!p(networktype.keyType=grpc.internal.transport.networktype)\u003e\": \"unix\" }, }] \u003cnil\u003e \u003cnil\u003e} \u003cnil\u003e}","pid":58264,"system":"system","time":"2023-09-05T20:25:59.217Z"}Tucked in that logging output are a few <nil>s (\u003c(<) and \u003e(>) and nil). The %!p in the Attributes makes me believe that there's no pointer. The customer even notes there's a null dereference.
They point this out in the stack trace:
000000c0009337b8 google.golang.org/grpc/grpclog.(*componentData).Info+0x12d()
000000c000933d68 google.golang.org/grpc/balancer/base.(*baseBalancer).UpdateClientConnState+0x111()And link to this line of code: https://github.com/grpc/grpc-go/blob/8eb4ac4c1514c190ee0b5d01a91c63218dac93c0/balancer/base/balancer.go#L98
At this point, I believe the logging recently added is causing issues for certain repositories. However, I'm not able to reproduce the problem. The customer can consistently invoke the issue.