git pull failling with `git fetch-pack: expected ACK/NAK` after enabling FF
Context
A customer has reported some intermittency when pulling repositories from a Geo secondary site through SSH. They enabled the geo_proxy_fetch_direct_to_primary_with_options
FF, which enables the bugfix for git clone with the --depth
option introduced by !905 (merged).
We might need to replicate and validate this problem.
Git command
➜ <user> git:(master) git pull
remote:
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
remote: git@xxxxxxx/<repository>.git
remote:
fatal: git fetch-pack: expected ACK/NAK, got '?Enumerating objects: 5363, done.'
Git trace
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
debug2: channel 0: written 114 to efd 6
debug2: channel 0: rcvd ext data 58
debug2: channel 0: rcvd ext data 9
remote: git@<repository>.git
remote:
debug2: channel 0: written 67 to efd 6
debug2: channel 0: window 1998701 sent adjust 98451
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: window 1966080 sent adjust 131072
debug2: channel 0: rcvd adjust 160
fatal: git fetch-pack: expected ACK/NAK, got '?Enumerating objects: 178066, done.'
debug2: channel 0: read failed rfd 4 maxlen 32768: Broken pipe
debug2: channel 0: read failed
debug2: chan_shutdown_read: channel 0: (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
~/cluster-state # debug2: channel 0: write failed
debug2: chan_shutdown_write: channel 0: (i3 o0 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
debug2: channel 0: window 1991258 sent adjust 101812
debug2: channel 0: window 1996219 sent adjust 96851
debug2: channel 0: window 1992664 sent adjust 100406
debug2: channel 0: window 1997086 sent adjust 95984
debug2: channel 0: window 1992672 sent adjust 100398
debug2: channel 0: window 1992277 sent adjust 100793
debug2: channel 0: window 1994169 sent adjust 98901
debug2: channel 0: window 1994162 sent adjust 98908
debug2: channel 0: window 1997086 sent adjust 95984
debug2: channel 0: window 1993416 sent adjust 99654
debug2: channel 0: window 1994154 sent adjust 98916
debug2: channel 0: window 1994891 sent adjust 98179
debug2: channel 0: window 1990803 sent adjust 102267
debug2: channel 0: window 1972248 sent adjust 120822
debug2: channel 0: window 1991556 sent adjust 101514
debug2: channel 0: window 1997133 sent adjust 95937
debug2: channel 0: window 1993448 sent adjust 99622
debug2: channel 0: window 1992811 sent adjust 100259
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1993274 sent adjust 99796
debug2: channel 0: window 1996138 sent adjust 96932
debug2: channel 0: window 1991823 sent adjust 101247
debug2: channel 0: window 1997576 sent adjust 95494
debug2: channel 0: window 1994719 sent adjust 98351
debug2: channel 0: window 1990605 sent adjust 102465
debug2: channel 0: window 1994711 sent adjust 98359
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1994701 sent adjust 98369
debug2: channel 0: window 1998794 sent adjust 94276
debug2: channel 0: window 1990633 sent adjust 102437
debug2: channel 0: window 1993227 sent adjust 99843
debug2: channel 0: window 1992074 sent adjust 100996
debug2: channel 0: window 1994693 sent adjust 98377
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1994706 sent adjust 98364
debug2: channel 0: window 1994732 sent adjust 98338
debug2: channel 0: window 1994680 sent adjust 98390
debug2: channel 0: window 1998838 sent adjust 94232
debug2: channel 0: window 1986574 sent adjust 106496
debug2: channel 0: window 1994830 sent adjust 98240
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1986645 sent adjust 106425
debug2: channel 0: window 1994829 sent adjust 98241
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1990726 sent adjust 102344
debug2: channel 0: window 1986622 sent adjust 106448
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1994821 sent adjust 98249
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1986646 sent adjust 106424
debug2: channel 0: window 1990742 sent adjust 102328
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1994830 sent adjust 98240
debug2: channel 0: window 1978438 sent adjust 114632
debug2: channel 0: window 1994846 sent adjust 98224
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1978438 sent adjust 114632
debug2: channel 0: window 1974342 sent adjust 118728
debug2: channel 0: window 1978422 sent adjust 114648
debug2: channel 0: window 1978462 sent adjust 114608
debug2: channel 0: window 1978462 sent adjust 114608
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1974342 sent adjust 118728
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1990734 sent adjust 102336
debug2: channel 0: window 1990717 sent adjust 102353
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1966205 sent adjust 126865
debug2: channel 0: window 1978454 sent adjust 114616
debug2: channel 0: window 1986614 sent adjust 106456
debug2: channel 0: window 1970246 sent adjust 122824
debug2: channel 0: window 1978422 sent adjust 114648
debug2: channel 0: window 1974358 sent adjust 118712
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1990702 sent adjust 102368
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1986590 sent adjust 106480
debug2: channel 0: window 1970238 sent adjust 122832
debug2: channel 0: window 1986590 sent adjust 106480
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1990718 sent adjust 102352
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1994798 sent adjust 98272
debug2: channel 0: window 1974342 sent adjust 118728
debug2: channel 0: window 1957965 sent adjust 135105
debug2: channel 0: window 1974326 sent adjust 118744
debug2: channel 0: window 1990710 sent adjust 102360
debug2: channel 0: window 1994798 sent adjust 98272
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1978469 sent adjust 114601
debug2: channel 0: window 1986606 sent adjust 106464
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1990710 sent adjust 102360
debug2: channel 0: window 1978438 sent adjust 114632
debug2: channel 0: window 1978462 sent adjust 114608
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1974374 sent adjust 118696
debug2: channel 0: window 1970262 sent adjust 122808
debug2: channel 0: window 1986605 sent adjust 106465
debug2: channel 0: window 1990734 sent adjust 102336
debug2: channel 0: window 1994789 sent adjust 98281
debug2: channel 0: window 1978421 sent adjust 114649
debug2: channel 0: window 1986637 sent adjust 106433
debug2: channel 0: window 1970262 sent adjust 122808
debug2: channel 0: window 1978422 sent adjust 114648
debug2: channel 0: window 1982518 sent adjust 110552
debug2: channel 0: window 1974342 sent adjust 118728
debug2: channel 0: window 1978398 sent adjust 114672
debug2: channel 0: window 1957942 sent adjust 135128
debug2: channel 0: window 1982502 sent adjust 110568
debug2: channel 0: window 1986598 sent adjust 106472
debug2: channel 0: window 1994829 sent adjust 98241
debug2: channel 0: window 1982565 sent adjust 110505
debug2: channel 0: window 1990742 sent adjust 102328
debug2: channel 0: window 1986654 sent adjust 106416
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1978422 sent adjust 114648
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1970229 sent adjust 122841
debug2: channel 0: window 1970270 sent adjust 122800
debug2: channel 0: window 1970254 sent adjust 122816
debug2: channel 0: window 1974326 sent adjust 118744
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1990734 sent adjust 102336
debug2: channel 0: window 1986645 sent adjust 106425
debug2: channel 0: window 1970253 sent adjust 122817
debug2: channel 0: window 1978413 sent adjust 114657
debug2: channel 0: window 1994798 sent adjust 98272
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1974348 sent adjust 118722
debug2: channel 0: window 1990718 sent adjust 102352
debug2: channel 0: window 1970222 sent adjust 122848
debug2: channel 0: window 1974326 sent adjust 118744
debug2: channel 0: window 1982502 sent adjust 110568
debug2: channel 0: window 1982566 sent adjust 110504
debug2: channel 0: window 1986606 sent adjust 106464
debug2: channel 0: window 1966142 sent adjust 126928
debug2: channel 0: window 1966118 sent adjust 126952
debug2: channel 0: window 1966149 sent adjust 126921
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1994837 sent adjust 98233
debug2: channel 0: window 1978454 sent adjust 114616
debug2: channel 0: window 1978437 sent adjust 114633
debug2: channel 0: window 1990710 sent adjust 102360
debug2: channel 0: window 1974326 sent adjust 118744
debug2: channel 0: window 1986598 sent adjust 106472
debug2: channel 0: window 1974366 sent adjust 118704
debug2: channel 0: window 1966133 sent adjust 126937
debug2: channel 0: window 1994830 sent adjust 98240
debug2: channel 0: window 1966149 sent adjust 126921
debug2: channel 0: window 1974318 sent adjust 118752
debug2: channel 0: window 1970246 sent adjust 122824
debug2: channel 0: window 1978461 sent adjust 114609
debug2: channel 0: window 1962086 sent adjust 130984
debug2: channel 0: window 1978454 sent adjust 114616
debug2: channel 0: window 1978429 sent adjust 114641
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1986590 sent adjust 106480
debug2: channel 0: window 1953862 sent adjust 139208
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1994798 sent adjust 98272
debug2: channel 0: window 1990694 sent adjust 102376
debug2: channel 0: window 1994806 sent adjust 98264
debug2: channel 0: window 1953877 sent adjust 139193
debug2: channel 0: window 1974342 sent adjust 118728
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1986630 sent adjust 106440
debug2: channel 0: window 1953877 sent adjust 139193
debug2: channel 0: window 1970206 sent adjust 122864
debug2: channel 0: window 1994798 sent adjust 98272
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1986606 sent adjust 106464
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1994782 sent adjust 98288
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1957966 sent adjust 135104
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1978422 sent adjust 114648
debug2: channel 0: window 1978430 sent adjust 114640
debug2: channel 0: window 1974350 sent adjust 118720
debug2: channel 0: window 1994813 sent adjust 98257
debug2: channel 0: window 1970262 sent adjust 122808
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1986614 sent adjust 106456
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1978406 sent adjust 114664
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1990741 sent adjust 102329
debug2: channel 0: window 1990766 sent adjust 102304
debug2: channel 0: window 1990766 sent adjust 102304
debug2: channel 0: window 1978478 sent adjust 114592
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1974366 sent adjust 118704
debug2: channel 0: window 1978437 sent adjust 114633
debug2: channel 0: window 1978446 sent adjust 114624
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1994822 sent adjust 98248
debug2: channel 0: window 1978462 sent adjust 114608
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1962030 sent adjust 131040
debug2: channel 0: window 1978414 sent adjust 114656
debug2: channel 0: window 1990701 sent adjust 102369
debug2: channel 0: window 1978398 sent adjust 114672
debug2: channel 0: window 1982549 sent adjust 110521
debug2: channel 0: window 1978470 sent adjust 114600
debug2: channel 0: window 1982582 sent adjust 110488
debug2: channel 0: window 1982582 sent adjust 110488
debug2: channel 0: window 1994838 sent adjust 98232
debug2: channel 0: window 1986622 sent adjust 106448
debug2: channel 0: window 1976267 sent adjust 116803
debug2: channel 0: window 1997065 sent adjust 96005
debug2: channel 0: window 1994854 sent adjust 98216
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1962086 sent adjust 130984
debug2: channel 0: window 1978454 sent adjust 114616
debug2: channel 0: window 1994790 sent adjust 98280
debug2: channel 0: window 1970214 sent adjust 122856
debug2: channel 0: window 1990709 sent adjust 102361
debug2: channel 0: window 1982526 sent adjust 110544
debug2: channel 0: window 1990750 sent adjust 102320
debug2: channel 0: window 1990758 sent adjust 102312
debug2: channel 0: window 1994814 sent adjust 98256
debug2: channel 0: window 1978494 sent adjust 114576
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 [session] r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1 io 0x00/0x00)
debug3: send packet: type 1
Transferred: sent 13000, received 33128264 bytes, in 4.4 seconds
Bytes per second: sent 2974.3, received 7579414.3
debug1: Exit status 0
The full Git trace can be found in the Zendesk ticket
Edited by Javiera Tapia