Replication not working on Praefect
@zj-gitlab noticed https://dashboards.gitlab.net/d/000000199/gitaly-feature-status?orgId=1&refresh=5m&from=now-30m&to=now&var-environment=gprd&var-method=ReplicateRepository&var-prometheus=prometheus-01-inf-gprd wasn't showing many calls to ReplicateRepository
.
It seems that the replication queue is growing, but nothing is draining it:
praefect_production=> select * from replication_queue where id = 351440;
-[ RECORD 1 ]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | 351440
state | ready
created_at | 2020-05-22 03:33:55.469994
updated_at |
attempt | 3
lock_id | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git
job | {"change": "update", "params": null, "relative_path": "@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git", "virtual_storage": "praefect-file01", "source_node_storage": "file-praefect-01", "target_node_storage": "file-praefect-02"}
meta | {"correlation_id": "mxap3qSIBG2"}
praefect_production=> select count(*) from replication_queue where state = 'ready';
-[ RECORD 1 ]-
count | 351424
More details:
praefect_production=> select * from replication_queue_job_lock;
job_id | lock_id | triggered_at
--------+---------------------------------------------------------------------------------------------------------------------+---------------------------
17 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
20 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
11 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
19 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
13 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
18 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
16 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
15 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
14 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
12 | praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | 2020-05-12 22:44:21.25794
(10 rows)
praefect_production=> select * from replication_queue_lock;;
id | acquired
-------------------------------------------------------------------------------------------------------------------------------------------+----------
praefect-file01|file-praefect-02|@hashed/3e/c4/3ec42657f8f80c1fade1a47e440933f2941cb78f74f5a37963303f65f55ea73c.git | f
praefect-file01|file-praefect-02|@hashed/25/d3/25d3d6bbfcc63575e14af29c980b6a5c195ea6276e849f2d8f91bfd61f7e65d1.wiki.git | f
praefect-file01|file-praefect-02|@hashed/89/49/89491797d2e1085d188f959a218489dd4ecb52166d8a11ba66e4d3f52408ea5f.wiki.git | f
praefect-file01|file-praefect-02|@hashed/37/78/3778f5247e5a60c76cd1f33aab39aeb6f504273eac976c052852dce87ec92f0e.wiki+18121104+deleted.git | f
praefect-file01|file-praefect-02|@hashed/37/78/3778f5247e5a60c76cd1f33aab39aeb6f504273eac976c052852dce87ec92f0e+18121104+deleted.git | f
praefect-file01|file-praefect-02|@hashed/8f/2f/8f2f22edfea0891b59f1df8856958b9b247975aa14b7609d214a95b73444c247.wiki.git | f
praefect-file01|file-praefect-02|@hashed/91/7c/917c2706eb08d3f36f4a818d00cbe78586f94db5c2ea1bd674c95a3d052fd0bd.git | f
praefect-file01|file-praefect-02|@hashed/59/b2/59b233ceaf2f80882e3bfe1af1a2de29c50c7d8d1b69765f0d49c71654da754f.wiki.git | f
praefect-file01|file-praefect-02|@hashed/e0/cb/e0cbcd42a0ce9fd1bd755d349763530e70d6deade8be2cba7f46ecbaed437e1e.wiki.git | f
praefect-file01|file-praefect-02|@hashed/e0/cb/e0cbcd42a0ce9fd1bd755d349763530e70d6deade8be2cba7f46ecbaed437e1e.git | f
praefect-file01|file-praefect-02|@hashed/e9/ba/e9baf03a6c394aedb88882a0dd3a9e028cc64aae0587d35e954481028c413eb2.git | f
praefect-file01|file-praefect-02|@pools/b7/60/b76064348c4436bea83633ca7b75606d65c4eed416f2ef1a58790bd02feb09b3.git | f
praefect-file01|file-praefect-02|@hashed/a8/bb/a8bb05f2ecbbb5c85572d68d4ca655375691f9e16796948ff3915427f9cfc3a6.wiki.git | f
praefect-file01|file-praefect-02|@hashed/44/7a/447a4d1a441a5dcafaeb3604795b1de8d4b835b2eee147d99caca5366b47621e.wiki.git | f
praefect-file01|file-praefect-02|@hashed/25/d3/25d3d6bbfcc63575e14af29c980b6a5c195ea6276e849f2d8f91bfd61f7e65d1.git | f
praefect-file01|file-praefect-02|@hashed/7d/b0/7db01d2e76f83c65d886e28d8577156b4d3c325fb60308623b2f77305c64671e.git | f
praefect-file01|file-praefect-02|@hashed/9e/42/9e42fbf64f64f09950b7656657646eb2e730df6f0770eb0769a0bf39051d971b.git | f
praefect-file01|file-praefect-02|@hashed/59/b2/59b233ceaf2f80882e3bfe1af1a2de29c50c7d8d1b69765f0d49c71654da754f.git | f
praefect-file01|file-praefect-02|@hashed/37/78/3778f5247e5a60c76cd1f33aab39aeb6f504273eac976c052852dce87ec92f0e.git | f
praefect-file01|file-praefect-02|@hashed/37/78/3778f5247e5a60c76cd1f33aab39aeb6f504273eac976c052852dce87ec92f0e.wiki.git | f
praefect-file01|file-praefect-02|@hashed/89/49/89491797d2e1085d188f959a218489dd4ecb52166d8a11ba66e4d3f52408ea5f.git | f
praefect-file01|file-praefect-02|@hashed/8f/2f/8f2f22edfea0891b59f1df8856958b9b247975aa14b7609d214a95b73444c247.git | f
praefect-file01|file-praefect-02|@hashed/20/87/20879df794ca864006949dd167d64d408162a6e0557513837a9193fc80d5add2.git | f
praefect-file01|file-praefect-02|@hashed/6b/e2/6be2783407f8ec9d1c64caa573e4642db0af80b58dacd36f9a3595ce699466c3.git | f
praefect-file01|file-praefect-02|@hashed/e9/ba/e9baf03a6c394aedb88882a0dd3a9e028cc64aae0587d35e954481028c413eb2.wiki.git | f
praefect-file01|file-praefect-02|@hashed/44/7a/447a4d1a441a5dcafaeb3604795b1de8d4b835b2eee147d99caca5366b47621e.git | f
praefect-file01|file-praefect-02|@pools/b4/e5/b4e59288da371adb63645667442c2b832a742836886994e5876c41e3b85c157b.git | f
praefect-file01|file-praefect-02|@hashed/17/ee/17eeca5b5fd945ccf471f79516fd962e634cfe0a503eba632ea6f0645eb71cc8.git | f
praefect-file01|file-praefect-02|@hashed/a8/bb/a8bb05f2ecbbb5c85572d68d4ca655375691f9e16796948ff3915427f9cfc3a6.git | f
praefect-file01|file-praefect-02|@hashed/f2/9a/f29a58f741530d3f4e733523c562e76c85d7b7cebbc383e652798a1bc237b269.git | f
praefect-file01|file-praefect-02|@hashed/91/7c/917c2706eb08d3f36f4a818d00cbe78586f94db5c2ea1bd674c95a3d052fd0bd.wiki.git | f
praefect-file01|file-praefect-02|@hashed/ad/6d/ad6dcbae6d05e01f4c1b3334ee428756eb707980c66d5ff1d087252a91448e6a.git | f
praefect-file01|file-praefect-02|@hashed/98/3d/983da4e2a1eb8b24fcf8cbda3762d6191e96bb0538d47d336b5d17dc298811b1.git | f
praefect-file01|file-praefect-02|@hashed/7d/b0/7db01d2e76f83c65d886e28d8577156b4d3c325fb60308623b2f77305c64671e.wiki.git | f
praefect-file01|file-praefect-02|@pools/62/69/6269c7359593be379dd4d205e5a26a060242995f18689f94d78d85120725e51f.git | f
praefect-file01|file-praefect-02|@hashed/76/41/7641b70f320552d340a78b7304e354bbc3302de6800a67960375d3002f0fc882.git | f
praefect-file01|file-praefect-02|@hashed/f2/d3/f2d3bc9b55d396b143412b9de49e4bdea9dee2acd398291985144ccefdc76090.wiki.git | f
praefect-file01|file-praefect-02|@hashed/8e/a5/8ea5fabfc78b5521b5953bf76ce0bee4d947f0e95703a3f306e8f2b3251abcb1.git | f
praefect-file01|file-praefect-02|@hashed/f2/d3/f2d3bc9b55d396b143412b9de49e4bdea9dee2acd398291985144ccefdc76090.git | f
praefect-file01|file-praefect-02|@hashed/1f/5d/1f5d3e73643a46923705811d5b3f55f7fc26bf36bc2d3378fb1575e46b87f4e5.git | f
praefect-file01|file-praefect-02|@pools/ec/c1/ecc16e5a1ae6ebb3354ba562e78a68729822b8caf4c221985df72ddc68ed0880.git | f
praefect-file01|file-praefect-02|@hashed/7a/d1/7ad194c2ddfa9630fb5d0ec91ebc03431a2fe2d2f3dad3e3e5298889fc02ba81.wiki.git | f
praefect-file01|file-praefect-02|@hashed/c1/af/c1af1f4d37a5e7c505d630e19138dd954d3adb78a758e48197b0dc76b43ae0cd.git | f
praefect-file01|file-praefect-02|@hashed/33/93/339341c83cda8aa182567768565982201cf89691c95311e27a8c1030c8fad40b.git | f
praefect-file01|file-praefect-02|@hashed/13/9b/139b4b03b5e42836ad56ee33760e242615ad659c22090d15459ed4123f7285ac.git | f
praefect-file01|file-praefect-02|@hashed/13/9b/139b4b03b5e42836ad56ee33760e242615ad659c22090d15459ed4123f7285ac.wiki.git | f
praefect-file01|file-praefect-02|@hashed/a9/dc/a9dc5b101e6fe6bc390df39733bb5c1b530befb439695ee26adb76981de0a09d.wiki.git | f
praefect-file01|file-praefect-02|@hashed/00/19/00193c54e1e1565579dcb33f81bafbc355b511c73e0bc9a1cc398765937835a6.git | f
praefect-file01|file-praefect-02|@hashed/00/19/00193c54e1e1565579dcb33f81bafbc355b511c73e0bc9a1cc398765937835a6.wiki.git | f
praefect-file01|file-praefect-02|@hashed/22/06/220611ff38a1bdd5ac2a75ffcbfe2a826a03272c4082211e38358290738d5898.wiki.git | f
praefect-file01|file-praefect-02|@hashed/64/b1/64b17035ec6e7c80bdd32c5fa3362eb6ef8c19d5694a594d37d8a89e1749a032.git | f
praefect-file01|file-praefect-02|@hashed/17/db/17dbf1ec9946072326a380336ae8ed5a5ef7bdb86657f2148453a5d1fcd5cc8b.git | f
praefect-file01|file-praefect-02|@hashed/ca/43/ca4386a565bd948bf0d78e7571bf6ef5901164a47fc21c27603b00f1750c2302.git | f
praefect-file01|file-praefect-02|@hashed/a9/dc/a9dc5b101e6fe6bc390df39733bb5c1b530befb439695ee26adb76981de0a09d.git | f
praefect-file01|file-praefect-02|@hashed/13/9b/139b4b03b5e42836ad56ee33760e242615ad659c22090d15459ed4123f7285ac+18191626+deleted.git | f
praefect-file01|file-praefect-02|@hashed/13/9b/139b4b03b5e42836ad56ee33760e242615ad659c22090d15459ed4123f7285ac.wiki+18191626+deleted.git | f
praefect-file01|file-praefect-02|@hashed/18/02/1802b93eba8213f3088f85351c7ce9df00a6d6055efe025862aca7dda1c477ce.wiki.git | f
praefect-file01|file-praefect-02|@hashed/33/93/339341c83cda8aa182567768565982201cf89691c95311e27a8c1030c8fad40b.wiki.git | f
praefect-file01|file-praefect-02|@hashed/22/06/220611ff38a1bdd5ac2a75ffcbfe2a826a03272c4082211e38358290738d5898.git | f
praefect-file01|file-praefect-02|@hashed/16/d8/16d8131101999baee14c9ecdfaa50c192f1cbae26b91037c221431fcae619ec5.wiki.git | f
praefect-file01|file-praefect-02|@hashed/8c/24/8c244f6616f31ded062cdcbed00aa4132dd43fc533005e551ed5763b15676da8.git | f
praefect-file01|file-praefect-02|@hashed/00/19/00193c54e1e1565579dcb33f81bafbc355b511c73e0bc9a1cc398765937835a6+18809844+deleted.git | f
praefect-file01|file-praefect-02|@hashed/00/19/00193c54e1e1565579dcb33f81bafbc355b511c73e0bc9a1cc398765937835a6.wiki+18809844+deleted.git | f
praefect-file01|file-praefect-02|@hashed/17/db/17dbf1ec9946072326a380336ae8ed5a5ef7bdb86657f2148453a5d1fcd5cc8b.wiki.git | f
praefect-file01|file-praefect-02|@hashed/18/02/1802b93eba8213f3088f85351c7ce9df00a6d6055efe025862aca7dda1c477ce.git | f
praefect-file01|file-praefect-02|@hashed/f0/21/f021af2c34cdb198070e90cd0436b1acc83ef03fd1d83b916de16a1f1c92ab44.wiki.git | f
praefect-file01|file-praefect-02|@hashed/56/91/56914e0e182155dde242612bb0c8d348a44a158f2084709de292c775e6aaadc0.wiki.git | f
praefect-file01|file-praefect-02|@hashed/f0/21/f021af2c34cdb198070e90cd0436b1acc83ef03fd1d83b916de16a1f1c92ab44.git | f
praefect-file01|file-praefect-02|@hashed/63/2f/632fdb666657688cbc4b42d93898038c7f6ef32673783759a5b391c966f2cae9.git | f
praefect-file01|file-praefect-02|@hashed/1d/78/1d78023cdb23631ee021f60f388d92e3bed36debe8894ead5284126f495b2e44.wiki.git | f
praefect-file01|file-praefect-02|@hashed/64/b1/64b17035ec6e7c80bdd32c5fa3362eb6ef8c19d5694a594d37d8a89e1749a032.wiki.git | f
praefect-file01|file-praefect-02|@hashed/7a/d1/7ad194c2ddfa9630fb5d0ec91ebc03431a2fe2d2f3dad3e3e5298889fc02ba81.git | f
praefect-file01|file-praefect-02|@hashed/43/d7/43d7ff0cfd3753f961beaeed37941f94d366cebddac233380d660315e06f091c.git | f
praefect-file01|file-praefect-02|@hashed/75/8b/758b77ed33c9ac77b6b98809c0d595ec5418a03a6a66b3bf1a4f7567da2ecbd3.git | f
praefect-file01|file-praefect-02|@hashed/02/e0/02e04474b615e6f703d4bad43cfdfefbfc8875a7f2c725a5e1ff6252b83d7485.wiki.git | f
praefect-file01|file-praefect-02|@hashed/16/d8/16d8131101999baee14c9ecdfaa50c192f1cbae26b91037c221431fcae619ec5.git | f
praefect-file01|file-praefect-02|@hashed/1d/78/1d78023cdb23631ee021f60f388d92e3bed36debe8894ead5284126f495b2e44.git | f
praefect-file01|file-praefect-02|@pools/14/c0/14c06873c7264ffcbf3f69e2b5a05764fc4c5dc72530b0262266f470f55301a1.git | f
praefect-file01|file-praefect-02|@hashed/56/91/56914e0e182155dde242612bb0c8d348a44a158f2084709de292c775e6aaadc0.git | f
praefect-file01|file-praefect-02|@hashed/75/8b/758b77ed33c9ac77b6b98809c0d595ec5418a03a6a66b3bf1a4f7567da2ecbd3.wiki.git | f
praefect-file01|file-praefect-02|@hashed/c1/af/c1af1f4d37a5e7c505d630e19138dd954d3adb78a758e48197b0dc76b43ae0cd.wiki.git | f
praefect-file01|file-praefect-02|@hashed/1d/54/1d5459a3d45fe2d0820912b8235016e0730ba2486818c59511eb8ffab4ab77f7.wiki.git | f
praefect-file01|file-praefect-02|@hashed/20/f5/20f5f9119d997e91926bda80fc9d205b0b4c0f354531de63909b738a4c7f4f4c.git | f
praefect-file01|file-praefect-02|@hashed/1d/54/1d5459a3d45fe2d0820912b8235016e0730ba2486818c59511eb8ffab4ab77f7.git | f
praefect-file01|file-praefect-02|@hashed/02/e0/02e04474b615e6f703d4bad43cfdfefbfc8875a7f2c725a5e1ff6252b83d7485.git | f
praefect-file01|file-praefect-02|@hashed/9a/2d/9a2d0607559dd05acdd90ef2fd54237db95af37414f0a3d4b734e0b76d0fe9b8.git | f
praefect-file01|file-praefect-02|@hashed/2e/c2/2ec2ab67ec6891a72193c1f98807f849d15be6feef319e1490a455f7aa78f1b5.git | f
praefect-file01|file-praefect-02|@hashed/2e/c2/2ec2ab67ec6891a72193c1f98807f849d15be6feef319e1490a455f7aa78f1b5.wiki.git | f
praefect-file01|file-praefect-02|@hashed/7e/41/7e4157c184fa6aad2e3bfe364743180f412603757eb02a3e1e07b2b2733fb53a.git | f
praefect-file01|file-praefect-02|@hashed/3e/f4/3ef45948f6e42c084ef0fa1d431bfedd17345ef80b8e006d99c71d04497837b1.git | f
praefect-file01|file-praefect-02|@hashed/88/27/8827d7e8aca066029c10f3fd61b4e9c9ad5f7f3bab51fb7fa5233a7748edcd37.wiki.git | f
praefect-file01|file-praefect-02|@hashed/2a/3e/2a3e0a7e19a6a8034cf6fe0293ba42b7b9070ad7b2f1219bcd36499d735ccd9d.git | f
praefect-file01|file-praefect-02|@hashed/95/17/95175598da38de3cb9e8b43aba395a7d30598fb476abe9696b67475f73c919f7.git | f
praefect-file01|file-praefect-02|@hashed/4b/53/4b53fc35e4e9bb5eb14907ef26f625fc099e6bde23babf723bbdb3cf9fcedd55.git | f
praefect-file01|file-praefect-02|@hashed/86/68/8668f25db831d913663185c9acad2af5f14897bf54562c40b72f200d72257869.wiki.git | f
praefect-file01|file-praefect-02|@hashed/e9/c7/e9c7aa99c23ee5ca91231f49afd565cb4c1747a509929516e935ab824b0c7047.wiki.git | f
praefect-file01|file-praefect-02|@hashed/2a/3e/2a3e0a7e19a6a8034cf6fe0293ba42b7b9070ad7b2f1219bcd36499d735ccd9d.wiki.git | f
praefect-file01|file-praefect-02|@hashed/a3/83/a3832c673ebf95242eea545e698126b5df5685f17647e00099840da18e32b4a2.git | f
praefect-file01|file-praefect-02|@hashed/2b/73/2b73100798aea441479f918ba6d7f2b64581a9fca762492d20c8e73b1786b5c1.git | f
praefect-file01|file-praefect-02|@hashed/86/68/8668f25db831d913663185c9acad2af5f14897bf54562c40b72f200d72257869.git | f
praefect-file01|file-praefect-02|@hashed/f8/e4/f8e46153a0bad73e60a9e647a1fe37799c8b2ab1e8c36d57054b97ddfdf5a4b5.git | f
praefect-file01|file-praefect-02|@hashed/1e/44/1e444c0a61ed6a89fcdf427a7bba1112977ab9e4eec162cf364ee1e4e5eb0beb.wiki.git | f
praefect-file01|file-praefect-02|@hashed/e9/c7/e9c7aa99c23ee5ca91231f49afd565cb4c1747a509929516e935ab824b0c7047.git | f
praefect-file01|file-praefect-02|@hashed/00/f0/00f0eec5d9a9a835d4d24aa01e1b59b3d8038551d74c24983ee1dd9a1bac908d.wiki.git | f
praefect-file01|file-praefect-02|@hashed/7f/8b/7f8beedb7576504402cbe0b727f8d62c4ec2ee4f742ad274344f27d819dc7d04.wiki.git | f
praefect-file01|file-praefect-02|@hashed/7f/8b/7f8beedb7576504402cbe0b727f8d62c4ec2ee4f742ad274344f27d819dc7d04.git | f
praefect-file01|file-praefect-02|@hashed/a4/fe/a4fe9c8a56151fa67eb6033880aee1b0c2ed3af203b190d75371a3e4ab6ed0e8.git | f
praefect-file01|file-praefect-02|@hashed/69/21/6921a22f648bb546e59bdb73e5b69e3b88918b99b8788bc74e74b7ce05ce3417.git | f
praefect-file01|file-praefect-02|@hashed/1e/44/1e444c0a61ed6a89fcdf427a7bba1112977ab9e4eec162cf364ee1e4e5eb0beb.git | f
praefect-file01|file-praefect-02|@hashed/88/27/8827d7e8aca066029c10f3fd61b4e9c9ad5f7f3bab51fb7fa5233a7748edcd37.git | f
praefect-file01|file-praefect-02|@hashed/00/f0/00f0eec5d9a9a835d4d24aa01e1b59b3d8038551d74c24983ee1dd9a1bac908d.git | f
praefect-file01|file-praefect-02|@hashed/c3/df/c3dfd2412dad03228ebac5a61e99c85e4a751fac4b536c892d533ee8f0d1b8d2.git | f
praefect-file01|file-praefect-02|@hashed/b7/71/b7713b6e71d8a1060ff400b6528117edbd5deae007b143ebf878c561de38e7d9.git | f
praefect-file01|file-praefect-02|@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git | t
praefect-file01|file-praefect-02|@hashed/b8/b0/b8b0e89984c0f56a554187cfc17d77e972e1e577e1dbf9cf13193c8d4e8db24b.git | f
praefect-file01|file-praefect-02|@hashed/b9/7a/b97aab9cfc909082eba5a3797d109ecaf65b7523fe62f63924a126a00f5d0aa1.git | f
praefect-file01|file-praefect-02|@hashed/94/d1/94d17a47b559db4e59e13ca10c0f1037ac49f3050e53adbf59ebc2feb528c344.git | f
praefect-file01|file-praefect-02|@hashed/a3/58/a35862491985dca4f7f0e6e2fe12fe7261472f0c3e6942abde2859ae7a36ee79.git | f
(118 rows)
On both praefect-01 and praefect-02, we see:
root@praefect-02-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/praefect# curl -s http://localhost:9652/metrics | grep -i replication
# HELP gitaly_praefect_replication_jobs
# TYPE gitaly_praefect_replication_jobs gauge
gitaly_praefect_replication_jobs 0