SmartHTTP transactions broken with storage generations
Whenever one pushes to a repo via the smarthttp transport, one will first call the smarthttp.InfoRefsReceivePack
service followed by smarthttp.PostReceivePack
. While the first call will usually not cause and on-disk modifications, both calls are still labeled as mutators and thus cause an increment of the storage generation. As they're called in quick succession, the storage generation will not yet have propagated from the primary to secondaries when smarthttp.PostReceivePack
. Because we only register transactions for nodes whose storage generation indicates a consistent node, the result will thus always be a transaction whose only voter is the primary itself, effectively rendering the transaction useless.
The following log excerpt shows the issue:
2020-08-03_12:27:30.91101 praefect : time="2020-08-03T12:27:30Z" level=debug msg=RegisterTransaction component=transactions.Manager correlation_id=JRSFR29hjHa grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-workhorse grpc.meta.deadline_type=none grpc.method=InfoRefsReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPServiceInfoRefsReceivePack grpc.service=gitaly.SmartHTTPService grpc.start_time="2020-08-03T12:27:30Z" peer.address=@ pid=57694 relative_path=@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git span.kind=server system=grpc transaction_id=894385949183117216 virtual_storage=default voters="[{praefect-internal-0 1 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0} {praefect-internal-1 1 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0} {praefect-internal-2 1 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0}]"
2020-08-03_12:27:30.92500 praefect : time="2020-08-03T12:27:30Z" level=info msg="transactions: increment" correlation_id=JRSFR29hjHa grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-workhorse grpc.meta.deadline_type=none grpc.method=InfoRefsReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPService/InfoRefsReceivePack grpc.service=gitaly.SmartHTTPService grpc.start_time="2020-08-03T12:27:30Z" peer.address=@ pid=57694 relative_path=@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git span.kind=server system=grpc virtual_storage=default
2020-08-03_12:27:31.10769 praefect : time="2020-08-03T12:27:31Z" level=debug msg=RegisterTransaction component=transactions.Manager correlation_id=3rCGbALNtu3 grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-workhorse grpc.meta.deadline_type=none grpc.method=PostReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPService/PostReceivePack grpc.service=gitaly.SmartHTTPService grpc.start_time="2020-08-03T12:27:31Z" peer.address=@ pid=57694 relative_path=@hashed/94/00/9400f1b21cb527d7fa3d3eabba93557a18ebe7a2ca4e471cfe5e4c5b4ca7f767.git span.kind=server system=grpc transaction_id=11998794077335055257 virtual_storage=default voters="[{praefect-internal-2 1 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0}]"