If possible, make GAIA MsgSolvency report the balance for the claimed chain height
Discord thread context:
https://discord.com/channels/838986635756044328/1193702350200766494/1194642943772991570
GAIA (being Tendermint-based) has no re-orgs.
There have been frequent automatic SolvencyHaltGAIAChain
halts and unhalts recently.
GAIA like THORChain has a 6-second block time, meaning the most recent chain height changes rapidly,
and if a node is (or worse, over 1/3rd of nodes are) lagging then reporting an unobserved-block balance as though an observed-block's balance is straightforwardly undesirable.
(1/3rd of all Active nodes is the number which satisfies HasMinority
for SolvencyVoter's HasConsensus
.)
Two examples of inconsistent MsgSolvency IDs despite identical vaults and chains and chain heights:
docker run --rm registry.gitlab.com/ninerealms/cosmoscan 'tx_msgs_listener(lambda h,txh,m: (h,m), types={"MsgSolvency"}), start=14197825, end=14197826, progress=True' | jq '[.[0], .[1].pub_key[-4:], .[1].chain, .[1].height, .[1].id]' -c | sort | uniq -c | grep utvk | grep 18642980
23 [14197825,"utvk","GAIA","18642980","20B0D59B1E5A32171B06C7DD49DEDB9F9BCB391C9244583A8362EFF3437EACF6"]
23 [14197825,"utvk","GAIA","18642980","79B5319A94F0F50AEB3D75302BC9911081FB1AA48621ED172E8B0EB6E40FE34B"]
7 [14197826,"utvk","GAIA","18642980","20B0D59B1E5A32171B06C7DD49DEDB9F9BCB391C9244583A8362EFF3437EACF6"]
39 [14197826,"utvk","GAIA","18642980","79B5319A94F0F50AEB3D75302BC9911081FB1AA48621ED172E8B0EB6E40FE34B"]
(30 nodes claiming one GAIA.ATOM balance, 62 claiming another)
https://thornode-v1.ninerealms.com/txs/8054CC6B804760E647DAF44BD932709A4246B1771F1965B77A89F5F067CF367F
https://thornode-v1.ninerealms.com/txs/C7E7B878BC1D7DEADE93A3A5FD1A0DA04C92192394938067B6AFD3B4CC931615
docker run --rm registry.gitlab.com/ninerealms/cosmoscan 'tx_msgs_listener(lambda h,txh,m: (h,m), types={"MsgSolvency"}), start=14198128, end=14198129, progress=True' | jq '[.[0], .[1].pub_key[-4:], .[1].chain, .[1].height, .[1].id]' -c | sort | uniq -c | grep utvk | grep 18643300
65 [14198128,"utvk","GAIA","18643300","3827B345FE94EA22A2B79786BB251A58D93ABC35AD817EFCAE18F52E1455FE75"]
15 [14198128,"utvk","GAIA","18643300","87685D4E2C6A4617A43D049BA64CCF9CDCD199C081B3D6C3A6447AE728FBB649"]
8 [14198129,"utvk","GAIA","18643300","3827B345FE94EA22A2B79786BB251A58D93ABC35AD817EFCAE18F52E1455FE75"]
4 [14198129,"utvk","GAIA","18643300","87685D4E2C6A4617A43D049BA64CCF9CDCD199C081B3D6C3A6447AE728FBB649"]
(73 nodes claiming one GAIA.ATOM balance, 19 claiming another)
https://thornode-v1.ninerealms.com/txs/E1244964E3887BE3513E24C6FB90F416EF86FF7130EA259305A88AD866A9C8C9
https://thornode-v1.ninerealms.com/txs/7260B0406EC3D4734A89EE050C52630F2D1FF582682AD90A73F82065732304A2
Code reasoning for how data from the 'ctx' in GetAccountByAddress
is currently propagated in the query:
(Collapsible section)
THORNode NewCosmosClient
:
https://gitlab.com/thorchain/thornode/-/blob/v1.125.3/bifrost/pkg/chainclients/gaia/cosmos_client.go#L138
bankClient: btypes.NewQueryClient(grpcConn),
Cosmos SDK:
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/x/bank/types/query.pb.go#L797C2-L797C2
func NewQueryClient(cc grpc1.ClientConn) QueryClient {
return &queryClient{cc}
}
(THORNode's CosmosClient's 'bankClient' field is of Cosmos SDK type queryClient.)
THORNode ReportSolvency
:
https://gitlab.com/thorchain/thornode/-/blob/v1.125.3/bifrost/pkg/chainclients/gaia/cosmos_client.go#L554-564
func (c *CosmosClient) ReportSolvency(blockHeight int64) error {
if !c.ShouldReportSolvency(blockHeight) {
return nil
}
asgardVaults, err := c.thorchainBridge.GetAsgards()
if err != nil {
return fmt.Errorf("fail to get asgards,err: %w", err)
}
for _, asgard := range asgardVaults {
var acct common.Account
acct, err = c.GetAccount(asgard.PubKey, big.NewInt(0))
(GetAccount is called with second argument 0, rather than the external chain height like ETH/EVM.)
THORNode GetAccount
:
https://gitlab.com/thorchain/thornode/-/blob/v1.125.3/bifrost/pkg/chainclients/gaia/cosmos_client.go#L246
return c.GetAccountByAddress(addr.String(), big.NewInt(0))
THORNode GetAccountByAddress
:
https://gitlab.com/thorchain/thornode/-/blob/v1.125.3/bifrost/pkg/chainclients/gaia/cosmos_client.go#L249-256
func (c *CosmosClient) GetAccountByAddress(address string, _ *big.Int) (common.Account, error) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
bankReq := &btypes.QueryAllBalancesRequest{
Address: address,
}
balances, err := c.bankClient.AllBalances(ctx, bankReq)
Cosmos SDK queryClient-received `AllBalances`: \
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/x/bank/types/query.pb.go#L808-L810
```go
func (c *queryClient) AllBalances(ctx context.Context, in *QueryAllBalancesRequest, opts ...grpc.CallOption) (*QueryAllBalancesResponse, error) {
out := new(QueryAllBalancesResponse)
err := c.cc.Invoke(ctx, "/cosmos.bank.v1beta1.Query/AllBalances", in, out, opts...)
https://gitlab.com/thorchain/thornode/-/blob/v1.125.3/go.mod#L288
github.com/gogo/protobuf => github.com/regen-network/protobuf v1.3.2-alpha.regen.4
https://github.com/regen-network/protobuf/blob/v1.3.2-alpha.regen.4/grpc/types.go#L12-L13
type ClientConn interface {
Invoke(ctx context.Context, method string, args, reply interface{}, opts ...grpc.CallOption) error
Cosmos SDK Invoke
(is this really the called Invoke
? If not, then what and how?):
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/grpc_query.go#L4
gocontext "context"
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/grpc_query.go#L22
var _ gogogrpc.ClientConn = Context{}
(relevant for the c.cc.Invoke
, where the cc is the connection from btypes.NewQueryClient(grpcConn)
)
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/grpc_query.go#L26-L27
// Invoke implements the grpc ClientConn.Invoke method
func (ctx Context) Invoke(grpcCtx gocontext.Context, method string, req, reply interface{}, opts ...grpc.CallOption) (err error) {
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/grpc_query.go#L59-L84
// parse height header
md, _ := metadata.FromOutgoingContext(grpcCtx)
if heights := md.Get(grpctypes.GRPCBlockHeightHeader); len(heights) > 0 {
height, err := strconv.ParseInt(heights[0], 10, 64)
if err != nil {
return err
}
if height < 0 {
return sdkerrors.Wrapf(
sdkerrors.ErrInvalidRequest,
"client.Context.Invoke: height (%d) from %q must be >= 0", height, grpctypes.GRPCBlockHeightHeader)
}
ctx = ctx.WithHeight(height)
}
abciReq := abci.RequestQuery{
Path: method,
Data: reqBz,
Height: ctx.Height,
}
res, err := ctx.QueryABCI(abciReq)
if err != nil {
return err
}
(The Height for the RequestQuery thus comes from the first-argument Context of Invoke
.)
Cosmos SDK QueryABCI
:
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/query.go#L52-L58
// QueryABCI performs a query to a Tendermint node with the provide RequestQuery.
// It returns the ResultQuery obtained from the query. The height used to perform
// the query is the RequestQuery Height if it is non-zero, otherwise the context
// height is used.
func (ctx Context) QueryABCI(req abci.RequestQuery) (abci.ResponseQuery, error) {
return ctx.queryABCI(req)
}
queryABCI
:
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/query.go#L81-L94
var queryHeight int64
if req.Height != 0 {
queryHeight = req.Height
} else {
// fallback on the context height
queryHeight = ctx.Height
}
opts := rpcclient.ABCIQueryOptions{
Height: queryHeight,
Prove: req.Prove,
}
result, err := node.ABCIQueryWithOptions(context.Background(), req.Path, req.Data, opts)
I initially thought WithHeight
could be used directly,
but that would only be for cosmos.Context rather than context.Context;
instead, it seems it could be necessary to use something like
ctx = metadata.AppendToOutgoingContext(ctx, grpctypes.GRPCBlockHeightHeader, height.String())
for the sake of Invoke
's
https://github.com/cosmos/cosmos-sdk/blob/v0.45.1/client/grpc_query.go#L61-L62
if heights := md.Get(grpctypes.GRPCBlockHeightHeader); len(heights) > 0 {
height, err := strconv.ParseInt(heights[0], 10, 64)
. Specifically: !3360 (merged)
'GAIA ReportSolvency
balances consistent with specified heights'.