[V109-specific; store migration] Requeue ETH-halt swallowed TxOutItems
[V109-specific]
According to
SELECT fact_set_mimir_events.block_timestamp, block_id, key, value
FROM (thorchain.core.fact_set_mimir_events INNER JOIN thorchain.core.dim_block
ON fact_set_mimir_events.dim_block_id = dim_block.dim_block_id)
HAVING LOWER(key) LIKE '%eth%'
ORDER BY block_id DESC, Mimir key changes related to the recent ETH chain halt were between blocks 10386531 (2023-04-12) and 10445609 (2023-04-17).
There is no Flipside fact_scheduled_outbound_events table (though perhaps Cosmoscan could be used);
looking specifically at block 10386531, there were nine ETH-chain TxOutItems in the outbound queue (the /thorchain/queue/swap and /thorchain/queue/scheduled endpoints were empty), of which five appear to have been swallowed by the vaults (thus being candidates for re-queuing without loss to the network).
|
[Edit: Now trying the below command (expecting successful-outbound TxIDs to be reported whether or not any no-outbound scheduled outbounds are reported), thus needing individual checking after the command completes.
docker run --rm registry.gitlab.com/ninerealms/cosmoscan 'block_events_listener(lambda h,e: (h,e), types={"scheduled_outbound"}), start=10386531, end=10445609, progress=True' | jq '.|select(.[1].chain == "ETH") | .[1].in_hash' | tr -d '"' | sort | uniq -c]
|
[Second edit: The above command identified 33 in_hash TxIDs, of which only 8 had 99 occurences in that period. 6 were the ones discussed elsewhere in this description (5 swallowed, one successful) and the other two were also successful (not swallowed):
346EECBCAB5EB820AE3CA8F763FF266521CB0BDD445B67E15F0EF6DBA3A678F7
B8E2E1FF050DB9B385D5A2D0EAEC8090E3CF68E22AD08CB0BF7F1335555243D3]
Please tell me if there are other TxIDs which should be added to this.
Checking the TxIDs one by one:
In block 10445609 (the block of the final relevant Mimir key change) the outbound queue was empty, thus using it for these statuses.
In block 10386531, SigningTransactionPeriod was 300 (Constants) and MaxOutboundAttempts was 100 (Mimir), so a sign of being swallowed is to disappear from the outbound queue 30,000 blocks after the scheduled outbound height.
"outbound_signed": {
"scheduled_outbound_height": 10386372,
"blocks_since_scheduled": 59237,
"completed": false
}https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416372
There.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416373
Not there; swallowed.
"outbound_signed": {
"scheduled_outbound_height": 10386382,
"blocks_since_scheduled": 59227,
"completed": false
}https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416382
There.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416383
Not there; swallowed.
"outbound_signed": {
"scheduled_outbound_height": 10386460,
"blocks_since_scheduled": 59149,
"completed": false
}https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416460
There.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416461
Not there; swallowed.
"outbound_signed": {
"scheduled_outbound_height": 10386480,
"blocks_since_scheduled": 59129,
"completed": false
}https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416480
There.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416481
Not there; swallowed.
"outbound_signed": {
"scheduled_outbound_height": 10386525,
"blocks_since_scheduled": 59084,
"completed": false
}https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416525
There.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416526
Not there (the queue now completely empty); swallowed.
Further thoughts:
This had a successful TxOutItem:
https://thornode-v1.ninerealms.com/thorchain/tx/details/1D08A72D3783B02615B8FB8E8ACFFF41775CBEF4C368F69F6C092BA513D510F9?height=10445609
"outbound_height": 10386423
This was at least 41 blocks (4 minutes) later than the block 10386372 and 10386382 scheduled outbound heights of the first two of the above five, yet its signing was successful while the signing of those two were not; I am curious then about what prevented the signing of the other two.
|
Notably, by block 10425829 (HaltSigningETH to 1) the outbound queue was already empty from swallowing, with the sole ETH-related Mimir key change before then (block 10386531) being HaltETHChain.
|
https://thornode-v1.ninerealms.com/thorchain/tx/details/26FE9192A5AECBF895CF89D378374E15ABAED79A2D60772D852FBBBD86CEA4CD?height=10445609
"external_observed_height": 17035043,
"finalised_height": 10432448
Ethereum block 17035043: Apr-12-2023 11:09:47 PM +UTC
THORChain block 10386531: 2023-04-12T23:18:18.04245416Z
THORChain block 10432448: 2023-04-16T04:00:22.008802325Z
|
This suggests that (unlike the dangling five) this transaction was successfully included in an Ethereum block, but only observed later.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416423
There (1D08).
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=10416424
Not there (the appearance of being swallowed).
At https://etherscan.io/address/0x134E2cA341d0128FDbad3C35a6e2F512eE2cb3F5#internaltx the '0x26f' transaction is visible for the amount with string 4176, matching both /details/ .
Looking at the planned_out_txs to_address of each of the five:
https://etherscan.io/address/0xe7cb65965c38fc94cf40def0e6d4efc2aa45b600#tokentxns
No ETH.UDST of digits string 9151.
https://etherscan.io/address/0x118660d2382C4F78E5bc630FB3f9c1e5caEf469A#internaltx
Nothing at all.
https://etherscan.io/address/0x4A8d173993F4E137daDB69957dd37940465FD951#internaltx
Nothing of digits string 6091 (and in fact no 'internal transactions').
https://etherscan.io/address/0x9EE7f28Db62C114ac0eb148Cdeb5d1d57d0589F7#internaltx
Nothing at all.
https://etherscan.io/address/0xA91Cd685EEC7ac0892e9CA1f44acDd7F901dcb96#internaltx
Nothing of digits string 1525 (and in fact no 'internal transactions').
The above paragraph reassures me that the selected five appear to be true swallowed TxOutItems.
(Reminding myself of the effects of Halt[Chain]Chain versus HaltSigning[Chain]:
the second stops signing of TxOutItems,
whereas the first entirely postpones further scanning of that chain,
and marks the block scanner as unhealthy,
which also stops signing from that chain.)