Host failed to submit proof before deadline while online and wallet unlocked
BUG REPORT
Seeing a bug on a couple of hosts where their node is online and wallet is unlocked after the expiration height but before proof deadline, but contracts stay unresolved within Sia even after the proof deadline has passed and Sia has been restarted.
I successfully reproduced the error on my own host by taking it offline right before the storage obligation with ID 9cff7dd0fd6279c3393cd61fbeead7fe0a3678a6391e5a01a3369057312599ff
was set to expire and bringing it back online before the proof deadline.
Stack Trace or error message
N/A
Expected Behavior
- Host goes offline at block
214072
- Storage obligation expires at block
214087
- Host comes back online at block
214120
well before the proof deadline at214231
- Host submits storage proof for storage obligation with id
9cff7dd0fd6279c3393cd61fbeead7fe0a3678a6391e5a01a3369057312599ff
Observed Behavior
- Host goes offline at block
214072
- Storage obligation expires at block
214087
- Host comes back online at block
214120
well before the proof deadline at214231
- Host submits storage proof for other storage obligations but seems to completely miss
9cff7dd0fd6279c3393cd61fbeead7fe0a3678a6391e5a01a3369057312599ff
- Proof deadline passes at block
214232
storage obligation remains unresolved even though host was online, unlocked, synced and submitted other valid storage proofs during that time period.
At the very least I would expect the status to change to "failed" instead of staying "unresolved".
How to reproduce it (as minimally and precisely as possible)
- Identify a storage obligation that is near expiration
- 10 blocks before the expiration height stop Sia
- Before the proof deadline start Sia back up, unlock wallet
- Wait for the proof deadline to fully elapse.
Environment
- Sia version: v1.4.0 official
- OS: Ubuntu 19.04 server
Additional Information
Output from my hosting node from block 214183
around July 2nd 11:00 UTC
siad
Sia Daemon v1.4.0
Git Revision 0e2bb7053
Loading...
(0/6) Loading siad...
(1/6) Loading gateway...
(2/6) Loading consensus...
(3/6) Loading transaction pool...
(4/6) Loading wallet...
(5/6) Loading host...
(6/6) Loading renter...
Sia Wallet Password found, attempting to auto-unlock wallet
Auto-unlock successful.
Finished loading in 65.92297553 seconds
siac consensus
Synced: Yes
Block: 000000000000000088176baa7df74937b888b741365a9b6e3f617e43a9e445d0
Height: 214183
Target: [0 0 0 0 0 0 0 8 210 22 85 25 17 202 179 228 219 14 191 237 181 92 12 142 61 153 135 220 67 85 57 214]
Difficulty: 2091312686064790800
siac wallet
Wallet status:
Encrypted, Unlocked
Height: 214183
Confirmed Balance: 23.19 KS
Unconfirmed Delta: +0 H
Exact: 23190544839404024003796886270 H
Siafunds: 0 SF
Siafund Claims: 0 H
Estimated Fee: 30 mS / KB
Here's the storage obligation from my host at block 214234
after proofdeadline
and after restarting Sia:
{
"contractcost": "150000000000000000000000",
"datasize": 4194304,
"lockedcollateral": "203135239999796219999988",
"obligationid": "9cff7dd0fd6279c3393cd61fbeead7fe0a3678a6391e5a01a3369057312599ff",
"potentialdownloadrevenue": "0",
"potentialstoragerevenue": "1516939946678802186240",
"potentialuploadrevenue": "8421376000000000000",
"riskedcollateral": "2527980421138203279360",
"sectorrootscount": 1,
"transactionfeesadded": "40050000000000000000000",
"transactionid": "7db3a3e90f9f5f1e95f605aac9b11d718b530caddb31af93823d8650660e7fb1",
"expirationheight": 214087,
"negotiationheight": 201141,
"proofdeadline": 214231,
"obligationstatus": "obligationUnresolved",
"originconfirmed": true,
"proofconfirmed": false,
"proofconstructed": false,
"revisionconfirmed": true,
"revisionconstructed": false
}
I've attached my host log. I removed every instance of incoming RPCLoopEnter failed: EOF
from the log.
Some interesting lines from my host.log file:
2019/07/02 13:06:21.916233 storageobligations.go:980: Host error when funding a storage proof transaction fee: wallet must be unlocked before it can be used
2019/07/02 13:06:23.565027 storageobligations.go:980: Host error when funding a storage proof transaction fee: wallet must be unlocked before it can be used
It looks like my host tried to submit a storage proof before the wallet was auto unlocked, but I don't know which obligation id failed. It then successfully submitted other proofs after the wallet was unlocked.
Here are some other storage obligation IDs that I found when talking to another host about this issue. These storage obligations appear to stay as unresolved indefinitely well after the proof deadline has passed on their node even after multiple reboots.
749927ed76145585099f20a4a5b6ee1e2b18b05c97e915b770f42f48e7498647
74b95fb709b6334b6f54cc76026565bd74e4f28aab274fe8774b0e3a954b87c6
b272dfaa77e891cd864594f30156396e40518590b4486f9a6b0fd70e41a37277
8ed3f34b7e3c7b26e7f3e68f46c6d10b81d8734addf9b0483bab73b330c2977c