Crash Report: Too many open file descriptors
BUG REPORT
Stack Trace or error message
Oct 02 00:45:16 fileserver systemd[1]: Started Sia Daemon.
Oct 02 00:45:16 fileserver siad[1084841]: Sia Daemon v1.5.6
Oct 02 00:45:16 fileserver siad[1084841]: Git Revision 0b253ab18
Oct 02 00:45:16 fileserver siad[1084841]: Loading...
Oct 02 00:45:16 fileserver siad[1084841]: (1/9) Loading siad...
Oct 02 00:45:16 fileserver siad[1084841]: (2/9) Loading gateway...
Oct 02 00:45:16 fileserver siad[1084841]: (3/9) Loading consensus...
Oct 02 00:45:19 fileserver siad[1084841]: (4/9) Loading transaction pool...
Oct 02 00:45:19 fileserver siad[1084841]: (5/9) Loading wallet...
Oct 02 00:45:22 fileserver siad[1084841]: (6/9) Loading feemanager...
Oct 02 00:45:22 fileserver siad[1084841]: (7/9) Loading host...
Oct 02 00:46:57 fileserver siad[1084841]: (8/9) Loading renter...
Oct 02 00:46:58 fileserver siad[1084841]: (9/9) Loading accounting...
Oct 02 00:46:58 fileserver siad[1084841]: API is now available, synchronous startup completed in 102.102 seconds
Oct 02 00:46:58 fileserver siad[1084841]: Sia Wallet Password found, attempting to auto-unlock wallet
Oct 02 00:47:06 fileserver siad[1084841]: Auto-unlock successful.
Oct 02 00:47:06 fileserver siad[1084841]: Finished full setup in 1m50s
Oct 05 06:38:23 fileserver siad[1084841]: goroutine 16451322 [running]:
Oct 05 06:38:23 fileserver siad[1084841]: runtime/debug.Stack(0xb9, 0x120, 0xc036d7e180)
Oct 05 06:38:23 fileserver siad[1084841]: runtime/debug/stack.go:24 +0x9d
Oct 05 06:38:23 fileserver siad[1084841]: runtime/debug.PrintStack()
Oct 05 06:38:23 fileserver siad[1084841]: runtime/debug/stack.go:16 +0x22
Oct 05 06:38:23 fileserver siad[1084841]: gitlab.com/NebulousLabs/log.(*Logger).Severe(0xc000114a80, 0xc05a84aed0, 0x2, 0x2)
Oct 05 06:38:23 fileserver siad[1084841]: gitlab.com/NebulousLabs/log@v0.0.0-20200604091839-0ba4a941cdc2/log.go:157 +0x333
Oct 05 06:38:23 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).commit.func1(0xc02220eb00, 0xc0002ef768)
Oct 05 06:38:23 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:186 +0x34c
Oct 05 06:38:23 fileserver siad[1084841]: created by gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).commit
Oct 05 06:38:23 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:170 +0x87
Oct 05 06:38:23 fileserver siad[1084841]: Severe error: (Sia v1.5.6, Release: release) Unable to open temporary settings file for writing: open /home/siad/Sia/host/contractmanager/contractmanager.json_temp: too many open files
Oct 05 06:38:24 fileserver siad[1084841]: goroutine 16451367 [running]:
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug.Stack(0x73, 0x80, 0xc02add4100)
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug/stack.go:24 +0x9d
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug.PrintStack()
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug/stack.go:16 +0x22
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/log.(*Logger).Severe(0xc000114a80, 0xc026a8bf90, 0x2, 0x2)
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/log@v0.0.0-20200604091839-0ba4a941cdc2/log.go:157 +0x333
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).syncResources.func1(0xc036373840, 0xc0002ef768)
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:38 +0x218
Oct 05 06:38:24 fileserver siad[1084841]: created by gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).syncResources
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:26 +0x93
Oct 05 06:38:24 fileserver siad[1084841]: Severe error: (Sia v1.5.6, Release: release) ERROR: unable to sync the contract manager settings: invalid argument
Oct 05 06:38:24 fileserver siad[1084841]: goroutine 16451367 [running]:
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug.Stack(0x106, 0x200, 0xc034007560)
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug/stack.go:24 +0x9d
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug.PrintStack()
Oct 05 06:38:24 fileserver siad[1084841]: runtime/debug/stack.go:16 +0x22
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/log.(*Logger).Severe(0xc000114a80, 0xc026a8bf90, 0x2, 0x2)
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/log@v0.0.0-20200604091839-0ba4a941cdc2/log.go:157 +0x333
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).syncResources.func1(0xc036373840, 0xc0002ef768)
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:56 +0x3e5
Oct 05 06:38:24 fileserver siad[1084841]: created by gitlab.com/NebulousLabs/Sia/modules/host/contractmanager.(*writeAheadLog).syncResources
Oct 05 06:38:24 fileserver siad[1084841]: gitlab.com/NebulousLabs/Sia/modules/host/contractmanager/writeaheadlogsync.go:26 +0x93
Oct 05 06:38:24 fileserver siad[1084841]: Severe error: (Sia v1.5.6, Release: release) ERROR: unable to atomically copy the contract manager settings: rename /home/siad/Sia/host/contractmanager/contractmanager.json_temp /home/siad/S
Oct 05 06:39:37 fileserver siad[1084841]: goroutine 16453001 [running]:
Oct 05 06:39:37 fileserver siad[1084841]: runtime/debug.Stack(0xb9, 0x200, 0xc033ffa000)
Oct 05 06:39:37 fileserver siad[1084841]: runtime/debug/stack.go:24 +0x9d
Oct 05 06:39:37 fileserver siad[1084841]: runtime/debug.PrintStack()
Oct 05 06:39:37 fileserver siad[1084841]: runtime/debug/stack.go:16 +0x22
Oct 05 06:39:37 fileserver siad[1084841]: gitlab.com/NebulousLabs/log.(*Logger).Severe(0xc000114a80, 0xc047188ed0, 0x2, 0x2)
Expected Behavior
The host not to crash.
Observed Behavior
- Crashes with stack trace above.
How to reproduce it (as minimally and precisely as possible)
- Host for a while.
- Get a big contract.
- Have that contract expire.
- Lose all collateral for that contract.
Note that I have increased my max file descriptors count. Proof:
root@fileserver:/home/siad# ulimit -n
97816
Environment
- Sia version: v1.5.6.
- OS: Ubuntu 20.04.3.