...
 
Commits (235)
......@@ -84,6 +84,7 @@ lockcheckpkgs = \
./cmd/siac \
./modules/host/mdm \
./modules/renter/hostdb \
./modules/renter/proto \
# run determines which tests run when running any variation of 'make test'.
run = .
......
- Add testing infrastructure to validate the output of siac commands.
- Add root siac Cobra command test with subtests.
\ No newline at end of file
- Optimise writes when we execute an MDM program on the host to lower overall
(upload) bandwidth consumption.
- Add `siac renter workers ea` command to siac
- Add `siac renter workers pt` command to siac
- Add `siac renter workers rj` command to siac
- Add `siac renter workers hsj` command to siac
- Add default timeouts to opening a stream on the mux
- add support for write MDM programs to host
\ No newline at end of file
- Add `EphemeralAccountExpiry` and `MaxEphemeralAccountBalance` to the Host's ExternalSettings
\ No newline at end of file
......@@ -30,6 +30,11 @@ Renter:
* `siac renter ls` list all renter files and subdirectories
* `siac renter upload [filepath] [nickname]` upload a file
* `siac renter download [nickname] [filepath]` download a file
* `siac renter workers` show worker status
* `siac renter workers ea` show worker account status
* `siac renter workers pt` show worker price table status
* `siac renter workers rj` show worker read jobs status
* `siac renter workers hsj` show worker has sector jobs status
Full Descriptions
......@@ -159,6 +164,26 @@ corresponding field flag, for example '--amount 500SC'.
you will use to refer to that file in the network. For example, it is common to
have the nickname be the same as the filename.
* `siac renter workers` shows a detailed overview of all workers. It shows
information about their accounts, contract and download and upload status.
* `siac renter workers ea` shows a detailed overview of the workers' ephemeral
account statuses, such as balance information, whether its on cooldown or not
and potentially the most recent error.
* `siac renter workers pt` shows a detailed overview of the workers's price table
statuses, such as when it was updated, when it expires, whether its on cooldown
or not and potentially the most recent error.
* `siac renter workers rj` shows information about the read jobs queue. How many
jobs are in the queue and their average completion time. In case there was an
error it will also display the most recent error and when it occurred.
* `siac renter workers hsj` shows information about the has sector jobs queue.
How many jobs are in the queue and their average completion time. In case
there was an error it will also display the most recent error and when it
occurred.
### Skykey tasks
TODO - Fill in
......@@ -258,3 +283,197 @@ siacoin address.
wallet, supplied by the `init` command. The wallet must be initialized and
unlocked before any actions can take place.
Siac Command Output Testing
===========================
New type of testing siac command line commands is now available from go tests.
Siac is using [Cobra](https://github.com/spf13/cobra) golang library to
generate command line commands (and subcommands) interface. In
`cmd/siac/main.go` file root siac Cobra command with all subcommands is created
using `initCmds()`, siac/siad node instance specific flags of siac commands are
initialized using `initClient(...)`.
## Test Group Structure
Pseudo code example of a test group:
```
func TestGroup() {
// Create test inputs
create test node
init Cobra command with subcommands and flags
create regex pattern constants
// Create subtests
define subtests
// Execute subtests
run subtests
}
```
## Test Inputs
The most of the siac tests require running instance of `siad` to execute the
tests against. A new instance of `siad` can be created using `newTestNode`.
Note that some of the `siac` tests don't require running an instance of `siad`.
This is the case when we're testing unknown `siac` subcommand or an unknown
command/subcommand flag for example, because these error cases are handled by
Cobra library itself.
Before testing siac Cobra command(s), siac Cobra command with its subcommands
and flags must be built and initialized. This is done by
`getRootCmdForSiacCmdsTests()` helper function.
## Subtests
Subtests are defined using `siacCmdSubTest` struct:
```
type siacCmdSubTest struct {
name string
test siacCmdTestFn
cmd *cobra.Command
cmdStrs []string
expectedOutPattern string
}
```
### name
`name` is the name of a subtest to appear in report.
### test
`test` is a subtest helper function that executes subtest.
### cmd
`cmd` is an initialized root Cobra command with all subcommands and flags.
### cmdStrs
`cmdStrs` is a list of string values you would normally enter to the command
line, but without leading `siac` and each space between command, subcommand(s),
flag(s) or parameter(s) starting a new string in a list.
Examples:
|CLI command|cmdStrs|
|---|---|
|./siac|cmdStrs: []string{},|
|./siac -h|cmdStrs: []string{"-h"},|
|./siac --address localhost:5555|cmdStrs: []string{"--address", "localhost:5555"},|
|./siac renter --address localhost:5555|cmdStrs: []string{"renter", "--address", "localhost:5555"},|
### expectedOutPattern
`expectedOutPattern` is expected regex pattern string to test actual output
against. It can be a multiline string to test complete output from beginning
(starting with `^`) till end (ending with `$`) or just a smaller pattern
testing multiple lines, a single line or just a part of a line in the complete
output.
Note that each siac command handler has to be prepared for these tests, for
more information see [below](#preparation-of-command-handler-for-cobra-Output-tests).
## Errors
In case of failure in the executed subtest, error log output from
`testGenericSiacCmd()` in `cmd/siac/helpers_test.go` will include the following 5 items:
* Regex pattern didn't match between row x, and row y
* Regex pattern part that didn't match
* ----- Expected output pattern: -----
* ----- Actual Cobra output: -----
* ----- Actual Sia output: -----
Error log example with 5 above items (part `...` of the message is cut):
```
=== RUN TestRootSiacCmd
=== RUN TestRootSiacCmd/TestRootCmdWithShortAddressFlagIPv6
--- FAIL: TestRootSiacCmd (2.18s)
maincmd_test.go:28: siad API address: [::]:35103
--- FAIL: TestRootSiacCmd/TestRootCmdWithShortAddressFlagIPv6 (0.02s)
helpers_test.go:141: Regex pattern didn't match between row 5, and row 5
helpers_test.go:142: Regex pattern part that didn't match:
Wallet XXX:
helpers_test.go:150: ----- Expected output pattern: -----
helpers_test.go:151: ^Consensus:
Synced: (No|Yes)
Height: [\d]+
Wallet XXX:
( Status: Locked| Status: unlocked
Siacoin Balance: [\d]+(\.[\d]*|) (SC|KS|MS))
...
$
helpers_test.go:153: ----- Actual Cobra output: -----
helpers_test.go:154:
helpers_test.go:156: ----- Actual Sia output: -----
helpers_test.go:157: Consensus:
Synced: Yes
Height: 14
Wallet:
Status: unlocked
Siacoin Balance: 3.3 MS
...
helpers_test.go:159:
FAIL
coverage: 5.3% of statements
FAIL gitlab.com/NebulousLabs/Sia/cmd/siac 2.242s
FAIL
```
Expected output regex pattern can have multiple lines and because spotting
errors in complex regex pattern matching can be difficult `testGenericSiacCmd`
tests in a for loop at first only the first line of the regex pattern, then
first 2 lines of the regex pattern, adding one more line each iteration. If
there is a regex pattern match error, it prints the line number of the regex
that didn't match. E.g. there is a 20 line of expected regex pattern, it passed
to test first 11 lines of regex but fails to match when first 12 lines are
matched against, it prints that it failed to match line 12 of regex pattern and
prints the content of 12th line.
Then it prints the complete expected regex pattern and actual Cobra output and
actual siac output. There are two actual outputs, because unknown subcommands,
unknown flags and command/subcommand help requests are handled by Cobra
library, while the rest is the output written to stdout by siac command
handlers.
## Examples
First examples of siac Cobra command tests are tests located in
`cmd/siac/maincmd_test.go` file in `TestRootSiacCmd` test group, helpers for
these tests are located in `cmd/siac/helpers_test.go` file.
Simplified example code:
```
func TestRootSiacCmd(t *testing.T) {
...
n, err := newTestNode(groupDir)
...
root := getRootCmdForSiacCmdsTests(t, groupDir)
...
regexPatternConstantX := "..."
...
subTests := []siacCmdSubTest{
{
name: "TestRootCmdWithShortAddressFlagIPv6",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-a", IPv6addr},
expectedOutPattern: regexPatternConstantX,
},
...
}
err = runSiacCmdSubTests(t, subTests)
...
}
```
......@@ -28,10 +28,12 @@ var (
)
// feeInfo is a helper struct for gathering some information about the fees
//
// NOTE: fields are exported so that json.MarshalIndent can be used
type feeInfo struct {
appUID modules.AppUID
fees []modules.AppFee
totalAmount types.Currency
AppUID modules.AppUID
Fees []modules.AppFee
TotalAmount types.Currency
}
// feemanagercmd prints out the basic information about the FeeManager and lists
......@@ -58,7 +60,10 @@ func feemanagercmd() {
fmt.Fprintf(w, " Next FeePayoutHeight:\t%v\n", fmg.PayoutHeight)
fmt.Fprintf(w, " Number Pending Fees:\t%v\n", len(pendingFees.PendingFees))
fmt.Fprintf(w, " Total Amount Pending:\t%v\n", pendingTotal.HumanString())
w.Flush()
err = w.Flush()
if err != nil {
die(err)
}
// Print Pending Fees
if len(pendingFees.PendingFees) == 0 {
......@@ -68,12 +73,15 @@ func feemanagercmd() {
fmt.Fprintln(w, "\nPending Fees:")
fmt.Fprintln(w, " AppUID\tFeeUID\tAmount\tRecurring\tPayout Height\tTxn Created")
for _, feeInfo := range fees {
for _, fee := range feeInfo.fees {
for _, fee := range feeInfo.Fees {
fmt.Fprintf(w, " %v\t%v\t%v\t%v\t%v\t%v\n",
fee.AppUID, fee.FeeUID, fee.Amount, fee.Recurring, fee.PayoutHeight, fee.TransactionCreated)
fee.AppUID, fee.FeeUID, fee.Amount.HumanString(), fee.Recurring, fee.PayoutHeight, fee.TransactionCreated)
}
}
w.Flush()
err = w.Flush()
if err != nil {
die(err)
}
// Check if verbose output was requested
if !feeManagerVerbose {
......@@ -98,12 +106,15 @@ func feemanagercmd() {
fmt.Fprintf(w, " Total Amount Paid:\t%v\n", paidTotal.HumanString())
fmt.Fprintln(w, " AppUID\tFeeUID\tAmount\tPayout Height")
for _, feeInfo := range fees {
for _, fee := range feeInfo.fees {
for _, fee := range feeInfo.Fees {
fmt.Fprintf(w, " %v\t%v\t%v\t%v\n",
fee.AppUID, fee.FeeUID, fee.Amount, fee.PayoutHeight)
fee.AppUID, fee.FeeUID, fee.Amount.HumanString(), fee.PayoutHeight)
}
}
w.Flush()
err = w.Flush()
if err != nil {
die(err)
}
}
// feemanagercancelfeecmd cancels a fee
......@@ -127,51 +138,40 @@ func parseFees(fees []modules.AppFee) ([]feeInfo, types.Currency) {
// Grab the entry from the map or create it
fi, ok := appToFeesMap[fee.AppUID]
if !ok {
fi = feeInfo{appUID: fee.AppUID}
fi = feeInfo{AppUID: fee.AppUID}
}
// Update the totalAmount and the entry information
totalAmount = totalAmount.Add(fee.Amount)
fi.totalAmount = fi.totalAmount.Add(fee.Amount)
fi.fees = append(fi.fees, fee)
fi.TotalAmount = fi.TotalAmount.Add(fee.Amount)
fi.Fees = append(fi.Fees, fee)
// Update Map
appToFeesMap[fee.AppUID] = fi
}
// Covert map to slice for sorting
// Convert the map to a slice and sort
var feeInfos []feeInfo
for _, fi := range appToFeesMap {
// Sort to slice of fees for each AppUID
sort.Sort(byAmount(fi.fees))
// Sort the slice of fees for each AppUID in descending order by the Amount.
// If the Amount for two fees is the same then sort by PayoutHeight in
// ascending order so that the fees are ordered by when they would be
// charged.
sort.SliceStable(fi.Fees, func(i, j int) bool {
cmp := fi.Fees[i].Amount.Cmp(fi.Fees[j].Amount)
if cmp == 0 {
return fi.Fees[i].PayoutHeight < fi.Fees[j].PayoutHeight
}
return cmp > 0
})
feeInfos = append(feeInfos, fi)
}
// Sort Slice and return
sort.Sort(byTotalAmount(feeInfos))
// Sort the slice of feeInfos by the total amount in descending order and
// return
sort.SliceStable(feeInfos, func(i, j int) bool {
cmp := feeInfos[i].TotalAmount.Cmp(feeInfos[j].TotalAmount)
return cmp > 0
})
return feeInfos, totalAmount
}
// byAmount is an implementation of a sort interface to sort the Fees by amount
type byAmount []modules.AppFee
func (s byAmount) Len() int { return len(s) }
func (s byAmount) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s byAmount) Less(i, j int) bool {
cmp := s[i].Amount.Cmp(s[j].Amount)
if cmp == 0 {
return s[i].PayoutHeight > s[j].PayoutHeight
}
return cmp > 0
}
// byTotalAmount is an implementation of a sort interface to sort the feeInfo by
// totalAmount
type byTotalAmount []feeInfo
func (s byTotalAmount) Len() int { return len(s) }
func (s byTotalAmount) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s byTotalAmount) Less(i, j int) bool {
cmp := s[i].totalAmount.Cmp(s[j].totalAmount)
return cmp > 0
}
package main
import (
"fmt"
"reflect"
"testing"
"gitlab.com/NebulousLabs/Sia/modules"
"gitlab.com/NebulousLabs/Sia/siatest"
"gitlab.com/NebulousLabs/Sia/types"
"gitlab.com/NebulousLabs/fastrand"
)
// TestParseFees tests the parseFees function to ensure expected return values
func TestParseFees(t *testing.T) {
// Create AppUIDs
cheapApp := modules.AppUID("cheapApp")
expensiveApp := modules.AppUID("expensiveApp")
// Create FeeUIDs
feeUID1 := modules.FeeUID("fee1")
feeUID2 := modules.FeeUID("fee2")
feeUID3 := modules.FeeUID("fee3")
feeUID4 := modules.FeeUID("fee4")
// Create Fee Amounts
feeAmount1 := types.NewCurrency64(fastrand.Uint64n(1000))
feeAmount2 := feeAmount1.Add(types.NewCurrency64(fastrand.Uint64n(1000)))
feeAmount3 := feeAmount2.Add(types.NewCurrency64(fastrand.Uint64n(1000)))
// Create Fees
cheapFee1 := modules.AppFee{
Amount: feeAmount1,
AppUID: cheapApp,
FeeUID: feeUID1,
}
cheapFee2 := modules.AppFee{
Amount: feeAmount2,
AppUID: cheapApp,
FeeUID: feeUID2,
}
expensiveFee1 := modules.AppFee{
Amount: feeAmount1,
AppUID: expensiveApp,
FeeUID: feeUID1,
}
expensiveFee2 := modules.AppFee{
Amount: feeAmount2,
AppUID: expensiveApp,
FeeUID: feeUID2,
PayoutHeight: 100,
}
expensiveFee3 := modules.AppFee{
Amount: feeAmount2,
AppUID: expensiveApp,
FeeUID: feeUID3,
}
expensiveFee4 := modules.AppFee{
Amount: feeAmount3,
AppUID: expensiveApp,
FeeUID: feeUID4,
}
// Create unsorted list
fees := []modules.AppFee{cheapFee1, cheapFee2, expensiveFee1, expensiveFee2, expensiveFee3, expensiveFee4}
// Create expected sorted list
expectedOrder := []feeInfo{
{
AppUID: expensiveApp,
Fees: []modules.AppFee{expensiveFee4, expensiveFee3, expensiveFee2, expensiveFee1},
TotalAmount: feeAmount1.Add(feeAmount2.Add(feeAmount2.Add(feeAmount3))),
},
{
AppUID: cheapApp,
Fees: []modules.AppFee{cheapFee2, cheapFee1},
TotalAmount: feeAmount1.Add(feeAmount2),
},
}
// Parse the Fees
parsedFees, totalAmount := parseFees(fees)
// Check the total
expectedTotal := expectedOrder[0].TotalAmount.Add(expectedOrder[1].TotalAmount)
if totalAmount.Cmp(expectedTotal) != 0 {
t.Errorf("Expected total to be %v but was %v", expectedTotal.HumanString(), totalAmount.HumanString())
}
// Check the sorting of the fees
if !reflect.DeepEqual(parsedFees, expectedOrder) {
fmt.Println("Expected Order:")
siatest.PrintJSON(expectedOrder)
fmt.Println("Parsed Order:")
siatest.PrintJSON(parsedFees)
t.Fatal("Fees not sorted as expected")
}
}
package main
import (
"bytes"
"io"
"os"
"regexp"
"strings"
"testing"
"github.com/spf13/cobra"
"gitlab.com/NebulousLabs/Sia/node"
"gitlab.com/NebulousLabs/Sia/node/api/client"
"gitlab.com/NebulousLabs/Sia/persist"
......@@ -11,6 +16,28 @@ import (
"gitlab.com/NebulousLabs/errors"
)
// outputCatcher is a helper struct enabling to catch stdout and stderr during
// tests
type outputCatcher struct {
origStdout *os.File
origStderr *os.File
outW *os.File
outC chan string
}
// siacCmdSubTest is a helper struct for running siac Cobra commands subtests
// when subtests need command to run and expected output
type siacCmdSubTest struct {
name string
test siacCmdTestFn
cmd *cobra.Command
cmdStrs []string
expectedOutPattern string
}
// siacCmdTestFn is a type of function to pass to siacCmdSubTest
type siacCmdTestFn func(*testing.T, *cobra.Command, []string, string)
// subTest is a helper struct for running subtests when tests can use the same
// test http client
type subTest struct {
......@@ -18,6 +45,107 @@ type subTest struct {
test func(*testing.T, client.Client)
}
// escapeRegexChars takes string and escapes all special regex characters
func escapeRegexChars(s string) string {
res := s
chars := `\+*?^$.[]{}()|/`
for _, c := range chars {
res = strings.ReplaceAll(res, string(c), `\`+string(c))
}
return res
}
// executeSiacCommand is a pass-through function to execute siac cobra command
func executeSiacCommand(root *cobra.Command, args ...string) (output string, err error) {
// Recover from expected die() panic, rethrow any not expected panic
defer func() {
if rec := recover(); rec != nil {
// We are recovering from panic
if err, ok := rec.(error); !ok || err.Error() != errors.New("die panic for testing").Error() {
// This is not our expected die() panic, rethrow panic
panic(rec)
}
}
}()
_, output, err = executeSiacCommandC(root, args...)
return output, err
}
// executeSiacCommandC executes cobra command
func executeSiacCommandC(root *cobra.Command, args ...string) (c *cobra.Command, output string, err error) {
buf := new(bytes.Buffer)
root.SetOut(buf)
root.SetErr(buf)
root.SetArgs(args)
c, err = root.ExecuteC()
return c, buf.String(), err
}
// getRootCmdForSiacCmdsTests creates and initializes a new instance of siac Cobra
// command
func getRootCmdForSiacCmdsTests(dir string) *cobra.Command {
// create new instance of siac cobra command
root := initCmds()
// initialize a siac cobra command
initClient(root, &statusVerbose, &httpClient, &dir)
return root
}
// newOutputCatcher starts catching stdout and stderr in tests
func newOutputCatcher() (outputCatcher, error) {
// redirect stdout, stderr
origStdout := os.Stdout
origStderr := os.Stderr
r, w, err := os.Pipe()
if err != nil {
return outputCatcher{}, errors.New("Error opening pipe")
}
os.Stdout = w
os.Stderr = w
// capture redirected output
outC := make(chan string)
go func() {
var b bytes.Buffer
io.Copy(&b, r)
outC <- b.String()
}()
c := outputCatcher{
origStdout: origStdout,
origStderr: origStderr,
outW: w,
outC: outC,
}
return c, nil
}
// newTestNode creates a new Sia node for a test
func newTestNode(dir string) (*siatest.TestNode, error) {
n, err := siatest.NewNode(node.AllModules(dir))
if err != nil {
return nil, errors.AddContext(err, "Error creating a new test node")
}
return n, nil
}
// runSiacCmdSubTests is a helper function to run siac Cobra command subtests
// when subtests need command to run and expected output
func runSiacCmdSubTests(t *testing.T, tests []siacCmdSubTest) error {
// Run subtests
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
test.test(t, test.cmd, test.cmdStrs, test.expectedOutPattern)
})
}
return nil
}
// runSubTests is a helper function to run the subtests when tests can use the
// same test http client
func runSubTests(t *testing.T, directory string, tests []subTest) error {
......@@ -40,15 +168,6 @@ func runSubTests(t *testing.T, directory string, tests []subTest) error {
return nil
}
// newTestNode creates a new Sia node for a test
func newTestNode(dir string) (*siatest.TestNode, error) {
n, err := siatest.NewNode(node.AllModules(dir))
if err != nil {
return nil, errors.AddContext(err, "Error creating a new test node")
}
return n, nil
}
// siacTestDir creates a temporary Sia testing directory for a cmd/siac test,
// removing any files or directories that previously existed at that location.
// This should only every be called once per test. Otherwise it will delete the
......@@ -60,3 +179,90 @@ func siacTestDir(testName string) string {
}
return path
}
// testGenericSiacCmd is a helper function to test siac cobra commands
// specified in cmds for expected output regex pattern
func testGenericSiacCmd(t *testing.T, root *cobra.Command, cmds []string, expOutPattern string) {
// catch stdout and stderr
c, err := newOutputCatcher()
if err != nil {
t.Fatal("Error starting catching stdout/stderr", err)
}
// execute command
cobraOutput, _ := executeSiacCommand(root, cmds...)
// stop catching stdout/stderr, get catched outputs
siaOutput, err := c.stop()
if err != nil {
t.Fatal("Error stopping catching stdout/stderr", err)
}
// check output
// There are 2 types of output:
// 1) Output generated by Cobra commands (e.g. when using -h) or Cobra
// errors (e.g. unknown cobra commands or flags).
// 2) Output generated by siac to stdout and to stderr
var output string
if cobraOutput != "" {
output = cobraOutput
} else if siaOutput != "" {
output = siaOutput
} else {
t.Fatal("There was no output")
}
// check regex pattern by increasing rows so it is easier to spot the regex
// match issues, do not split on regex pattern rows with open regex groups
regexErr := false
regexRows := strings.Split(expOutPattern, "\n")
offsetFromLastOKRow := 0
for i := 0; i < len(regexRows); i++ {
// test only first i+1 rows from regex pattern
expSubPattern := strings.Join(regexRows[0:i+1], "\n")
// do not split on open regex group "("
openRegexGroups := strings.Count(expSubPattern, "(") - strings.Count(expSubPattern, `\(`)
closedRegexGroups := strings.Count(expSubPattern, ")") - strings.Count(expSubPattern, `\)`)
if openRegexGroups != closedRegexGroups {
offsetFromLastOKRow++
continue
}
validPattern := regexp.MustCompile(expSubPattern)
if !validPattern.MatchString(output) {
t.Logf("Regex pattern didn't match between row %v, and row %v", i+1-offsetFromLastOKRow, i+1)
t.Logf("Regex pattern part that didn't match:\n%s", strings.Join(regexRows[i-offsetFromLastOKRow:i+1], "\n"))
regexErr = true
break
}
offsetFromLastOKRow = 0
}
if regexErr {
t.Log("----- Expected output pattern: -----")
t.Log(expOutPattern)
t.Log("----- Actual Cobra output: -----")
t.Log(cobraOutput)
t.Log("----- Actual Sia output: -----")
t.Log(siaOutput)
t.Fatal()
}
}
// stop stops catching stdout and stderr, catched output is
// returned
func (c outputCatcher) stop() (string, error) {
// stop Stdout
err := c.outW.Close()
if err != nil {
return "", err
}
os.Stdout = c.origStdout
os.Stderr = c.origStderr
output := <-c.outC
return output, nil
}
......@@ -131,10 +131,17 @@ func wrap(fn interface{}) func(*cobra.Command, []string) {
}
}
// die prints its arguments to stderr, then exits the program with the default
// error code.
// die prints its arguments to stderr, in production exits the program with the
// default error code, during tests it passes panic so that tests can catch the
// panic and check printed errors
func die(args ...interface{}) {
fmt.Fprintln(os.Stderr, args...)
if build.Release == "testing" {
// In testing pass panic that can be catched and the test can continue
panic(errors.New("die panic for testing"))
}
// In production exit
os.Exit(exitCodeGeneral)
}
......@@ -240,6 +247,41 @@ func rateLimitSummary(download, upload int64) {
}
func main() {
// initialize commands
rootCmd = initCmds()
// initialize client
initClient(rootCmd, &statusVerbose, &httpClient, &siaDir)
// set API password if it was not set
setAPIPasswordIfNotSet()
// Check if the siaDir is set.
if siaDir == "" {
// No siaDir passed in, fetch the siaDir
siaDir = build.SiaDir()
}
// Check for Critical Alerts
alerts, err := httpClient.DaemonAlertsGet()
if err == nil && len(alerts.CriticalAlerts) > 0 {
printAlerts(alerts.CriticalAlerts, modules.SeverityCritical)
fmt.Println("------------------")
fmt.Printf("\n The above %v critical alerts should be resolved ASAP\n\n", len(alerts.CriticalAlerts))
}
// run
if err := rootCmd.Execute(); err != nil {
// Since no commands return errors (all commands set Command.Run instead of
// Command.RunE), Command.Execute() should only return an error on an
// invalid command or flag. Therefore Command.Usage() was called (assuming
// Command.SilenceUsage is false) and we should exit with exitCodeUsage.
os.Exit(exitCodeUsage)
}
}
// initCmds initializes root command and its subcommands
func initCmds() *cobra.Command {
root := &cobra.Command{
Use: os.Args[0],
Short: "Sia Client v" + build.Version,
......@@ -248,8 +290,6 @@ func main() {
}
// create command tree (alphabetized by root command)
rootCmd = root
root.AddCommand(consensusCmd)
consensusCmd.Flags().BoolVarP(&consensusCmdVerbose, "verbose", "v", false, "Display full consensus information")
......@@ -287,6 +327,7 @@ func main() {
renterFilesListCmd, renterFilesRenameCmd, renterFilesUnstuckCmd, renterFilesUploadCmd,
renterFuseCmd, renterPricesCmd, renterRatelimitCmd, renterSetAllowanceCmd,
renterSetLocalPathCmd, renterTriggerContractRecoveryScanCmd, renterUploadsCmd, renterWorkersCmd)
renterWorkersCmd.AddCommand(renterWorkersAccountsCmd, renterWorkersPriceTableCmd, renterWorkersReadJobsCmd, renterWorkersHasSectorJobSCmd)
renterAllowanceCmd.AddCommand(renterAllowanceCancelCmd)
renterContractsCmd.AddCommand(renterContractsViewCmd)
......@@ -375,13 +416,20 @@ func main() {
walletTransactionsCmd.Flags().Uint64Var(&walletStartHeight, "startheight", 0, " Height of the block where transaction history should begin.")
walletTransactionsCmd.Flags().Uint64Var(&walletEndHeight, "endheight", math.MaxUint64, " Height of the block where transaction history should end.")
// initialize client
root.Flags().BoolVarP(&statusVerbose, "verbose", "v", false, "Display additional siac information")
root.PersistentFlags().StringVarP(&httpClient.Address, "addr", "a", "localhost:9980", "which host/port to communicate with (i.e. the host/port siad is listening on)")
root.PersistentFlags().StringVarP(&httpClient.Password, "apipassword", "", "", "the password for the API's http authentication")
root.PersistentFlags().StringVarP(&siaDir, "sia-directory", "d", "", "location of the sia directory")
root.PersistentFlags().StringVarP(&httpClient.UserAgent, "useragent", "", "Sia-Agent", "the useragent used by siac to connect to the daemon's API")
return root
}
// initClient initializes client cmd flags and default values
func initClient(root *cobra.Command, verbose *bool, client *client.Client, siaDir *string) {
root.Flags().BoolVarP(verbose, "verbose", "v", false, "Display additional siac information")
root.PersistentFlags().StringVarP(&client.Address, "addr", "a", "localhost:9980", "which host/port to communicate with (i.e. the host/port siad is listening on)")
root.PersistentFlags().StringVarP(&client.Password, "apipassword", "", "", "the password for the API's http authentication")
root.PersistentFlags().StringVarP(siaDir, "sia-directory", "d", "", "location of the sia directory")
root.PersistentFlags().StringVarP(&client.UserAgent, "useragent", "", "Sia-Agent", "the useragent used by siac to connect to the daemon's API")
}
// setAPIPasswordIfNotSet sets API password if it was not set
func setAPIPasswordIfNotSet() {
// Check if the API Password is set
if httpClient.Password == "" {
// No password passed in, fetch the API Password
......@@ -392,27 +440,4 @@ func main() {
}
httpClient.Password = pw
}
// Check if the siaDir is set.
if siaDir == "" {
// No siaDir passed in, fetch the siaDir
siaDir = build.SiaDir()
}
// Check for Critical Alerts
alerts, err := httpClient.DaemonAlertsGet()
if err == nil && len(alerts.CriticalAlerts) > 0 {
printAlerts(alerts.CriticalAlerts, modules.SeverityCritical)
fmt.Println("------------------")
fmt.Printf("\n The above %v critical alerts should be resolved ASAP\n\n", len(alerts.CriticalAlerts))
}
// run
if err := root.Execute(); err != nil {
// Since no commands return errors (all commands set Command.Run instead of
// Command.RunE), Command.Execute() should only return an error on an
// invalid command or flag. Therefore Command.Usage() was called (assuming
// Command.SilenceUsage is false) and we should exit with exitCodeUsage.
os.Exit(exitCodeUsage)
}
}
package main
import (
"strings"
"testing"
"github.com/spf13/cobra"
"gitlab.com/NebulousLabs/Sia/build"
)
// TestRootSiacCmd tests root siac command for expected outputs. The test
// runs its own node and requires no service running at port 5555.
func TestRootSiacCmd(t *testing.T) {
if testing.Short() {
t.SkipNow()
}
t.Parallel()
// Create a test node for this test group
groupDir := siacTestDir(t.Name())
n, err := newTestNode(groupDir)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := n.Close(); err != nil {
t.Fatal(err)
}
}()
// Initialize siac root command with its subcommands and flags
root := getRootCmdForSiacCmdsTests(groupDir)
// define test constants:
// Regular expressions to check siac output
begin := "^"
nl := `
` // platform agnostic new line
end := "$"
// Capture root command usage for test comparison
// catch stdout and stderr
rootCmdUsagePattern := getCmdUsage(t, root)
IPv6addr := n.Address
IPv4Addr := strings.ReplaceAll(n.Address, "[::]", "localhost")
rootCmdOutPattern := `Consensus:
Synced: (No|Yes)
Height: \d+
Wallet:
( Status: Locked| Status: unlocked
Siacoin Balance: \d+(\.\d*|) (SC|KS|MS))
Renter:
Files: \d+
Total Stored: \d+(\.\d+|) ( B|kB|MB|GB|TB)
Total Contract Data: \d+(\.\d+|) ( B|kB|MB|GB|TB)
Min Redundancy: (\d+.\d{2}|-)
Active Contracts: \d+
Passive Contracts: \d+
Disabled Contracts: \d+`
rootCmdVerbosePartPattern := `Global Rate limits:
Download Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))
Upload Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))
Gateway Rate limits:
Download Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))
Upload Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))
Renter Rate limits:
Download Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))
Upload Speed: (no limit|\d+(\.\d+)? (B/s|KB/s|MB/s|GB/s|TB/s))`
connectionRefusedPattern := `Could not get consensus status: \[failed to get reader response; GET request failed; Get "?http://localhost:5555/consensus"?: dial tcp \[::1\]:5555: connect: connection refused\]`
siaClientVersionPattern := "Sia Client v" + strings.ReplaceAll(build.Version, ".", `\.`)
// Define subtests
// We can't test siad on default address (port) when test node has
// dynamically allocated port, we have to use node address.
subTests := []siacCmdSubTest{
{
name: "TestRootCmdWithShortAddressFlagIPv6",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-a", IPv6addr},
expectedOutPattern: begin + rootCmdOutPattern + nl + nl + end,
},
{
name: "TestRootCmdWithShortAddressFlagIPv4",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-a", IPv4Addr},
expectedOutPattern: begin + rootCmdOutPattern + nl + nl + end,
},
{
name: "TestRootCmdWithLongAddressFlagIPv6",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"--addr", IPv6addr},
expectedOutPattern: begin + rootCmdOutPattern + nl + nl + end,
},
{
name: "TestRootCmdWithLongAddressFlagIPv4",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"--addr", IPv4Addr},
expectedOutPattern: begin + rootCmdOutPattern + nl + nl + end,
},
{
name: "TestRootCmdWithVerboseFlag",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"--addr", IPv4Addr, "-v"},
expectedOutPattern: begin + rootCmdOutPattern + nl + nl + rootCmdVerbosePartPattern + nl + nl + end,
},
{
name: "TestRootCmdWithInvalidFlag",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-x"},
expectedOutPattern: begin + "Error: unknown shorthand flag: 'x' in -x" + nl + rootCmdUsagePattern + nl + end,
},
{
name: "TestRootCmdWithInvalidAddress",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-a", "localhost:5555"},
expectedOutPattern: begin + connectionRefusedPattern + nl + nl + end,
},
{
name: "TestRootCmdWithHelpFlag",
test: testGenericSiacCmd,
cmd: root,
cmdStrs: []string{"-h"},
expectedOutPattern: begin + siaClientVersionPattern + nl + nl + rootCmdUsagePattern + end,
},
}
// run tests
err = runSiacCmdSubTests(t, subTests)
if err != nil {
t.Fatal(err)
}
}
// getCmdUsage gets root command usage regex pattern by calling usage function
func getCmdUsage(t *testing.T, cmd *cobra.Command) string {
// Capture usage by calling a usage function
c, err := newOutputCatcher()
if err != nil {
t.Fatal("Error starting catching stdout/stderr", err)
}
usageFunc := cmd.UsageFunc()
err = usageFunc(cmd)
if err != nil {
t.Fatal("Error getting reference root siac usage", err)
}
baseUsage, err := c.stop()
// Escape regex special chars
usage := escapeRegexChars(baseUsage)
// Inject 2 missing rows
beforeHelpCommand := "Perform gateway actions"
helpCommand := " help Help about any command"
nl := `
`
usage = strings.ReplaceAll(usage, beforeHelpCommand, beforeHelpCommand+nl+helpCommand)
beforeHelpFlag := "the password for the API's http authentication"
helpFlag := ` -h, --help help for .*siac(\.test|)`
cmdUsagePattern := strings.ReplaceAll(usage, beforeHelpFlag, beforeHelpFlag+nl+helpFlag)
return cmdUsagePattern
}
......@@ -271,6 +271,34 @@ have a reasonable number (>30) of hosts in your hostdb.`,
Long: "View the status of the Renter's workers",
Run: wrap(renterworkerscmd),
}
renterWorkersAccountsCmd = &cobra.Command{
Use: "ea",
Short: "View the workers' ephemeral account",
Long: "View detailed information of the workers' ephemeral account",
Run: wrap(renterworkerseacmd),
}
renterWorkersPriceTableCmd = &cobra.Command{
Use: "pt",
Short: "View the workers's price table",
Long: "View detailed information of the workers' price table",
Run: wrap(renterworkersptcmd),
}
renterWorkersReadJobsCmd = &cobra.Command{
Use: "rj",
Short: "View the workers' read jobs",
Long: "View detailed information of the workers' read jobs",
Run: wrap(renterworkersrjcmd),
}
renterWorkersHasSectorJobSCmd = &cobra.Command{
Use: "hsj",
Short: "View the workers' has sector jobs",
Long: "View detailed information of the workers' has sector jobs",
Run: wrap(renterworkershsjcmd),
}
)
// abs returns the absolute representation of a path.
......@@ -1453,7 +1481,7 @@ func writeContracts(contracts []api.RenterContract) {
w.Flush()
}
// rentercontractscmd is the handler for the comand `siac renter contracts`.
// rentercontractscmd is the handler for the command `siac renter contracts`.
// It lists the Renter's contracts.
func rentercontractscmd() {
rc, err := httpClient.RenterDisabledContractsGet()
......@@ -1779,8 +1807,9 @@ func renterfilesdeletecmd(cmd *cobra.Command, paths []string) {
return
}
// renterfilesdownload is the handler for the comand `siac renter download [path] [destination]`.
// It determines whether a file or a folder is downloaded and calls the corresponding sub-handler.
// renterfilesdownload is the handler for the command `siac renter download
// [path] [destination]`. It determines whether a file or a folder is downloaded
// and calls the corresponding sub-handler.
func renterfilesdownloadcmd(path, destination string) {
// Parse SiaPath.
siaPath, err := modules.NewSiaPath(path)
......@@ -2017,7 +2046,7 @@ func downloadprogress(tfs []trackedFile) []api.DownloadInfo {
progressStr += fmt.Sprint(progressLine)
}
}
print(progressStr)
fmt.Print(progressStr)
progressStr = clearStr
}
// This code is unreachable, but the compiler requires this to be here.
......@@ -2554,7 +2583,7 @@ func renterratelimitcmd(downloadSpeedStr, uploadSpeedStr string) {
fmt.Println("Set renter maxdownloadspeed to ", downloadSpeedInt, " and maxuploadspeed to ", uploadSpeedInt)
}
// renterworkerscmd is the handler for the comand `siac renter workers`.
// renterworkerscmd is the handler for the command `siac renter workers`.
// It lists the Renter's workers.
func renterworkerscmd() {
rw, err := httpClient.RenterWorkersGet()
......@@ -2597,6 +2626,213 @@ func renterworkerscmd() {
}
}
// renterworkerseacmd is the handler for the command `siac renter workers ea`.
// It lists the status of the account of every worker.
func renterworkerseacmd() {
rw, err := httpClient.RenterWorkersGet()
if err != nil {
die("Could not get worker statuses:", err)
}
// collect some overal account stats
var wocd, nfw uint64
for _, worker := range rw.Workers {
if worker.AccountStatus.OnCoolDown {
wocd++
}
if !worker.AccountStatus.Funded {
nfw++
}
}
fmt.Println("Worker Accounts Summary")
w := tabwriter.NewWriter(os.Stdout, 2, 0, 2, ' ', 0)
defer func() {
err := w.Flush()
if err != nil {
die("Could not flush tabwriter:", err)
}
}()
// print summary
fmt.Fprintf(w, "Total Workers: \t%v\n", rw.NumWorkers)
fmt.Fprintf(w, "Workers On Cooldown: \t%v\n", wocd)
fmt.Fprintf(w, "Non Funded Workers: \t%v\n", nfw)
// print header
hostInfo := "Host PubKey"
accountInfo := "\tFunded\tAvailBal\tNegBal\tBalTarget"
queueInfo := "\tOnCoolDown\tCoolDownUntil\tConsecFail\tErrorAt\tError"
header := hostInfo + accountInfo + queueInfo
fmt.Fprintln(w, "\nWorker Accounts Detail \n\n"+header)
// print rows
for _, worker := range rw.Workers {
as := worker.AccountStatus
// Host Info
fmt.Fprintf(w, "%v", worker.HostPubKey.String())
// Account Info
fmt.Fprintf(w, "\t%t\t%s\t%s\t%s",
as.Funded,
as.AvailableBalance.HumanString(),
as.NegativeBalance.HumanString(),
worker.AccountBalanceTarget.HumanString())
// Queue Info
fmt.Fprintf(w, "\t%t\t%v\t%v\t%v\t%v\n",
as.OnCoolDown,
sanitizeTime(as.OnCoolDownUntil, as.OnCoolDown),
as.ConsecutiveFailures,
sanitizeTime(as.RecentErrTime, as.RecentErr != ""),
sanitizeErr(as.RecentErr))
}
}
// renterworkersptcmd is the handler for the command `siac renter workers pt`.
// It lists the status of the price table of every worker.
func renterworkersptcmd() {
rw, err := httpClient.RenterWorkersGet()
if err != nil {
die("Could not get worker statuses:", err)
}
// collect some overal account stats
var wocd, wnpt uint64
for _, worker := range rw.Workers {
if worker.PriceTableStatus.OnCoolDown {
wocd++
}
if !worker.PriceTableStatus.Active {
wnpt++
}
}
fmt.Println("Worker Price Tables Summary")
w := tabwriter.NewWriter(os.Stdout, 2, 0, 2, ' ', 0)
defer func() {
err := w.Flush()
if err != nil {
die("Could not flush tabwriter:", err)
}
}()
// print summary
fmt.Fprintf(w, "Total Workers: \t%v\n", rw.NumWorkers)
fmt.Fprintf(w, "Workers On Cooldown: \t%v\n", wocd)
fmt.Fprintf(w, "Workers Without Price Table: \t%v\n", wnpt)
// print header
hostInfo := "Host PubKey"
priceTableInfo := "\tActive\tExpiry\tUpdate"
queueInfo := "\tOnCoolDown\tCoolDownUntil\tConsecFail\tErrorAt\tError"
header := hostInfo + priceTableInfo + queueInfo
fmt.Fprintln(w, "\nWorker Price Tables Detail \n\n"+header)
// print rows
for _, worker := range rw.Workers {
pts := worker.PriceTableStatus
// Host Info
fmt.Fprintf(w, "%v", worker.HostPubKey.String())
// Price Table Info
fmt.Fprintf(w, "\t%t\t%s\t%s",
pts.Active,
sanitizeTime(pts.ExpiryTime, pts.Active),
sanitizeTime(pts.UpdateTime, pts.Active))
// QueueInfo
fmt.Fprintf(w, "\t%t\t%v\t%v\t%v\t%v\n",
pts.OnCoolDown,
sanitizeTime(pts.OnCoolDownUntil, pts.OnCoolDown),
pts.ConsecutiveFailures,
sanitizeTime(pts.RecentErrTime, pts.RecentErr != ""),
sanitizeErr(pts.RecentErr))
}
}
// renterworkersrjcmd is the handler for the command `siac renter workers rj`.
// It lists the status of the read job queue for every worker.
func renterworkersrjcmd() {
rw, err := httpClient.RenterWorkersGet()
if err != nil {
die("Could not get worker statuses:", err)
}
w := tabwriter.NewWriter(os.Stdout, 2, 0, 2, ' ', 0)
defer func() {
err := w.Flush()
if err != nil {
die("Could not flush tabwriter:", err)
}
}()
// print header
hostInfo := "Host PubKey"
queueInfo := "\tJobs\tAvgJobTime64k (ms)\tAvgJobTime1m (ms)\tAvgJobTime4m (ms)\tConsecFail\tErrorAt\tError"
header := hostInfo + queueInfo
fmt.Fprintln(w, "\nWorker Read Jobs \n\n"+header)
// print rows
for _, worker := range rw.Workers {
rjs := worker.ReadJobsStatus
// Host Info
fmt.Fprintf(w, "%v", worker.HostPubKey.String())
// ReadJobs Info
fmt.Fprintf(w, "\t%v\t%v\t%v\t%v\t%v\t%v\t%v\n",
rjs.JobQueueSize,
rjs.AvgJobTime64k,
rjs.AvgJobTime1m,
rjs.AvgJobTime4m,
rjs.ConsecutiveFailures,
sanitizeTime(rjs.RecentErrTime, rjs.RecentErr != ""),
sanitizeErr(rjs.RecentErr))
}
}
// renterworkershsjcmd is the handler for the command `siac renter workers hs`.
// It lists the status of the has sector job queue for every worker.
func renterworkershsjcmd() {
rw, err := httpClient.RenterWorkersGet()
if err != nil {
die("Could not get worker statuses:", err)
}
w := tabwriter.NewWriter(os.Stdout, 2, 0, 2, ' ', 0)
defer func() {
err := w.Flush()
if err != nil {
die("Could not flush tabwriter:", err)
}
}()
// print header
hostInfo := "Host PubKey"
queueInfo := "\tJobs\tAvgJobTime (ms)\tConsecFail\tErrorAt\tError"
header := hostInfo + queueInfo
fmt.Fprintln(w, "\nWorker Has Sector Jobs \n\n"+header)
// print rows
for _, worker := range rw.Workers {
hsjs := worker.HasSectorJobsStatus
// Host Info
fmt.Fprintf(w, "%v", worker.HostPubKey.String())
// HasSector Jobs Info
fmt.Fprintf(w, "\t%v\t%v\t%v\t%v\t%v\n",
hsjs.JobQueueSize,
hsjs.AvgJobTime,
hsjs.ConsecutiveFailures,
sanitizeTime(hsjs.RecentErrTime, hsjs.RecentErr != ""),
sanitizeErr(hsjs.RecentErr))
}
}
// writeWorkers is a helper function to display workers
func writeWorkers(workers []modules.WorkerStatus) {
fmt.Println(" Number of Workers:", len(workers))
......@@ -2636,8 +2872,8 @@ func writeWorkers(workers []modules.WorkerStatus) {
// EA Info
fmt.Fprintf(w, "\t%v\t%v",
worker.AvailableBalance,
worker.BalanceTarget)
worker.AccountStatus.AvailableBalance,
worker.AccountBalanceTarget)
// Job Info
fmt.Fprintf(w, "\t%v\t%v\n",
......@@ -2646,3 +2882,23 @@ func writeWorkers(workers []modules.WorkerStatus) {
}
w.Flush()
}
// sanitizeTime is a small helper function that sanitizes the output for the
// given time. If the given 'cond' value is false, it will print "-", if it is
// true it will print the time in a predefined format.
func sanitizeTime(t time.Time, cond bool) string {
if !cond {
return "-"
}
return fmt.Sprintf("%v", t.Format(time.RFC3339))
}
// sanitizeErr is a small helper function that sanitizes the output for the
// given error string. It will print "-", if the error string is the equivalent
// of a nil error.
func sanitizeErr(errStr string) string {
if errStr == "" {
return "-"