Use Clap instead of Arg for command-line arguments
Description / Motivation
This MR replaces Arg
with Clap (https://github.com/rbardou/clap/).
- It results in a much more readable
--help
. - The
Cli
module changes a lot, including its interface. We could try to stay backward-compatible but I don't think users use the parts ofCli
that changed. - I had to move test filters from
Test.register
toTest.run
, which is something that was already kind of the case but only for some options. Now it's all done at the same place.
How to Test the MR Manually
opam install clap.0.3.0
dune runtest
dune exec test/unix/main.exe -- --help
--help
The New SYNOPSIS
main.exe [--keep-temp|--delete-temp|--delete-temp-if-success]
[--keep-going|-k] [--global-timeout <SECONDS>]
[--test-timeout <SECONDS>] [--retry <COUNT>] [--reset-regressions]
[--on-unknown-regression-files <MODE>] [--loop] [--loop-count <COUNT>]
[--resume-file <FILE>] [--resume|-r] [(--job-count|-j) <COUNT>]
[(--test-arg|-a) <<PARAMETER>=<VALUE>>...] [--seed <SEED>]
[--color|--no-color] [--log-level <LEVEL>]
[--verbose|-v|--quiet|-q|--info|-i] [--log-file <FILE>]
[--log-buffer-size <COUNT>] [--log-worker-id] [--commands|-c] [--time]
[--record <FILE>] [--from-record <FILE>...] [--junit <FILE>]
[--list|-l|--list-tsv|--suggest-jobs] [(--file|-f) <FILE>...]
[--not-file <FILE>...] [(--match|-m) <PERL_REGEXP>...]
[(--not-match|-m) <PERL_REGEXP>...] [(--title|--test|-t) <VALUE>...]
[(--not-title|--not-test) <VALUE>...] [--job <<INDEX>/<COUNT>>]
[--skip <COUNT>] [--only <COUNT>] [<TAG>...]
DESCRIPTION
Run or manage test suite.
OPTIONS
--global-timeout SECONDS
Fail if the set of tests takes more than SECONDS to run.
--job-count COUNT, -j COUNT (default: 1)
Run COUNT tests in parallel, in separate processes. With --suggest-jobs,
set the number of target jobs for --suggest-jobs instead.
--keep-going, -k
If a test fails, continue with the remaining tests instead of stopping.
Aborting manually with Ctrl+C still stops everything.
--keep-temp, --delete-temp, --delete-temp-if-success
(default: --delete-temp)
Whether to delete temporary files and directories that were created.
--keep-temp: Keep temporary files and directories after tests.
--delete-temp: Delete temporary files and directories after tests.
--delete-temp-if-success: Delete temporary files and directories, except
if the test failed. If a test succeeds after it however, it causes all
temporary files to be deleted, even those of the failed tests; so you
should probably avoid running with --keep-temp and --job-count.
--loop
Restart from the beginning once all tests are done. All tests are
repeated until one of them fails or if you interrupt with Ctrl+C. This
is useful to reproduce non-deterministic failures. When used in
conjunction with --keep-going, tests are repeated even if they fail,
until you interrupt them with Ctrl+C.
--loop-count COUNT
Stop after all tests have been run COUNT times. Implies --loop. A value
of 0 means tests are not run.
--on-unknown-regression-files MODE (default: warn)
How to handle regression test outputs that are not declared by any test.
MODE can be:
- warn: emit a warning for unknown output files;
- ignore: ignore unknown output files;
- fail: terminate execution with exit code 1 and without running any
further action when unknown output files are found;
- delete: delete unknown output files.
To check which files would be deleted, run with this option set to
'warn', which is the default.
--reset-regressions
Remove regression test outputs if they exist, and regenerate them.
--resume, -r
Resume from a previous run. This reads the resume file located at
--resume-file to resume from it.
If --resume-file is not specified, --resume implies --resume-file
tezt-resume.json. If the resume file does not exist, act as if it was
empty.
Before running a test, it is checked whether this test was already
successfully ran according to the resume file. If it was, the test is
skipped.
When using --loop or --loop-count, the test is skipped as many times as
it was successful according to the resume file.
--resume-file FILE
Record test results to FILE for use with --resume. When using --resume,
test results that existed in FILE are kept, contrary to --record.
--retry COUNT (default: 0)
Retry each failing test up to COUNT times. If one retry is successful,
the test is considered successful.
--seed SEED
Force tests declared with ~seed: Random to initialize the pseudo-random
number generator with this seed.
--test-arg <PARAMETER>=<VALUE>, -a <PARAMETER>=<VALUE>
Pass a generic argument to tests. Tests can get this argument with
Cli.get. --test-arg <PARAMETER> is a short-hand for: --test-arg
<PARAMETER>=true
--test-timeout SECONDS
Fail if a test takes, on its own, more than SECONDS to run.
LOGS
--color, --no-color (default: --no-color)
Whether to use colors in output. Default value depends on whether stdout
is a terminal and on the value of the TERM environment variable.
--commands, -c
Output commands which are run, in a way that is easily copy-pasted for
manual reproductibility.
--log-buffer-size COUNT (default: 50)
Before logging an error on stdout, also log the last COUNT messages that
have been ignored because of the log level since the last message that
was not ignored.
--log-file FILE
Also log to FILE (in verbose mode: --log-level only applies to stdout).
In the presence of --job-count, the main process will log test results
to FILE while each worker writes test logs to a separate file
BASENAME-WORKER_ID[.EXT]. BASENAME is the basename of FILE, WORKER_ID is
the zero-indexed id of the worker and .EXT is the extension of FILE if
present.
--log-level LEVEL (default: report)
Set log level. Possible LEVELs are: quiet, error, warn, report, info,
debug.
--log-worker-id
Decorate logs with worker IDs when --job-count is more than 1.
--verbose, -v, --quiet, -q, --info, -i
Set log level. Overrides --log-level.
--verbose, -v: Same as --log-level debug.
--quiet, -q: Same as --log-level quiet.
--info, -i: Same as --log-level info.
REPORTS
--from-record FILE
Start from a file recorded with --record. If specified multiple times,
start from the union of those records.
If <FILE> is a directory, this is equivalent to specifying --from-record
for all files in this directory that have the .json extension.
When using --time, test durations include tests found in record files.
When using --record, the new record which is output does NOT include the
input records.
When using --junit, reports do NOT include input records.
--junit FILE
Store test results in FILE using JUnit XML format. Time information for
each test is the sum of all runs of this test for the current session.
Test result (success or failure) is the result for the last run of the
test.
--record FILE
Record test results to FILE. This file can then be used with
--from-record. If you use --loop or --loop-count, times are averaged for
each test.
--time
Print a summary of the total time taken by each test. Ignored if a test
failed. Includes the time read from records: to display a record, you
can use --time --loop-count 0 --from-record <FILE>.
COMMANDS
--list, -l, --list-tsv, --suggest-jobs
Do not run test. Instead:
--list, -l: List tests.
Pass --time to also display results and timings (in seconds) from a
previous execution given through --record, in the format TIME (COUNT).
TIME is the average time of successful executions. COUNT is
SCOUNT/(SCOUNT+FCOUNT) where SCOUNT (resp. FCOUNT) is the number of
successful (resp. failed) tests in the record. If there is only one
successful test, then (COUNT) is omitted. Tests lacking a past record of
successful executions are noted '-'. A final row is added containing the
total of the averages of successful test executions, and the total
number of selected tests.
--list-tsv: List tests as tab-separated values in the format FILE TITLE
TAGS.
Pass --time to also display results and timings (in nanoseconds) from a
previous execution given through --record. Then each line is appended
with STIME SCOUNT FTIME FCOUNT. STIME (resp. FTIME) is the total running
time in nanoseconds of successful (resp. failed) previous runs. SCOUNT
(resp. FCOUNT) is the count of successful (resp. failed) previous runs.
--suggest-jobs: Read test results records specified with --from-records
and suggest a partition of the tests that would result in --job-count
sets of roughly the same total duration. Output each job as a list of
flags that can be passed to Tezt, followed by a shell comment that
denotes the expected duration of the job.
A similar result can be obtained with --list --job, except that the last
job suggested by --suggest-jobs uses --not-test to express "all tests
that are not already in other jobs", meaning that the last job acts as a
catch-all for unknown tests.
SELECTING TESTS
You can specify multiple tags, negated tags, titles, title patterns and
filenames on the command line. Only tests which match all the following
conditions will be run:
- the test must have all tags and none of the negated tags;
- the test must have one of the specified titles;
- the test must have a title matching one of the specified patterns;
- the test must be implemented in one of the specified files.
The tags of a test are given by the ~tags argument of Test.register. To
negate a tag, prefix it with a slash: /
The title of a test is given by the ~title argument of Test.register. It is
what is printed after [SUCCESS] (or [FAILURE] or [ABORTED]) in the reports.
Use --title (respectively --not-title) to select (respectively unselect) a
test by its title on the command-line. You can also select (respectively
unselect) tests for which 'filename: title' matches one or several Perl
regular expressions using --match (respectively --not-match).
The file in which a test is implemented is specified by the ~__FILE__
argument of Test.register. In other words, it is the path of the file in
which the test is defined. Use --file (respectively --not-file) to select
(respectively unselect) a test by its path (or a suffix thereof) on the
command-line.
For instance:
_build/default/test/unix/main.exe node bake /rpc --file bootstrap.ml --file
sync.ml
will run all tests defined in either bootstrap.ml or sync.ml, which have at
least tags 'node' and 'bake', but which do not have the 'rpc' tag.
--file FILE, -f FILE
Only run tests implemented in source files ending with FILE.
--job <INDEX>/<COUNT>
Split the set of selected tests into COUNT subsets of roughly the same
total duration. Execute only one of these subsets, specified by INDEX.
COUNT must be at least 1 and INDEX must be between 1 and COUNT.
Use --from-record to feed duration data from past runs. Tests for which
no time data is available are given a default duration of 1 second.
You can use --list to see what tests are in a subset without actually
running the tests.
A typical use is to run tests in parallel on different machines. For
instance, have one machine run with --job 1/3, one with --job 2/3 and
one with --job 3/3. Be sure to provide exactly the same records with
--from-record, in the same order, and to select exactly the same set of
tests (same tags, same --file and same --test) for all machines,
otherwise some tests may not be run at all.
--match PERL_REGEXP, -m PERL_REGEXP
Only run tests for which 'FILE: TITLE' matches PERL_REGEXP (case
insensitive), where FILE is the source file of the test and TITLE its
title.
--not-file FILE
Only run tests not implemented in source files ending with FILE.
--not-match PERL_REGEXP, -m PERL_REGEXP
Only run tests for which 'FILE: TITLE' does not match PERL_REGEXP (case
insensitive), where FILE is the source file of the test and TITLE its
title.
--not-title VALUE, --not-test VALUE
Only run tests which are not exactly entitled TITLE.
--only COUNT
Only run the first COUNT tests. This filter is applied after --job and
--skip.
--skip COUNT (default: 0)
Skip the first COUNT tests. This filter is applied after --job and
before --only.
--title VALUE, --test VALUE, -t VALUE
Only run tests which are exactly entitled TITLE.
TAG
If TAG does not start with '/', only run tests tagged with TAG. If TAG
is of the form '/X', only run tests that are not tagged with X.
Checklist
-
Update CHANGES.md
. No need to document changes to documentation and tests. -
Make sure all new values, types etc. are documented in .mli
files and that the generated documentation looks ok. -
Add tests in test/
, if relevant.
Edited by Romain