Skip to content

Use Clap instead of Arg for command-line arguments

Romain requested to merge clap into master

Description / Motivation

This MR replaces Arg with Clap (https://github.com/rbardou/clap/).

  • It results in a much more readable --help.
  • The Cli module changes a lot, including its interface. We could try to stay backward-compatible but I don't think users use the parts of Cli that changed.
  • I had to move test filters from Test.register to Test.run, which is something that was already kind of the case but only for some options. Now it's all done at the same place.

How to Test the MR Manually

opam install clap.0.3.0
dune runtest
dune exec test/unix/main.exe -- --help

The New --help

SYNOPSIS

    main.exe [--keep-temp|--delete-temp|--delete-temp-if-success]
        [--keep-going|-k] [--global-timeout <SECONDS>]
        [--test-timeout <SECONDS>] [--retry <COUNT>] [--reset-regressions]
        [--on-unknown-regression-files <MODE>] [--loop] [--loop-count <COUNT>]
        [--resume-file <FILE>] [--resume|-r] [(--job-count|-j) <COUNT>]
        [(--test-arg|-a) <<PARAMETER>=<VALUE>>...] [--seed <SEED>]
        [--color|--no-color] [--log-level <LEVEL>]
        [--verbose|-v|--quiet|-q|--info|-i] [--log-file <FILE>]
        [--log-buffer-size <COUNT>] [--log-worker-id] [--commands|-c] [--time]
        [--record <FILE>] [--from-record <FILE>...] [--junit <FILE>]
        [--list|-l|--list-tsv|--suggest-jobs] [(--file|-f) <FILE>...]
        [--not-file <FILE>...] [(--match|-m) <PERL_REGEXP>...]
        [(--not-match|-m) <PERL_REGEXP>...] [(--title|--test|-t) <VALUE>...]
        [(--not-title|--not-test) <VALUE>...] [--job <<INDEX>/<COUNT>>]
        [--skip <COUNT>] [--only <COUNT>] [<TAG>...]

DESCRIPTION

    Run or manage test suite.

OPTIONS

    --global-timeout SECONDS
        Fail if the set of tests takes more than SECONDS to run.

    --job-count COUNT, -j COUNT (default: 1)
        Run COUNT tests in parallel, in separate processes. With --suggest-jobs,
        set the number of target jobs for --suggest-jobs instead.

    --keep-going, -k
        If a test fails, continue with the remaining tests instead of stopping.
        Aborting manually with Ctrl+C still stops everything.

    --keep-temp, --delete-temp, --delete-temp-if-success
    (default: --delete-temp)
        Whether to delete temporary files and directories that were created.

        --keep-temp: Keep temporary files and directories after tests.

        --delete-temp: Delete temporary files and directories after tests.

        --delete-temp-if-success: Delete temporary files and directories, except
        if the test failed. If a test succeeds after it however, it causes all
        temporary files to be deleted, even those of the failed tests; so you
        should probably avoid running with --keep-temp and --job-count.

    --loop
        Restart from the beginning once all tests are done. All tests are
        repeated until one of them fails or if you interrupt with Ctrl+C. This
        is useful to reproduce non-deterministic failures. When used in
        conjunction with --keep-going, tests are repeated even if they fail,
        until you interrupt them with Ctrl+C.

    --loop-count COUNT
        Stop after all tests have been run COUNT times. Implies --loop. A value
        of 0 means tests are not run.

    --on-unknown-regression-files MODE (default: warn)
        How to handle regression test outputs that are not declared by any test.
        MODE can be:
        - warn: emit a warning for unknown output files;
        - ignore: ignore unknown output files;
        - fail: terminate execution with exit code 1 and without running any
        further action when unknown output files are found;
        - delete: delete unknown output files.

        To check which files would be deleted, run with this option set to
        'warn', which is the default.

    --reset-regressions
        Remove regression test outputs if they exist, and regenerate them.

    --resume, -r
        Resume from a previous run. This reads the resume file located at
        --resume-file to resume from it.

        If --resume-file is not specified, --resume implies --resume-file
        tezt-resume.json. If the resume file does not exist, act as if it was
        empty.

        Before running a test, it is checked whether this test was already
        successfully ran according to the resume file. If it was, the test is
        skipped.

        When using --loop or --loop-count, the test is skipped as many times as
        it was successful according to the resume file.

    --resume-file FILE
        Record test results to FILE for use with --resume. When using --resume,
        test results that existed in FILE are kept, contrary to --record.

    --retry COUNT (default: 0)
        Retry each failing test up to COUNT times. If one retry is successful,
        the test is considered successful.

    --seed SEED
        Force tests declared with ~seed: Random to initialize the pseudo-random
        number generator with this seed.

    --test-arg <PARAMETER>=<VALUE>, -a <PARAMETER>=<VALUE>
        Pass a generic argument to tests. Tests can get this argument with
        Cli.get. --test-arg <PARAMETER> is a short-hand for: --test-arg
        <PARAMETER>=true

    --test-timeout SECONDS
        Fail if a test takes, on its own, more than SECONDS to run.

LOGS

    --color, --no-color (default: --no-color)
        Whether to use colors in output. Default value depends on whether stdout
        is a terminal and on the value of the TERM environment variable.

    --commands, -c
        Output commands which are run, in a way that is easily copy-pasted for
        manual reproductibility.

    --log-buffer-size COUNT (default: 50)
        Before logging an error on stdout, also log the last COUNT messages that
        have been ignored because of the log level since the last message that
        was not ignored.

    --log-file FILE
        Also log to FILE (in verbose mode: --log-level only applies to stdout).
        In the presence of --job-count, the main process will log test results
        to FILE while each worker writes test logs to a separate file
        BASENAME-WORKER_ID[.EXT]. BASENAME is the basename of FILE, WORKER_ID is
        the zero-indexed id of the worker and .EXT is the extension of FILE if
        present.

    --log-level LEVEL (default: report)
        Set log level. Possible LEVELs are: quiet, error, warn, report, info,
        debug.

    --log-worker-id
        Decorate logs with worker IDs when --job-count is more than 1.

    --verbose, -v, --quiet, -q, --info, -i
        Set log level. Overrides --log-level.

        --verbose, -v: Same as --log-level debug.
        --quiet, -q: Same as --log-level quiet.
        --info, -i: Same as --log-level info.

REPORTS

    --from-record FILE
        Start from a file recorded with --record. If specified multiple times,
        start from the union of those records.

        If <FILE> is a directory, this is equivalent to specifying --from-record
        for all files in this directory that have the .json extension.

        When using --time, test durations include tests found in record files.

        When using --record, the new record which is output does NOT include the
        input records.

        When using --junit, reports do NOT include input records.

    --junit FILE
        Store test results in FILE using JUnit XML format. Time information for
        each test is the sum of all runs of this test for the current session.
        Test result (success or failure) is the result for the last run of the
        test.

    --record FILE
        Record test results to FILE. This file can then be used with
        --from-record. If you use --loop or --loop-count, times are averaged for
        each test.

    --time
        Print a summary of the total time taken by each test. Ignored if a test
        failed. Includes the time read from records: to display a record, you
        can use --time --loop-count 0 --from-record <FILE>.

COMMANDS

    --list, -l, --list-tsv, --suggest-jobs
        Do not run test. Instead:

        --list, -l: List tests.

        Pass --time to also display results and timings (in seconds) from a
        previous execution given through --record, in the format TIME (COUNT).
        TIME is the average time of successful executions. COUNT is
        SCOUNT/(SCOUNT+FCOUNT) where SCOUNT (resp. FCOUNT) is the number of
        successful (resp. failed) tests in the record. If there is only one
        successful test, then (COUNT) is omitted. Tests lacking a past record of
        successful executions are noted '-'. A final row is added containing the
        total of the averages of successful test executions, and the total
        number of selected tests.

        --list-tsv: List tests as tab-separated values in the format FILE TITLE
        TAGS.

        Pass --time to also display results and timings (in nanoseconds) from a
        previous execution given through --record. Then each line is appended
        with STIME SCOUNT FTIME FCOUNT. STIME (resp. FTIME) is the total running
        time in nanoseconds of successful (resp. failed) previous runs. SCOUNT
        (resp. FCOUNT) is the count of successful (resp. failed) previous runs.

        --suggest-jobs: Read test results records specified with --from-records
        and suggest a partition of the tests that would result in --job-count
        sets of roughly the same total duration. Output each job as a list of
        flags that can be passed to Tezt, followed by a shell comment that
        denotes the expected duration of the job.

        A similar result can be obtained with --list --job, except that the last
        job suggested by --suggest-jobs uses --not-test to express "all tests
        that are not already in other jobs", meaning that the last job acts as a
        catch-all for unknown tests.

SELECTING TESTS

    You can specify multiple tags, negated tags, titles, title patterns and
    filenames on the command line. Only tests which match all the following
    conditions will be run:
    - the test must have all tags and none of the negated tags;
    - the test must have one of the specified titles;
    - the test must have a title matching one of the specified patterns;
    - the test must be implemented in one of the specified files.

    The tags of a test are given by the ~tags argument of Test.register. To
    negate a tag, prefix it with a slash: /

    The title of a test is given by the ~title argument of Test.register. It is
    what is printed after [SUCCESS] (or [FAILURE] or [ABORTED]) in the reports.
    Use --title (respectively --not-title) to select (respectively unselect) a
    test by its title on the command-line. You can also select (respectively
    unselect) tests for which 'filename: title' matches one or several Perl
    regular expressions using --match (respectively --not-match).

    The file in which a test is implemented is specified by the ~__FILE__
    argument of Test.register. In other words, it is the path of the file in
    which the test is defined. Use --file (respectively --not-file) to select
    (respectively unselect) a test by its path (or a suffix thereof) on the
    command-line.

    For instance:

    _build/default/test/unix/main.exe node bake /rpc --file bootstrap.ml --file
    sync.ml

    will run all tests defined in either bootstrap.ml or sync.ml, which have at
    least tags 'node' and 'bake', but which do not have the 'rpc' tag.

    --file FILE, -f FILE
        Only run tests implemented in source files ending with FILE.

    --job <INDEX>/<COUNT>
        Split the set of selected tests into COUNT subsets of roughly the same
        total duration. Execute only one of these subsets, specified by INDEX.

        COUNT must be at least 1 and INDEX must be between 1 and COUNT.

        Use --from-record to feed duration data from past runs. Tests for which
        no time data is available are given a default duration of 1 second.

        You can use --list to see what tests are in a subset without actually
        running the tests.

        A typical use is to run tests in parallel on different machines. For
        instance, have one machine run with --job 1/3, one with --job 2/3 and
        one with --job 3/3. Be sure to provide exactly the same records with
        --from-record, in the same order, and to select exactly the same set of
        tests (same tags, same --file and same --test) for all machines,
        otherwise some tests may not be run at all.

    --match PERL_REGEXP, -m PERL_REGEXP
        Only run tests for which 'FILE: TITLE' matches PERL_REGEXP (case
        insensitive), where FILE is the source file of the test and TITLE its
        title.

    --not-file FILE
        Only run tests not implemented in source files ending with FILE.

    --not-match PERL_REGEXP, -m PERL_REGEXP
        Only run tests for which 'FILE: TITLE' does not match PERL_REGEXP (case
        insensitive), where FILE is the source file of the test and TITLE its
        title.

    --not-title VALUE, --not-test VALUE
        Only run tests which are not exactly entitled TITLE.

    --only COUNT
        Only run the first COUNT tests. This filter is applied after --job and
        --skip.

    --skip COUNT (default: 0)
        Skip the first COUNT tests. This filter is applied after --job and
        before --only.

    --title VALUE, --test VALUE, -t VALUE
        Only run tests which are exactly entitled TITLE.

    TAG
        If TAG does not start with '/', only run tests tagged with TAG. If TAG
        is of the form '/X', only run tests that are not tagged with X.

Checklist

  • Update CHANGES.md. No need to document changes to documentation and tests.
  • Make sure all new values, types etc. are documented in .mli files and that the generated documentation looks ok.
  • Add tests in test/, if relevant.
Edited by Romain

Merge request reports