Commit 350172f6 authored by Philippe B.'s avatar Philippe B. 🏂 Committed by Clément Hurlin
Browse files

Tests/python: delete multibranch tests

parent 25c8b965
......@@ -191,7 +191,7 @@ Tests can be classified with tags. Tags are added with the annotation
@pytest.mark.TAG
The configuration file ``pytest.ini`` defines the list of allowed tags.
It includes ``vote``, ``multinode``, ``baker``, ``endorser``, ``contract``, ``slow``, ``multibranch``.
It includes ``vote``, ``multinode``, ``baker``, ``endorser``, ``contract``, ``slow``.
Examples
""""""""
......@@ -384,131 +384,6 @@ To reduce coupling between tests and the actual branch to be tested, tests
refer to protocol Alpha using ``constants.ALPHA`` and
``constants.ALPHA_DAEMON`` rather than by hard-coded identifiers.
Tests based on fixed revisions (multibranch)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is useful to test interactions between different server versions. There
are currently two ways of doing this.
1. The ``Sandbox`` launcher can use binaries built from different revisions.
Methods ``add_node``, ``add_baker`` and ``add_endorser`` have an optional
parameter ``branch`` that points to a subdirectory where binaries are to be
looked for.
2. The ``SandboxMultibranch`` launcher is instantiated by map from ids to
branches. Then every time we launch a node or a daemon the actual binary will
be selected according to the map.
Tests using specific revisions are in ``tests/multibranch`` and aren't run by
default. They are not regression tests and are usually launched separately
from the rest of the tests. To run these tests, you need to set up the
``TEZOS_BINARIES`` environment variable to a directory that contains the
binaries for all revisions needed by test (see below). The tests will be
skipped if this variable isn't set, and fail if the binaries aren't
available.
Building binaries for several revisions
"""""""""""""""""""""""""""""""""""""""
Before running the tests, the user has to build the binaries and copy them to
the right location. This can be done by the ``scripts/build_branches.py``
script.
For instance, suppose we want to build binaries for two different revisions
of zeronet:
::
A = b8de4297db6a681eb13343d2773c6840969a5537
B = 816625bed0983f7201e4c369440a910f006beb1a
TEZOS_HOME=~/tezos # TEZOS repo, read-only access from the script
TEZOS_BINARIES=~/tezos-binaries # where the binaries will be stored
TEZOS_BUILD=~/tmp/tezos_tmp # where the binaries will be built
The following command will generate binaries for each of the specified
branches in ``TEZOS_BINARIES``.
::
scripts/build_branches.py --clone $TEZOS_HOME --build-dir $TEZOS_BUILD \
--bin-dir $TEZOS_BINARIES \
b8de4297db6a681eb13343d2773c6840969a5537 \
816625bed0983f7201e4c369440a910f006beb1a
> ls $TEZOS_BINARIES *
816625bed0983f7201e4c369440a910f006beb1a:
tezos-accuser-003-PsddFKi3 tezos-baker-004-Pt24m4xi tezos-node
tezos-accuser-004-Pt24m4xi tezos-client tezos-protocol-compiler
tezos-admin-client tezos-endorser-003-PsddFKi3 tezos-signer
tezos-baker-003-PsddFKi3 tezos-endorser-004-Pt24m4xi
b8de4297db6a681eb13343d2773c6840969a5537:
tezos-accuser-003-PsddFKi3 tezos-baker-004-Pt24m4xi tezos-node
tezos-accuser-004-Pt24m4xi tezos-client tezos-protocol-compiler
tezos-admin-client tezos-endorser-003-PsddFKi3 tezos-signer
tezos-baker-003-PsddFKi3 tezos-endorser-004-Pt24m4xi
Note: One can specify a branch instead of a revision but this is error-prone.
For instance, protocols may have different hashes on different revisions
on the same branch, and these hashes are typically hard-coded in the tests to
activate the protocols.
Example 1: ``test_baker_endorser_mb.py``
""""""""""""""""""""""""""""""""""""""""
The test ``test_baker_endorser_mb.py`` uses two different revisions.
the ``sandbox_multibranch`` fixtures (which uses the ``SandboxMultibranch``
launcher) parameterized by a map that alternates between the two revisions.
The executables will be selected from revisions A and B as specified by:
::
A = "d272059bf474018d0c39f5a6e60634a95f0c44aa" # MAINNET
B = "6718e80254d4cb8d7ad86bce8cf3cb692550c6e7" # MAINNET SNAPSHOT
MAP = {i:A if i % 2 == 0 else B for i in range(20)}
@pytest.mark.parametrize('sandbox_multibranch', [MAP], indirect=True)
Run the test with
::
# mkdir tmp
poetry run pytest tests/multibranch/test_baker_endorser_mb.py --log-dir=tmp
Example 2: A full voting scenario ``test_voting_full.py``
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
This tests uses binaries from revision
``b8de4297db6a681eb13343d2773c6840969a5537`` and implements a full voting
scenario (voting, launching a test chain and a test chain baker, upgrading to
a new protocol, performing operations on the new protocol). It uses two
protocols implemented by this specific revision,
::
ALPHA = 'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP'
NEW_PROTO = 'Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'
as well the corresponding bakers ``tezos-baker-003-PsddFKi3`` ``tezos-baker-004-Pt24m4xi``.
::
scripts/build_branches.py --clone $TEZOS_HOME --build-dir $TEZOS_BUILD \
--bin-dir $TEZOS_BINARIES \ b8de4297db6a681eb13343d2773c6840969a5537
It can be run with
::
poetry run pytest tests/multibranch/test_baker_endorser_mb.py`
Note: this test uses only one revision but it can't run
on branch ``master`` as we need an extra protocol with bakers.
.. _pytest_regression_testing:
Regression testing
......
......@@ -33,12 +33,6 @@ EOF
done | grep ^tests_.*\.py | uniq | LC_COLLATE=C sort)
for test in $slow_tests; do
case "$test" in
*/multibranch/*)
# skip multibranch tests for now
;;
*)
testname=${test##*/test_}
testname=${testname%%.py}
......@@ -50,7 +44,6 @@ integration:${PROTO_DIR_BASE}_${testname}:
stage: test
EOF
esac
done
done
......
......@@ -22,7 +22,7 @@ PYCODESTYLE_OPTIONS=\
# black is configured in pyproject.toml
BLACK=poetry run black
SRCS?=process daemons launchers client tools scripts tests_* tests_alpha/multibranch examples codec conftest.py
SRCS?=process daemons launchers client tools scripts tests_* examples codec conftest.py
LOG_DIR=tmp
......
......@@ -50,10 +50,8 @@ sed -i '1 s@^tests_alpha/test_@tests_${SNAPSHOT_PROTO}/test_@' \
# 4. sanity check: `rgrep alpha` and other protocol names in the set of
# snapshotted tests. remove absolute protocol references to relative ones
# using the constants of tests_python/tests_alpha/protocol.py
# 5. remove multibranch test tests_${SNAPSHOT_PROTO}/multibranch/test_baker_endorser_mb.py if still
# present. it is no longer maintained, and there cannot be two modules named multibranch.
# 6. update tests_python/README.md (this file) with a description of the new folders.
# 7. update .gitlab-ci.yml by running ./scripts/update_integration_tests.sh
# 5. update tests_python/README.md (this file) with a description of the new folders.
# 6. update .gitlab-ci.yml by running ./scripts/update_integration_tests.sh
# Sanity checks
......
......@@ -620,118 +620,3 @@ class Sandbox:
print(f'# endorser {endo_id} for proto {proto} has failed')
daemons_alive = False
return daemons_alive
class SandboxMultiBranch(Sandbox):
"""Specialized version of `Sandbox` using binaries with different versions.
Binaries are looked up according to a map from node_id to branch
For instance, if we define `branch_map` as:
branch_map = {i: 'zeronet' if i % 2 == 0 else 'alphanet'
for i in range(20)}
Nodes/client for even ids will be looked up in `binaries_path/zeronet`
Nodes/client for odd ids will be looked up in `binaries_path/alphanet`
One advantage of using `SandboxMultibranch` rather than `Sandbox` is that
we can sometimes run the same tests with different binaries revision by
simply changing the sandbox.
"""
def __init__(
self,
binaries_path: str,
identities: Dict[str, Dict[str, str]],
branch_map: Dict[int, str],
rpc: int = 18730,
p2p: int = 19730,
num_peers: int = 45,
log_dir: str = None,
singleprocess: bool = False,
):
"""Same semantics as Sandbox class, plus a `branch_map` parameter"""
super().__init__(
binaries_path,
identities,
rpc,
p2p,
num_peers,
log_dir,
singleprocess,
)
self._branch_map = branch_map
for branch in list(branch_map.values()):
error_msg = f'{binaries_path}/{branch} not a dir'
assert os.path.isdir(f'{binaries_path}/{branch}'), error_msg
def add_baker(
self,
node_id: int,
account: str,
proto: str,
params: List[str] = None,
branch: str = "",
) -> None:
"""branch is overridden by branch_map"""
branch = self._branch_map[node_id]
super().add_baker(node_id, account, proto, params, branch)
def add_endorser(
self,
node_id: int,
account: str,
proto: str,
endorsement_delay: int = 0,
branch: str = "",
) -> None:
"""branch is overridden by branch_map"""
branch = self._branch_map[node_id]
super().add_endorser(node_id, account, proto, endorsement_delay, branch)
def add_accuser(
self,
node_id: int,
proto: str,
branch: str = "",
) -> None:
"""branch is overridden by branch_map"""
branch = self._branch_map[node_id]
super().add_accuser(node_id, proto, branch)
def add_node(
self,
node_id: int,
node_dir: str = None,
peers: List[int] = None,
params: List[str] = None,
log_levels: Dict[str, str] = None,
private: bool = True,
config_client: bool = True,
use_tls: Tuple[str, str] = None,
snapshot: str = None,
reconstruct: bool = False,
branch: str = "",
node_config: dict = None,
mode: str = None,
client_factory: Callable = Client,
) -> None:
assert not branch
branch = self._branch_map[node_id]
super().add_node(
node_id,
node_dir,
peers,
params,
log_levels,
private,
config_client,
use_tls,
snapshot,
reconstruct,
branch,
node_config,
mode,
client_factory,
)
[pytest]
; if you unignore multibranch tests, please update update_integration_test.sh
addopts = --strict --ignore=tests/multibranch --verbose --tb=short
addopts = --strict --verbose --tb=short
markers =
incremental
slow
......@@ -9,7 +8,6 @@ markers =
endorser
multinode
contract
multibranch
mempool
client
snapshot
......
#!/usr/bin/env python3
import subprocess
import shutil
import glob
import pathlib
import os
import re
import argparse
from typing import List, Any
def print_log(log, color=True):
color_code = '\033[34m'
endc = '\033[0m'
print(f'{color_code}# {log}{endc}' if color else f'# {log}')
def print_command(cmd, color=True):
color_code = '\033[34m'
endc = '\033[0m'
cmd_str = " ".join(cmd)
print(f'{color_code}# {cmd_str}{endc}' if color else f'# {cmd_str}')
# simple parser for `opam env`
TERM_REGEX = r'''(?mx)
\s*(?:
(?P<brackl>\()|
(?P<brackr>\))|
(?P<num>\-?\d+\.\d+|\-?\d+)|
(?P<sq>"[^"]*")|
(?P<s>[^(^)\s]+)
)'''
def parse_sexp(sexp):
stack = []
out: List[Any] = []
for termtypes in re.finditer(TERM_REGEX, sexp):
term, value = [(t, v) for t, v in termtypes.groupdict().items() if v][0]
if term == 'brackl':
stack.append(out)
out = []
elif term == 'brackr':
assert stack, "Trouble with nesting of brackets"
tmpout, out = out, stack.pop(-1)
out.append(tmpout)
elif term == 'num':
val = float(value)
if val.is_integer():
val = int(val)
out.append(val)
elif term == 'sq':
out.append(value[1:-1])
elif term == 's':
out.append(value)
else:
raise NotImplementedError(f'Error: {term, value}')
assert not stack, "Trouble with nesting of brackets"
return out[0]
def opam_env(tezos_build):
process = subprocess.Popen(
['opam', 'env', '--sexp', '--set-switch'],
stdout=subprocess.PIPE,
cwd=tezos_build,
)
out, _err = process.communicate()
out_str = out.decode('utf-8')
env = {x[0]: x[1] for x in parse_sexp(out_str)}
return env
def run(cmd, cwd, env=None):
print_command(cmd)
if env is None:
subprocess.run(cmd, check=True, cwd=cwd)
else:
subprocess.run(cmd, check=True, cwd=cwd, env=env)
def build(branch, tezos_home, tezos_build, tezos_binaries):
if os.listdir(tezos_build):
error_msg = f'{tezos_build} is not empty. Should be a git directory'
assert os.path.isdir(f"{tezos_build}/.git"), error_msg
else:
print_log(f'{tezos_build} is empty. Cloning {tezos_home}')
run(['git', 'clone', tezos_home], tezos_build)
run(['git', 'clean', '-f'], tezos_build)
run(['git', 'reset', '--hard'], tezos_build)
run(['git', 'checkout', branch], tezos_build)
run(['make', 'build-deps'], tezos_build)
new_env = opam_env(tezos_build)
print_log(f'Extending current env with opam env: {new_env}')
env = {**os.environ, **new_env}
run(['make'], tezos_build, env=env)
branch_dir = os.path.join(tezos_binaries, branch)
print_log(f'Copying binaries to {branch_dir}')
pathlib.Path(branch_dir).mkdir(parents=True, exist_ok=True)
for filename in glob.glob(f'{tezos_build}/tezos-*'):
dest = os.path.join(branch_dir, os.path.basename(filename))
if os.path.exists(dest):
print_log(f"{dest} already exists, don't copy")
print_log(f'copy {filename} to {branch_dir}')
shutil.copy(filename, branch_dir)
def prepare_binaries(tezos_home, tezos_build, tezos_binaries, branch_list):
assert branch_list, "branch list is empty"
assert os.path.isdir(tezos_binaries), f"{tezos_binaries} doesn't exist"
assert os.path.isdir(tezos_build), f"{tezos_build} doesn't exist"
assert os.path.isdir(tezos_home), f"{tezos_home} doesn't exist"
for branch in branch_list:
branch_dir = os.path.join(tezos_binaries, branch)
if os.path.isdir(branch_dir) and os.listdir(branch_dir):
print_log(f"Binaries for branch {branch} found. Skip.")
else:
print_log(f"Binaries for branch {branch} not found. Build.")
build(branch, tezos_home, tezos_build, tezos_binaries)
def main():
parser = argparse.ArgumentParser(description='build_branch.py')
parser.add_argument(
'--clone',
dest='clone_dir',
metavar='DIR',
help='repository to be cloned',
required=True,
)
parser.add_argument(
'--build-dir',
dest='build_dir',
metavar='DIR',
help='repository where executables will be built',
required=True,
)
parser.add_argument(
'--bin-dir',
dest='bin_dir',
metavar='DIR',
help='repository where executables will be copied',
required=True,
)
parser.add_argument(
'branches',
metavar='BRANCH',
type=str,
nargs='*',
help='list of branches',
)
args = parser.parse_args()
prepare_binaries(
args.clone_dir, args.build_dir, args.bin_dir, args.branches
)
if __name__ == "__main__":
main()
......@@ -14,7 +14,7 @@ from pytest_regtest import (
deregister_converter_pre,
_std_conversion,
)
from launchers.sandbox import Sandbox, SandboxMultiBranch
from launchers.sandbox import Sandbox
from tools import constants, paths, utils
from tools.client_regression import ClientRegression
from client.client import Client
......@@ -186,47 +186,6 @@ def clients(sandbox: Sandbox, request) -> Iterator[List[Client]]:
yield clients
@pytest.fixture(scope="class")
def sandbox_multibranch(log_dir, request) -> Iterator[SandboxMultiBranch]:
"""Multi-branch sandbox fixture. Parameterized by map of branches.
This fixture is identical to `sandbox` except that each node_id is
mapped to a pair (git revision, protocol version). For instance,
suppose a mapping:
MAP = { 0: ('zeronet', 'alpha'), 1:('mainnet', '003-PsddFKi3'),
2: ('alphanet', '003-PsddFKi3' }
If we annotate the class test as follows.
@pytest.mark.parametrize('sandbox_multibranch', [MAP], indirect=True)
The executables (node, baker, endorser)
- for node_id 0 will be looked up in `TEZOS_BINARY/zeronet`,
- for node_id 1 will be looked up in `TEZOS_BINARY/mainnet` and so on...
baker and endorser will use the specified protocol version, according
to the tezos executables naming conventions.
"""
if paths.TEZOS_BINARIES is None:
pytest.skip()
branch_map = request.param
assert branch_map is not None
num_peers = max(branch_map) + 1
assert paths.TEZOS_BINARIES is not None # helps `mypy`
with SandboxMultiBranch(
paths.TEZOS_BINARIES,
constants.IDENTITIES,
num_peers=num_peers,
log_dir=log_dir,
branch_map=branch_map,
) as sandbox:
yield sandbox
# this assertion checks that daemons (baker, endorser, node...) didn't
# fail unexpected.
assert sandbox.are_daemons_alive(), DEAD_DAEMONS_WARN
def pytest_collection_modifyitems(config, items):
"""Adapted from pytest-fixture-marker: adds the regression marker
to all tests that use the regtest fixture.
......
......@@ -14,7 +14,7 @@ from pytest_regtest import (
deregister_converter_pre,
_std_conversion,
)
from launchers.sandbox import Sandbox, SandboxMultiBranch
from launchers.sandbox import Sandbox
from tools import constants, paths, utils
from tools.client_regression import ClientRegression
from client.client import Client
......@@ -186,47 +186,6 @@ def clients(sandbox: Sandbox, request) -> Iterator[List[Client]]:
yield clients
@pytest.fixture(scope="class")
def sandbox_multibranch(log_dir, request) -> Iterator[SandboxMultiBranch]:
"""Multi-branch sandbox fixture. Parameterized by map of branches.
This fixture is identical to `sandbox` except that each node_id is
mapped to a pair (git revision, protocol version). For instance,
suppose a mapping:
MAP = { 0: ('zeronet', 'alpha'), 1:('mainnet', '003-PsddFKi3'),
2: ('alphanet', '003-PsddFKi3' }
If we annotate the class test as follows.
@pytest.mark.parametrize('sandbox_multibranch', [MAP], indirect=True)
The executables (node, baker, endorser)
- for node_id 0 will be looked up in `TEZOS_BINARY/zeronet`,
- for node_id 1 will be looked up in `TEZOS_BINARY/mainnet` and so on...
baker and endorser will use the specified protocol version, according
to the tezos executables naming conventions.
"""
if paths.TEZOS_BINARIES is None:
pytest.skip()
branch_map = request.param
assert branch_map is not None
num_peers = max(branch_map) + 1
assert paths.TEZOS_BINARIES is not None # helps `mypy`
with SandboxMultiBranch(
paths.TEZOS_BINARIES,
constants.IDENTITIES,
num_peers=num_peers,
log_dir=log_dir,
branch_map=branch_map,
) as sandbox:
yield sandbox
# this assertion checks that daemons (baker, endorser, node...) didn't
# fail unexpected.
assert sandbox.are_daemons_alive(), DEAD_DAEMONS_WARN