Testware and examples for testing infrastructure and its configuration

This MR adds some testware for testing our infrastructure configuration, i.e. helm charts.

CI pipeline

Two new jobs, helm-template-pytest and chart-pytest have been created. Each will execute pytest with appropriate markers enabled in the gitlab CI runner.

Helm template tests

helm-template-pytest runs as part of the static_analysis stage in the pipeline and is intended to execute tests that do not require resources to be deployed in a cluster. Tests marked with no_deploy marker will be executed here and overlap somewhat with what helm-unittest provides.

See: /tests/tango_base_chart_test.py:test_databaseds_resource_definition_should_have_TANGO_HOST_set_to_its_own_hostname

Helm chart tests

chart-pytest runs as part of the unit_test stage and will execute tests marked with the chart_deploymarker. A unit in this case is a helm chart that will be deployed and tested.

See: /tests/tango_base_chart_test.py:test_tangodb_pod_should_have_mysql_server_running

Gitlab runner image

Since pytest will be running directly in the gitlab-runner, it's important that the base docker image has the dependencies installed. These dependencies are encapsulated in the test-requirements.txt file which is used by the build of the ska-docker/deploy docker image.

Pytest

Pytest is configured to collect python tests in the /tests/ directory and the following markers are explicitly defined (see pytest.ini):

• no_deploy: indicate tests that do not require deployments of resources to a cluster.
• chart_deploy: indicates tests that will deploy a chart to cluster.

These correspond to the make targets defined in test.mk, template_pytests and chart_pytests, which are invoked as part of the CI pipeline.

Test lifecycle

The lifecycle (setup, teardown) of tests are managed by pytest fixtures, defined in conftest.py.

The infratest_context fixture in particular will determine if tests that involve deployments are included in the pytest run, i.e. the chart_deploy marker is included. It will then:

1. invoke kubectl to create a namespace for the test resources(pods) to be deployed into
2. ensure this namespace is deleted after the test run

Note: the default namespace is ci, but can be overriden by specifying the custom pytest option, --test-namespace. When running inside the pipeline, this flag is set to ci-$CI_JOB_ID so each job will use its own namespace and resources, ensuring test isolation.

Test Support

Charts are deployed via Helm and the HelmTestAdaptor is a rudimentary adaptor class to manage the interaction with the Helm CLI.

The ChartDeployment class is an abstraction to represent a deployed chart and offers access to its resources (by querying the kubernetes API) and metadata (such as release_name). In fact, instantiating a ChartDeployment in code will deploy the specified chart.

A useful pattern is to create pytest fixture that represents the chart to be deployed and yields a ChartDeployment object. It can also call .delete() to ensure the chart is deleted and pytest fixture scope can be used to a chart's lifespan. For an example see the tango_base_release fixture in /tests/tango_base_chart_test.py.

Third-party libraries

• python kubernetes client is the official kubernetes API client for Python. It's provided as a pytest fixture, k8s_api and also used by ChartDeployment to obtain a list of deployed pods(see get_pods method).
• testinfra is a library that allows connecting to pods and asserting on the state of various things inside them such as open ports, directory structure, user accounts, etc.

Executing locally

You should be able to execute the tests locally as long as you have the following dependencies:

1. kubectl installed and configured pointing to a k8s cluster of your choice (such as minikube).
2. helm v2 is installed with the tiller plugin*.
3. pytest and dependencies are installed: pip install -r test-requirements.txt. Use a virtualenv so you don't pollute your system's Python site-packages.

* This has only been built and tested with helm v2 since that's what's running in the pipeline currently. helm plugin install https://github.com/rimusz/helm-tiller

Note: the custom --use-tiller-plugin flag will just determine whether the helm commands that HelmTestAdaptor invokes will be prefixed with helm tiller run --

no_deploy tests

pytest --use-tiller-plugin -m no_deploy or make template_pytests

chart_deploy tests

pytest --use-tiller-plugin -m chart_deploy or make chart_pytests

all tests

pytest --use-tiller-plugin