Commit 6840598d authored by Abhilash Raj's avatar Abhilash Raj

Improve testing by auto starting and stopping Mailman Core for tests.

parent a1569872
image: maxking/mailman-ci-runner
- tox -e py35
# Example Mailman config file suitable for testing
enabled: yes
testing: yes
recipient: [email protected]
smtp_port: 9025
lmtp_port: 9024
incoming: mailman.testing.mta.FakeMTA
port: 9001
enable: yes
enable: yes
enable: yes
addopts = --doctest-glob='*.rst'
addopts = --doctest-glob='*.rst' --tb=short --run-services
# Copyright (C) 2017-2018 by the Free Software Foundation, Inc.
# This file is part of mailman.client.
# mailman.client is free software: you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by the
# Free Software Foundation, version 3 of the License.
# mailman.client is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public
# License for more details.
# You should have received a copy of the GNU Lesser General Public License
# along with mailman.client. If not, see <>.
"""Testing utilities."""
import pytest
import subprocess
import socket
from contextlib import closing
def check_core():
with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
if sock.connect_ex(('localhost', 9001)) == 0:
return True
return False
@pytest.fixture(scope='module', autouse=True)
def mailman_core(request, watcher_getter):
"""Mailman Core instance which is ready to be used by the tests"""
def teardown_core():
print('Stopping Mailman Server')['mailman', 'stop'])['rm', '-rf', 'var/'])
print('Starting Mailman Server')
return watcher_getter(
arguments=['-C', 'mailman_test.cfg'],
......@@ -7,98 +7,25 @@ Running Tests
The test suite is run with the `tox`_ tool, which allows it to be run against
multiple versions of Python. There are two modes to the test suite:
multiple versions of Python. The tests are discovered and run using `pytest`_.
* *Record mode* which is used to record the HTTP traffic against a live
Mailman 3 REST server.
* *Replay mode* which allows you to run the test suite off-line, without
running the REST server.
To run the test suite, run::
Whenever you add tests for other parts of the REST API, you need to run the
suite once in record mode to generate the YAML file of HTTP requests and
$ tox
Then you can run the test suite in replay mode as often as you want, and
Mailman 3 needn't even be installed on your system.
To run tests for only one version of Python, you can run::
Since this branch ships with a recording file, you don't need to run in record
mode to start with.
$ tox -e py36
``pytest`` starts Mailman Core using ``pytest-services`` plugin and
automatically manages it's start and stop cycle for every module.
Replay mode
To run the test suite in replay mode (the default), just run the following::
$ tox
This will attempt to run the test suite against Python 2.7, 3.4, 3.5 and 3.6 or
whatever combination of those that are available on your system.
Record mode
Start by branching the Mailman 3 code base, then you should install it into a
virtual environment. The easiest way to do this is with `tox`::
$ tox --notest -r
Now, use the virtual environment that `tox` creates to create a template `var`
directory in the current directory::
$ .tox/py34/bin/mailman info
Now you need to modify the ``var/etc/mailman.cfg`` configuration file, so that
it contains the following::
enabled: yes
testing: yes
recipient: [email protected]
smtp_port: 9025
lmtp_port: 9024
incoming: mailman.testing.mta.FakeMTA
port: 9001
enable: yes
enable: yes
enable: yes
Now you can start Mailman 3::
$ .tox/py34/bin/mailman start
Back in your ``mailmanclient`` branch, run the test suite in record mode::
$ tox -e record
You should now have an updated recording file (``tape.yaml``).
If you find you need to re-run the test suite, you *must* first stop the
Mailman REST server, and then delete the ``mailman.db`` file, since it
contains state that will mess up the ``mailmanclient`` test suite::
$ cd <mailman3-branch>
$ .tox/py34/bin/mailman stop
$ rm -f var/data/mailman.db
$ .tox/py34/bin/mailman start
$ cd <mailmanclient-branch>
$ tox -e record
Once you're done recording the HTTP traffic, you can stop the Mailman 3 server
and you won't need it again. It's a good idea to commit the ``tape.yaml``
changes for other users of your branch.
.. note:: Previously, we used ``vcrpy`` and ``pytest-vcr`` packages to manage
recorded tapes for interaction with Mailman Core. That was replaced
with ``pytest-services`` plugin, which instead start Core for every
.. _`tox`:
.. _`pytest`:
......@@ -119,6 +119,7 @@ one.
... print('{0}: {1}'.format(style['name'], style['description']))
legacy-announce: Announce only mailing list style.
legacy-default: Ordinary discussion mailing list style.
private-default: Discussion mailing list style with private archives.
>>> print(styles['default'])
......@@ -652,7 +653,7 @@ for the settings is returned which behaves much like a dictionary.
>>> settings = test_one.settings
>>> len(settings)
>>> for attr in sorted(settings):
... print(attr + ': ' + str(settings[attr]))
......@@ -2,9 +2,8 @@
envlist = py{35,36,37},lint
basepython = python3
usedevelop = True
commands = python -m pytest --vcr-record-mode=once
commands = pytest --vcr-record-mode=all
deps =
......@@ -12,6 +11,8 @@ deps =
basepython = python3
Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment