devsetup.rst 13.1 KB
Newer Older
1 2 3
=====================
The Contributor Guide
=====================
4 5 6

Mailman 3 consists of a collection of separate-but-linked projects, each of
which has its own development setup guide.  This makes sense when you want to
7
focus on a single piece of Mailman, but can make setting up the entire Mailman
8 9 10 11
Suite in one go somewhat confusing.  This guide attempts to move all the
information currently in the wiki and various package documentation into a
single "definitive" guide.

12
Main package documentation on Readthedocs.io:
13 14 15 16 17
  * `Mailman core start guide <https://mailman.readthedocs.io/en/release-3.0/src/mailman/docs/START.html>`_
  * `Mailman core "web ui in 5" guide <https://mailman.readthedocs.io/en/release-3.0/src/mailman/docs/WebUIin5.html>`_
  * `Mailman core "archive in 5" <https://mailman.readthedocs.io/en/release-3.0/src/mailman/docs/ArchiveUIin5.html>`_
  * `Postorius dev guide <http://postorius.readthedocs.io/en/latest/development.html>`_
  * `Hyperkitty dev guide <http://hyperkitty.readthedocs.io/en/latest/development.html>`_
18 19 20 21


Getting prerequisites
---------------------
Terri Oda's avatar
Terri Oda committed
22

23 24 25 26 27
For the most part, setup for each project will download any needed packages.
However, you will need a few system packages to be sure you've got the
necessary version of Python and its tools, git (to get the source code),
postfix (a mail server), and a few other tools that are used during setup.

Barry Warsaw's avatar
Barry Warsaw committed
28
On Fedora, you probably want to run::
Terri Oda's avatar
Terri Oda committed
29

30
    $ sudo yum install python3-setuptools  python3-virtualenv python3-devel git gcc nodejs-less postfix python3-tox
31 32

On Debian and Ubuntu, this may be something like::
Terri Oda's avatar
Terri Oda committed
33

34
    $ sudo apt install python3-setuptools  python3-virtualenv python3-dev git gcc node-less nodejs postfix python3-tox
35

Barry Warsaw's avatar
Barry Warsaw committed
36 37 38
If you prefer, you can substitute Exim4 for Postfix.  Postfix is the MTA used
by most Mailman developers, but we do support Exim 4.  (`Sendmail support is
very much desired`_, but the Mailman core developers need contributors with
39 40
Sendmail expertise to help.) For development purposes it doesn't matter, since
we will mock all interactions to external MTA.
41

42 43
You will need `tox <https://tox.readthedocs.io/en/latest/>`_ to run tests. 
HyperKitty also needs ``sassc``.
44 45

**FIXME:** Add instructions on how to get sassc on a few platforms.
Terri Oda's avatar
Terri Oda committed
46

Barry Warsaw's avatar
Barry Warsaw committed
47

48 49 50
Gitlab Setup
------------

51
We use `Gitlab <https://gitlab.com/mailman>`_ for source code hosting and
52
our CI. You can `fork
53
<https://docs.gitlab.com/ee/gitlab-basics/fork-project.html>`_ any of the projects
54
you want and start working on it. If you don't already have an account on
55 56
`Gitlab <https://gitlab.com>`_, please create one, you will need that for contributing
code or participating in any other way.
57

58
We also use Gitlab for code reviews. Our workflow looks very
59
similar to the official `Gitlab Workflow
60
<https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/>`_. Please
61
remember to `enable shared runners
62
<https://docs.gitlab.com/ee/ci/quick_start/#shared-runners>`_ on your fork, it
63 64 65 66 67
will be used to build your code and run unittests on pull requests that you will
make. It is mandatory that you have runners enabled before you send any pull
requests.


68 69
Set up a directory
------------------
Terri Oda's avatar
Terri Oda committed
70

Barry Warsaw's avatar
Barry Warsaw committed
71 72 73 74
Setting up the whole Mailman suite means you have to pull code from a bunch of
different related repositories.  You can put all the code anywhere you want,
but you might want to set up a directory to keep all the pieces of mailman
together.  For example::
Terri Oda's avatar
Terri Oda committed
75

Barry Warsaw's avatar
Barry Warsaw committed
76 77
    $ mkdir ~/mailman
    # cd ~/mailman
78 79

For the rest of this development guide, we are going to assume you're using
Barry Warsaw's avatar
Barry Warsaw committed
80 81
``~/mailman`` as your directory, but you can use whatever you want.

82 83 84

Set up virtual environments
---------------------------
Terri Oda's avatar
Terri Oda committed
85

86 87 88
All parts of Mailman support only Python 3.5+. For your development, it is
advised that you create a virtualenv so that the packages you install don't
break any of the system packages using Python.
89

90
To create the virtualenv run the following command::
Terri Oda's avatar
Terri Oda committed
91

92
    $ python3 -m venv venv3
93 94

To activate a virtualenv, you need to run the appropriate activate script::
Terri Oda's avatar
Terri Oda committed
95

96
    $ source venv3/bin/activate
Barry Warsaw's avatar
Barry Warsaw committed
97

98 99 100
You *must* use ``source`` (or ``.`` if your shell is a pure POSIX shell)
everytime you want to activate your development environment. To make your life
easier when managing virtualenvs, see `virtualenvwrapper
101
<https://virtualenvwrapper.readthedocs.io/en/latest/>`_ .
102 103 104 105


Set up and run Mailman Core
---------------------------
Terri Oda's avatar
Terri Oda committed
106

107
First, get the code::
Terri Oda's avatar
Terri Oda committed
108

Barry Warsaw's avatar
Barry Warsaw committed
109
    $ cd ~/mailman
110
    $ git clone https://gitlab.com/mailman/mailman.git
Barry Warsaw's avatar
Barry Warsaw committed
111 112 113

To set up Mailman Core, you'll need to switch to your Python 3 virtualenv::

114
    $ source venv3/bin/activate
Terri Oda's avatar
Terri Oda committed
115

Barry Warsaw's avatar
Barry Warsaw committed
116 117 118
Then, go into the mailman directory, run setup, and then run ``mailman info``
to be sure everything is set up correctly, and that the right settings are in
place::
119

Barry Warsaw's avatar
Barry Warsaw committed
120 121 122
    $ cd mailman
    $ python setup.py develop
    $ mailman info
123

124 125
You can edit your ``mailman.cfg`` file to make any necessary changes.  By
default, during development, it is located at ``var/etc/mailman.cfg``. Then
Barry Warsaw's avatar
Barry Warsaw committed
126
start things up::
Terri Oda's avatar
Terri Oda committed
127

Barry Warsaw's avatar
Barry Warsaw committed
128 129
    $ mailman start
    $ cd ..
130

131 132 133 134 135
Note that mailman just makes a ``var/`` directory wherever you start it and uses
that to store your data.  This is great for the purposes of testing so you can
easily make fresh installs, but might be confusing if you restart your instance
later from a different location and don't have your original mailman.db file, or
if you start looking around and finding var/ directories everywhere.
Terri Oda's avatar
Terri Oda committed
136

137
Later on, if you need to restart Mailman (i.e. if you get the error "Mailman
Barry Warsaw's avatar
Barry Warsaw committed
138 139
REST API not available. Please start Mailman core.") then you can also do that
by calling the ``mailman`` executable from the venv as follows::
Terri Oda's avatar
Terri Oda committed
140

141
    $ ~/mailman/venv3/bin/mailman start
142

Barry Warsaw's avatar
Barry Warsaw committed
143 144
Note that the ``mailman`` executable has several sub-commands.  One
that is particularly useful for debugging is ``mailman shell``.
145

146
.. note:: If you like `IPython <https://ipython.org/>`_ shell (like I do!), you
147 148
          add the following to your ``mailman.cfg``::

149 150
            [shell]
            use_ipython: yes
151

152
		Also, remember to install ipython using pip::
153

154 155 156 157 158 159
            $ pip install ipython


You can run tests for Mailman Core (or any Mailman project) using `tox <https://tox.readthedocs.io/en/latest/>`
::

160

161 162 163 164
    $ tox -e py37-nocov

This requires that you have Python3.7 installed. You change it to ``py36-nocov`` and ``py35-nocov`` to run
tests with Python 3.6 and 3.5 respectively. 
165 166 167

Set up Mailman Client
---------------------
Terri Oda's avatar
Terri Oda committed
168

169
Get the code::
Terri Oda's avatar
Terri Oda committed
170

Barry Warsaw's avatar
Barry Warsaw committed
171
    $ cd ~/mailman
172
    $ git clone https://gitlab.com/mailman/mailmanclient.git
173 174

Then set up mailmanclient::
Terri Oda's avatar
Terri Oda committed
175

Barry Warsaw's avatar
Barry Warsaw committed
176 177 178
    $ cd mailmanclient
    $ python setup.py develop
    $ cd ..
179

180 181 182 183
To run the tests::

    $ tox -e py37

184 185
Set up Django-mailman3
----------------------
186

Barry Warsaw's avatar
Barry Warsaw committed
187 188
This package holds the Django libraries and templates used by Postorius and
HyperKitty.
189 190

Get the code and set it up::
Terri Oda's avatar
Terri Oda committed
191

Barry Warsaw's avatar
Barry Warsaw committed
192
    $ cd ~/mailman
193
    $ git clone https://gitlab.com/mailman/django-mailman3.git
Barry Warsaw's avatar
Barry Warsaw committed
194 195 196
    $ cd django-mailman3
    $ python setup.py develop
    $ cd ..
197

198 199 200 201
To run the tests::

    $ tox -e py37-django21

202 203 204 205 206 207 208 209 210 211

Set up and run Postorius
------------------------

The Postorius documentation, including a more extensive setup guide, can be
found here: http://postorius.readthedocs.org/

Make sure to install mailmanclient and django-mailman3 before setting up
Postorius. (If you're following this guide in order, you've just done that.)

212
Get the code and run setup.  Make sure you're in venv which has Python 3.5+ for Postorius::
Terri Oda's avatar
Terri Oda committed
213

Barry Warsaw's avatar
Barry Warsaw committed
214
    $ cd ~/mailman
215
    $ git clone https://gitlab.com/mailman/postorius.git
Barry Warsaw's avatar
Barry Warsaw committed
216 217 218
    $ cd postorius
    $ python setup.py develop
    $ cd ..
219

Barry Warsaw's avatar
Barry Warsaw committed
220
Postorius and HyperKitty both come with ``example_project`` directories with
221 222
basic configuration so you can try them out.  For this tutorial, however,
we'll be using a project that combines both instead.
Terri Oda's avatar
Terri Oda committed
223

224 225 226
You can run tests using::

    $ tox -e py37-django21
227

228 229 230

Set up a Fake mail server
-------------------------
231

232 233 234 235
To be able to actually receive emails, you need to setup a mail server. Mailman
core receives emails over LMTP Protocol, which most of the modern MTAs
support. However, setup instructions are provided only for Postfix, Exim4 and
qmail. Please refer to the `MTA documentation`_ at Mailman Core for the details.
Barry Warsaw's avatar
Barry Warsaw committed
236

237 238
You will also have to add some settings to your django configuration. The setup
instructions are provided in `django's email documentation`_.
239

240 241 242 243 244 245
For development setup, you don't _have_ to install a working MTA. You can add
the following to your ``mailman.cfg`` to make sure that it doesn't try to send
emails out::

  [devmode]
  enabled: yes
246
gitlab  recipient: you@yourdomain.com
247 248 249 250 251 252 253 254 255 256 257

  [mta]
  smtp_port: 9025
  lmtp_port: 9024
  incoming: mailman.testing.mta.FakeMTA

Also, in Django you can add the following configuration to your
``settings.py``::

  EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

258 259 260 261 262 263
This writes everything to ``stdout``. There are `other email backends <https://docs.djangoproject.com/en/2.1/topics/email/#obtaining-an-instance-of-an-email-backend>`_ 
available to use for testing like 
``django.core.mail.backends.filebased.EmailBackend`` that one can use
to write outgoing emails to a file on disk. Please see the docs for
other options.

264

Barry Warsaw's avatar
Barry Warsaw committed
265
Set up and run HyperKitty
266
-------------------------
Terri Oda's avatar
Terri Oda committed
267

268 269 270 271 272 273 274
Complete guide here:
https://hyperkitty.readthedocs.org/en/latest/development.html

Make sure to install mailmanclient and django-mailman3 before setting up
Hyperkitty. (If you're following this guide in order, you've just done that.)

Get the code and run setup::
Terri Oda's avatar
Terri Oda committed
275

Barry Warsaw's avatar
Barry Warsaw committed
276
    $ cd ~/mailman
277
    $ git clone https://gitlab.com/mailman/hyperkitty.git
Barry Warsaw's avatar
Barry Warsaw committed
278 279 280
    $ cd hyperkitty
    $ python setup.py develop
    $ cd ..
281

Barry Warsaw's avatar
Barry Warsaw committed
282
Postorius and HyperKitty both come with ``example_project`` directories with
283 284 285 286
basic configuration so you can try them out.  By default, they both use port
8000, so if you do want to run both example projects at the same time, do
remember that you'll need to specify a different port on the command line for
one of them.
287

288 289 290 291
You can run tests using::

    $ tox -e py37-django21

Barry Warsaw's avatar
Barry Warsaw committed
292
However, we're going to run them both in a single Django instance at the end
293
of this guide, so don't worry about ports right now.
294

Barry Warsaw's avatar
Barry Warsaw committed
295

296 297 298
Set up mailman-hyperkitty
-------------------------

Barry Warsaw's avatar
Barry Warsaw committed
299 300
``mailman-hyperkitty`` is the package that actually sends the incoming emails
to HyperKitty for archiving. Note that this is one of the components that uses
301 302
Python 3.

303
Setting it up::
Terri Oda's avatar
Terri Oda committed
304

Barry Warsaw's avatar
Barry Warsaw committed
305
    $ cd ~/mailman
306
    $ git clone https://gitlab.com/mailman/mailman-hyperkitty.git
Barry Warsaw's avatar
Barry Warsaw committed
307 308 309
    $ cd mailman-hyperkitty
    $ python setup.py develop
    $ cd ..
310

Barry Warsaw's avatar
Barry Warsaw committed
311 312 313
You'll need to fix the default ``mailman-hyperkitty.cfg`` file to use the
correct url for HyperKitty.  If you're running it on http://localhost:8002
then you need to change ``base_url`` to match that.
314

315 316 317
You can run tests using::

    $ tox -e py37-coverage
318 319 320 321

Link Mailman to HyperKitty
--------------------------

Barry Warsaw's avatar
Barry Warsaw committed
322
Now you have to enable HyperKitty in Mailman.  To do that, edit the
323
``mailman.cfg`` (in ``~/mailman/mailman/var/etc``, or wherever the output of
Barry Warsaw's avatar
Barry Warsaw committed
324 325 326
``mailman info`` says it is) and add the following config. Note that you need
to fill in the absolute path to your ``mailman-hyperkitty.cfg`` in the
configuration below::
Terri Oda's avatar
Terri Oda committed
327

Barry Warsaw's avatar
Barry Warsaw committed
328 329 330 331 332
    # mailman.cfg
    [archiver.hyperkitty]
    class: mailman_hyperkitty.Archiver
    enable: yes
    configuration: <absolute path to mailman-hyperkitty.cfg>
333 334


335 336
Run the Mailman Suite (combined hyperkitty+postorius)
-----------------------------------------------------
Barry Warsaw's avatar
Barry Warsaw committed
337

338 339 340 341 342 343 344 345 346
You can run HyperKitty and Postorius as separate applications, but many
developers are going to want to run them on a single server.  The
configuration files for this are in a repository called mailman-suite.

The first time you run the suite, you will want to set up a superuser
account.  This is the account you will use in the web interface to set up
your first domains.  Please enter an email address otherwise the database won't
be setup correctly and you will run into errors later::

Barry Warsaw's avatar
Barry Warsaw committed
347
    $ cd ~/mailman
348
    $ git clone https://gitlab.com/mailman/mailman-suite.git
Barry Warsaw's avatar
Barry Warsaw committed
349 350 351
    $ cd mailman-suite/mailman-suite_project
    $ python manage.py migrate
    $ python manage.py createsuperuser
352

Barry Warsaw's avatar
Barry Warsaw committed
353 354
You'll want to run the following commands in a window where you can leave them
running, since it dumps all the django logs to the console::
355

Barry Warsaw's avatar
Barry Warsaw committed
356
    $ python manage.py runserver
357

Barry Warsaw's avatar
Barry Warsaw committed
358 359 360 361
At this point, you should be able to see Mailman Suite running!  In the
default setup, you can go to http://127.0.0.1:8000 and start poking around.
You should be able to use the superuser account you created to log in and
create a domain and then some lists.
362 363 364 365

The default config file uses a dummy email backend created by this line in
settings.py::

Barry Warsaw's avatar
Barry Warsaw committed
366
    EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
367

Barry Warsaw's avatar
Barry Warsaw committed
368
Using this backend, all emails will be printed to the Postorius console
369
(rather than sent as email) so you can get the url to verify your email from
370 371 372
the console. You can also use 
`FileBackend <https://docs.djangoproject.com/en/2.1/topics/email/#file-backend>` 
to write emails to a file on disk.
373

Barry Warsaw's avatar
Barry Warsaw committed
374 375 376
Don't leave the console email backend configured and running once you get to
the point where you want to send real emails, though!

377

Barry Warsaw's avatar
Barry Warsaw committed
378
.. _`Sendmail support is very much desired`: https://gitlab.com/mailman/mailman/issues/307
379
.. _`MTA documentation`: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html
380
.. _`django's email documentation`: https://docs.djangoproject.com/en/1.10/topics/email/#topic-email-backends