This patch contains some readability improvements to testlib: it
adds/reformats some comments for exported functions for consistency, and
unexports some structs that are not used outside the library.

The smtpsrv fuzzer doesn't handle DATA commands particularly well:
it will continue to read but will skip lines that have STARTTLS as
content, and only really care for the first line due to a bug.

This patch fixes the handling, and moves the logic to a separate
function for readability.

When the client closes the connection, which is very common, chasquid
logs it as an error ("exiting with error: EOF").

That can be confusing and mislead users, and also makes a lot of
traces be marked as errored, when nothing wrong occurred.

So this patch changes the log to not treat it as an error.

This patch adds a section on dovecot authentication troubleshooting,
with common suggestions that can help identify what is going on when
the chasquid-dovecot interaction isn't working as expected.

When an alias has a remote destination, chasquid uses sender rewriting
(also known as SRS [1]) to forward the email without risking being in
violation of SPF policies.

See https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme for more
details.

This, however, wasn't documented anywhere, as noted in
https://github.com/albertito/chasquid/issues/6.

This patch adds a paragraph to the alias documentation explaining this
behaviour.

In the SMTP courier, when the MX lookup fails with a DNS temporary
error, we should propagate that up and cause delivery to fail with a
temporary error too.

That way the delivery can be attempted later by the queue.

However, the code today doesn't make the distinction and will treat any
temporary DNS error as a permanent delivery failure, which is a bug.

The bug was found by ludusrusso and reported in
https://github.com/albertito/chasquid/issues/4

This patch fixes the bug by propagating the error properly.

In https://github.com/albertito/chasquid/issues/4, ludusrusso comments
that the DNS lookup error handling in the SMTP courier could be improved
by using DNSError.IsNotFound.

That is true, but unfortunately it was only added in Go 1.13, and we are
currently trying to support Go 1.11 since that's what in Debian stable.

So this patch updates the TODO to note this, so that when we can use a
newer Go version, we improve this code.

This patch improves the readability and usability of the generated
coverage HTML by making the list of files be a table.

The example post-data hook was missing a quote around a sub-shell
execution.

This is harmless because the content itself is admin-provided and not
related to user input, but this commit fixes the quote for defense in
depth and consistency.

This patch replaces the upgrading notes, which focused only on
backwards-incompatible changes, with more detailed release notes.

When the DATA input is too large, we should keep on reading through it
until we reach the end marker, otherwise there is a security problem:
the remaining data will be interpreted as SMTP commands, so for example
a forwarded message that is too long might end up executing SMTP
commands under an authenticated user.

This patch implements this behaviour, while being careful not to consume
extra memory to avoid opening up the possibility of a DoS.

Note the equivalent logic for single long lines is already implemented.

This patch adds a new integration test to cover SPF checks. The main
goal is not to cover the SPF parsing, since that's handled by the
library already, but the higher level aspects: that the mails are indeed
rejected, that the DSN looks reasonable, etc.

Hook output is checked to see if it looks like a header, which includes
the possibility of multi-line headers.

This patch extends the tests to include a multi-line header, to prevent
accidental regressions.

This patch adds a test to make sure we handle correctly the case where a
hook exits with a permanent failure.

This patch adds a test to make sure we don't allow too many recipients.

Reloading during tests will cause the testing aliases to be removed,
which makes test runs that extend beyond 30s to be flaky.

This patch fixes the bug by disabling reloads during these tests.

The testing couriers are currently only used in the queue tests, but we
also want to use them in smtpsrv tests so we can make them more robusts
by checking the emails got delivered.

This patch moves the testing couriers to testlib, and makes both queue
and smtpsrv use them.

Currently, there is no limit to incoming line length, so an evil client
could cause a memory exhaustion DoS by issuing very long lines.

This patch fixes the bug by limiting the size of the lines.

To do that, we replace the textproto.Conn with a pair of buffered reader
and writer, which simplify the code and allow for better and cleaner
control.

Thanks to Max Mazurov ([email protected]) for finding and reporting
this issue.

On the smtp client package, there is no limit to the length of the
server's replies, so an evil server could cause a memory exhaustion DoS
by issuing very long lines.

This patch fixes the bug by limiting the total size of received data.
Ideally this would be done per-line instead, but gets much more complex,
so this is a compromise.

The limit chosen is 2 MiB, which should be plenty for any the total size
of server-side replies, considering we only send a single message per
connection anyway.

This is similar to 06d808c8 (smtpsrv: Limit incoming line length), which
was found and reported by Max Mazurov ([email protected]).

We want Travis CI to check against the Go version shipped in the latest
Debian stable, to make sure chasquid can be built and run there.

There was a new Debian release which has Go 1.11, so raise the CI config
version accordingly.

This patch adds a fuzz test for the smtpsrv package.

It brings up a server for test, and then fuzz the data sent over the
network.

The smtpsrv tests hard-code ports, but this patch fixes that by making it
use the new testlib.GetFreePort function.

The smtpsrv server tests allow manual override of testing ports via
flags, but the submission+TLS port was missing (accidental oversight).

Some tests require picking ports, and today resort to hard-coding,
which is brittle.

This patch adds a testlib.GetFreePort function to help pick free ports.

It is not totally race-free, but is much better than hard-coding.

This patch adds a section to the monitoring docs with an example of how
to use the prometheus-expvar-exporter to monitor chasquid with
Prometheus.

To make the coverage report a bit more accessible and easier to
navigate, this patch makes the coverage tests generate a new HTML
coverage report (in addition to the classic variant).

This patch updates the README file, adjusting some of the existing
references, as well as adding new ones (mainly pointing to new
documentation).

This patch updates the installation guide to try to make the
installation section more readable.

It also assumes a modern Go environment is installed, which
simplifies the process of building from source.

If we fail to put the message in the queue (e.g. because we're out of
storage space, or the aliases-resolve hook errored out), it should be
considered a transient failure.

Currently we return a permanent error, which is misleading, as we want
clients to retry in these situations.

So this patch changes the error returned accordingly.

This patch updates the auto-generated code to match the latest tooling
versions.

In particular, the protobufs are regenerated, and the new version no
longer supports unkeyed literals, so some minor changes are needed.

Other than that, the cipher list is extended with the latest ciphers.

This patch implements two new hooks: alias-resolve and alias-exists.

They are called during the aliases resolution process, to allow for more
complex integration with other systems, such as storing the aliases in a
database.

See the included documentation for more details.

A pipe alias must have a command, if it doesn't, we should treat the
line as bad and skip it like we do for others.

This patch does some cleanups on the tracing output for the post-data
hook, to quote the output better and more consistently.

Rspamd (https://rspamd.com/) is a popular open-source spam filtering
system.

This patch adds integration with it in the example hook, which uses the
rspamc client to get a veredict, similar to what it does for
Spamassassin.

smtp-check exits on the first error, which is not ideal when
troubleshooting, as seeing only one error can mask others, or make it
more difficult to find the underlying cause.

This patch improves how smtp-check reports errors by tweaking the
presentation a bit, as well as perform almost all checks regardless of
whether they pass or not.

This patch replaces test/README, which was becoming a bit outdated, with
a more general description of the different tests at a high level, and
includes it in the documentation index.