Messages from prod.outlook.com (with no DKIM header) are always shunted (due to malformed authentication-result headers)
I have been working to upgrade from Mailman2 to Mailman3, and have hit a blocking issue while going into production. If I send any message from our corporate onmicrosoft.com account, it gets shunted. Digging into this, it's due to a malformed authentication-results header. I can generate emails which trigger the fault at will. Sample malformed header received when I email myself directly:
authentication-results: groups.carrubbers.org; dkim=none (message not signed)
header.d=none;groups.carrubbers.org; dmarc=none action=none
header.from=CarrubbersCC.onmicrosoft.com;
If these emails reach MailMan3, it causes the email to be shunted.
Apr 25 22:26:28 2021 (1801) ACCEPT: <ef9b047c-105b-54ef-745d-a3d661c0d495@carrubbers.org>
Apr 25 22:26:31 2021 (1798) HyperKitty archived message <ef9b047c-105b-54ef-745d-a3d661c0d495@carrubbers.org> to https://mailman.planningsuite.com/hyperkitty/list/mailmantest@groups.carrubbers.org/message/M47X5UF7ZDZK7B5NKBQQ4S7DBYIDRHYE/
Apr 25 22:27:13 2021 (1805) Uncaught runner exception: Syntax error: Expected "=" at: ; dmarc=none action=none
header.from=C...
Apr 25 22:27:13 2021 (1805) Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/mailman/core/runner.py", line 173, in _one_iteration
self._process_one_file(msg, msgdata)
File "/usr/local/lib/python3.6/dist-packages/mailman/core/runner.py", line 266, in _process_one_file
keepqueued = self._dispose(mlist, msg, msgdata)
File "/usr/local/lib/python3.6/dist-packages/mailman/runners/pipeline.py", line 37, in _dispose
process(mlist, msg, msgdata, pipeline)
File "/usr/local/lib/python3.6/dist-packages/mailman/core/pipelines.py", line 50, in process
handler.process(mlist, msg, msgdata)
File "/usr/local/lib/python3.6/dist-packages/mailman/handlers/validate_authenticity.py", line 118, in process
authenticate(msg, msgdata)
File "/usr/local/lib/python3.6/dist-packages/mailman/utilities/retry.py", line 44, in f_retry
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/mailman/handlers/validate_authenticity.py", line 85, in authenticate
prev = trusted_auth_res(msg)
File "/usr/local/lib/python3.6/dist-packages/mailman/handlers/validate_authenticity.py", line 73, in trusted_auth_res
authserv_id = AuthenticationResultsHeader.parse(header).authserv_id
File "/usr/local/lib/python3.6/dist-packages/authres/__init__.py", line 206, in parse
return authres.core.AuthenticationResultsHeader.parse(core_features(), string)
File "/usr/local/lib/python3.6/dist-packages/authres/core.py", line 442, in parse
return self.parse_value(feature_context, string)
File "/usr/local/lib/python3.6/dist-packages/authres/core.py", line 454, in parse_value
header._parse()
File "/usr/local/lib/python3.6/dist-packages/authres/core.py", line 517, in _parse
result = self._parse_resinfo()
File "/usr/local/lib/python3.6/dist-packages/authres/core.py", line 550, in _parse_resinfo
method, version, result = self._parse_methodspec()
File "/usr/local/lib/python3.6/dist-packages/authres/core.py", line 573, in _parse_methodspec
raise SyntaxError('Expected "="', self._parse_text)
authres.core.SyntaxError: Syntax error: Expected "=" at: ; dmarc=none action=none
header.from=C...
Apr 25 22:27:13 2021 (1805) SHUNTING: 1619389633.6900759+c5e9c7e7905cbd46fd3b59b716d3940c893576e6
I appreciate that the best possible fix here is for Microsoft to fix their bug, but that seems to be wishful thinking. A report from https://answers.microsoft.com/en-us/outlook_com/forum/all/malformed-authentication-results-header-in-some/4e6d4d8d-2721-4953-b320-a5b57ef1c7c5 in 2019 suggests it's been a fault for long enough that I propose MailMan needs to not shunt these messages. But with that said I'm confused why MailMan3 hasn't encountered/fixed this already so would welcome someone telling me that I'm doing something silly.
I also wondered why the messages don't have DKIM headers, but don't think MailMan3 should be shunting inbound messages which have no DKIM header: at worst it should be bouncing them, at best accepting them.
My reading of the RFC is that the header is malformed for at least three reasons:
- It ends in a ";", but the BNF says ";" is part of the resinfo, which means it must be followed by a methodspec. So a valid authentication-results header must not end in a ";"
- The header should be "Authentication-Results:" rather than "authentication-results:"
- the middle resinfo token is "; groups.carrubbers.org". But the BNF says that the ";" must follow a methodspec, and this must take the form "method=result". I think it's actually this point which is throwing the exception.
I had been using 3.3.1, but reproduced it today using 3.3.4. It shunts the message. A mailman unshunt re-produces the same error. The actual content of the subject/body don't seem to matter at all. Very happy to send you/your list a sample message if you need one.
I'm now going to embark on patching the Python code myself. I think the approach should be to trap exceptions failing to parse Authentication-Result headers, and proceed as though the header wasn't there. Haven't contributed a patch before, so my just post the patch inline here should I end up with something workable/usable.
Nigel