hyperkitty_import treats message content as header and adds invalid email address, causing mailman_sync to fail
I'm migrating to Debian 10 and moving our lists from mailman 2.1 to mailman 3:
Mailman Core Version GNU Mailman 3.2.1 (La Villa Strangiato)
Mailman Core API Version 3.0
Mailman Core Python Version 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]
While the list configurations moved over, after I started importing the archives yesterday we got a similar error, and now see the same error on our daily cron. It comes down to a mailman_sync:
Traceback (most recent call last):
File "/usr/share/mailman3-web/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/lib/python3/dist-packages/hyperkitty/management/commands/mailman_sync.py", line 44, in handle
sync_with_mailman(overwrite=options.get("overwrite", False))
File "/usr/lib/python3/dist-packages/hyperkitty/lib/mailman.py", line 145, in sync_with_mailman
sender.set_mailman_id()
File "/usr/lib/python3/dist-packages/hyperkitty/models/sender.py", line 54, in set_mailman_id
mm_user = client.get_user(self.address)
File "/usr/lib/python3/dist-packages/mailmanclient/client.py", line 321, in get_user
'users/{0}'.format(address))
File "/usr/lib/python3/dist-packages/mailmanclient/restbase/connection.py", line 95, in call
response, content = Http().request(url, method, data_str, headers)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1513, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1263, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1216, in _conn_request
response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1326, in getresponse
raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-started
I added a bit of debugging and it comes down to this. The email address being looked up is:
Firstname Surname@Company-UK
^
which is obviously invalid (note the space).
The mailing list archive imported that started this off does NOT have that email address anywhere within the headers of any emails in the .mbox archive. However, the body content of a couple of emails contains quotes of this format:
From send@example.com
...standard header...
body starts
....
blah blah blah
Signature
-----Original Message-----
From: Another User [mailto:user@example.com]
Sent: 09 June 2020 12:50
To: Firstname Surname at Company-UK
Cc: original.list@example.com
Subject: topic
blah blah blah
body ends
Something in the import has decided that the -----Original Message----- quote is a header and the To: field has been modified to protect the sender so has "recreated" the email address and ended with Firstname Surname@Company-UK
. Not only that, it has decided this is a valid email address. So a critical bug.