Skip to content

hyperkitty_import treats message content as header and adds invalid email address, causing mailman_sync to fail

I'm migrating to Debian 10 and moving our lists from mailman 2.1 to mailman 3:

Mailman Core Version	GNU Mailman 3.2.1 (La Villa Strangiato)
Mailman Core API Version	3.0
Mailman Core Python Version	3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]

While the list configurations moved over, after I started importing the archives yesterday we got a similar error, and now see the same error on our daily cron. It comes down to a mailman_sync:

Traceback (most recent call last):
  File "/usr/share/mailman3-web/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/hyperkitty/management/commands/mailman_sync.py", line 44, in handle
    sync_with_mailman(overwrite=options.get("overwrite", False))
  File "/usr/lib/python3/dist-packages/hyperkitty/lib/mailman.py", line 145, in sync_with_mailman
    sender.set_mailman_id()
  File "/usr/lib/python3/dist-packages/hyperkitty/models/sender.py", line 54, in set_mailman_id
    mm_user = client.get_user(self.address)
  File "/usr/lib/python3/dist-packages/mailmanclient/client.py", line 321, in get_user
    'users/{0}'.format(address))
  File "/usr/lib/python3/dist-packages/mailmanclient/restbase/connection.py", line 95, in call
    response, content = Http().request(url, method, data_str, headers)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1513, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1263, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1216, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.7/http/client.py", line 1326, in getresponse
    raise ResponseNotReady(self.__state)
http.client.ResponseNotReady: Request-started

I added a bit of debugging and it comes down to this. The email address being looked up is:

  Firstname Surname@Company-UK
           ^

which is obviously invalid (note the space).

The mailing list archive imported that started this off does NOT have that email address anywhere within the headers of any emails in the .mbox archive. However, the body content of a couple of emails contains quotes of this format:

From send@example.com
...standard header...

body starts
....
blah blah blah
Signature

-----Original Message-----
From: Another User [mailto:user@example.com]
Sent: 09 June 2020 12:50
To: Firstname Surname at Company-UK
Cc: original.list@example.com
Subject: topic

blah blah blah
body ends

Something in the import has decided that the -----Original Message----- quote is a header and the To: field has been modified to protect the sender so has "recreated" the email address and ended with Firstname Surname@Company-UK . Not only that, it has decided this is a valid email address. So a critical bug.

Edited by Mark Sapiro