UnicodeEncodeError in mailman/rest/post_moderation.py
This seems to be related to #437 (closed) and #256 (closed) and !197 (closed) => exception in mm3 and it renders postorius moderation websites for the whole list unusable.
MM3 is running on the current master HEAD.
Dec 12 10:11:47 2017 (1564) REST request handler error:
Traceback (most recent call last):
File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/database/transaction.py", line 50, in wrapper
rtn = function(*args, **kws)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/wsgiapp.py", line 214, in __call__
return super().__call__(environ, start_response)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/falcon-1.2.0-py3.5.egg/falcon/api.py", line 227, in __call__
responder(req, resp, **params)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/post_moderation.py", line 167, in on_get
resource = self._make_collection(request)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/helpers.py", line 159, in _make_collection
for resource in collection]
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/helpers.py", line 159, in <listcomp>
for resource in collection]
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/post_moderation.py", line 157, in _resource_as_dict
resource = self._make_resource(request.id)
File "/opt/mailman/virtualenv/lib/python3.5/site-packages/mailman-3.1.0b5-py3.5.egg/mailman/rest/post_moderation.py", line 78, in _make_resource
resource['msg'] = msg.as_string()
File "/usr/lib/python3.5/email/message.py", line 159, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
self._write(msg)
File "/usr/lib/python3.5/email/generator.py", line 181, in _write
self._dispatch(msg)
File "/usr/lib/python3.5/email/generator.py", line 214, in _dispatch
meth(msg)
File "/usr/lib/python3.5/email/generator.py", line 272, in _handle_multipart
g.flatten(part, unixfrom=False, linesep=self._NL)
File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
self._write(msg)
File "/usr/lib/python3.5/email/generator.py", line 181, in _write
self._dispatch(msg)
File "/usr/lib/python3.5/email/generator.py", line 214, in _dispatch
meth(msg)
File "/usr/lib/python3.5/email/generator.py", line 243, in _handle_text
msg.set_payload(payload, charset)
File "/usr/lib/python3.5/email/message.py", line 316, in set_payload
payload = payload.encode(charset.output_charset)
UnicodeEncodeError: 'ascii' codec can't encode character '\ufffd' in position 90: ordinal not in range(128)
I locally fixed it according to the following diff:
diff --git a/src/mailman/rest/post_moderation.py b/src/mailman/rest/post_moderation.py
index b972bed..ff4bfcc 100644
--- a/src/mailman/rest/post_moderation.py
+++ b/src/mailman/rest/post_moderation.py
@@ -82,6 +82,12 @@ class _HeldMessageBase(_ModerationBase):
#
# See http://bugs.python.org/issue27321 and GL#256
resource['msg'] = 'This message is defective'
+ except UnicodeEncodeError:
+ # If the message can't be parsed, return a generic message instead
+ # of raising an error.
+ #
+ # See http://bugs.python.org/issue27321 and GL#256
+ resource['msg'] = 'This message is defective'
# Some of the _mod_* keys we want to rename and place into the JSON
# resource. Others we can drop. Since we're mutating the dictionary,
# we need to make a copy of the keys. When you port this to Python 3,
Edited by Mark Sapiro