Skip to content

UnicodeEncodeError when running cms.bin.xtm_translations create

How to reproduce

  1. Use cms version 6942-add-xtm (d83b36a1)
  2. Checkout web.adblockplus.org version 8efef51
  3. Login using cms.bin.xtm_translations login
  4. Run printed export command export XTM_TOKEN=SECRET_TOKEN
  5. Run python -m cms.bin.xtm_translations create --name "web.adblockplus.org#142 - Privacy policy: Missing info on issue reporter data collection" --desc "https://gitlab.com/eyeo/websites/web.adblockplus.org/issues/142" --client-id 12449 --ref-id eyeo/websites/web.adblockplus.org#142 --workflow-id 2926 web.adblockplus.org

Observed behavior

The following exception is raised:

/usr/lib/python2.7/pkgutil.py:428: ImportWarning: Not importing directory 'cms': missing __init__.py
  __import__(pkg)
WARNING:root:The following locale is not supported: "el". Ignoring...
WARNING:root:The following locale is not supported: "vi". Ignoring...
WARNING:root:The following locale is not supported: "ar". Ignoring...
WARNING:root:The following locale is not supported: "cs". Ignoring...
WARNING:root:The following locale is not supported: "ja". Ignoring...
WARNING:root:The following locale is not supported: "he". Ignoring...
WARNING:root:The following locale is not supported: "ko". Ignoring...
WARNING:root:The following locale is not supported: "ms". Ignoring...
/home/wspee/.local/lib/python2.7/site-packages/Markdown-2.6.8-py2.7.egg/markdown/__init__.py:259: DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.extra" instead of "extra"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/wspee/1my/dev/git/cms/cms/bin/xtm_translations.py", line 20, in <module>
    main()
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/cli.py", line 158, in main
    args.func(args)
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/cli.py", line 41, in handle_projects
    args.projects_func(args, api, fs)
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/projects_handler.py", line 55, in create_project
    files_to_upload = utils.get_files_to_upload(source)
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/utils.py", line 76, in get_files_to_upload
    raw_strings = extract_strings(source)
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/utils.py", line 167, in extract_strings
    localized_string_callback=record_string)
  File "/home/wspee/1my/dev/git/cms/cms/utils.py", line 175, in process_page
    localized_string_callback)
  File "/home/wspee/1my/dev/git/cms/cms/utils.py", line 157, in get_page_params
    converted = converter()
  File "/home/wspee/1my/dev/git/cms/cms/converters.py", line 301, in __call__
    result = self.get_html(self._data, self._filename)
  File "/home/wspee/1my/dev/git/cms/cms/converters.py", line 307, in get_html
    result = self.insert_localized_strings(source, html_escapes)
  File "/home/wspee/1my/dev/git/cms/cms/converters.py", line 240, in insert_localized_strings
    flags=re.S,
  File "/usr/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/home/wspee/1my/dev/git/cms/cms/converters.py", line 226, in lookup_string
    escapes)
  File "/home/wspee/1my/dev/git/cms/cms/converters.py", line 175, in localize_string
    callback(page, locale, name, result, comment, fixed_strings)
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/utils.py", line 161, in record_string
    for i_s in enumerate(fixed_strings, 1))
  File "/home/wspee/1my/dev/git/cms/cms/translations/xtm/utils.py", line 161, in <genexpr>
    for i_s in enumerate(fixed_strings, 1))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 2: ordinal not in range(128)

Expected behavior

A new xtm project should be created as advertised in https://gitlab.com/eyeo/websites/cms/blob/6942-add-xtm/docs/usage/xml-sync.md#2-project-creation