Skip to content

Non-ASCII characters in filenames are not supported

Filenames with non ASCII characters currently lead to python errors, e.g. creating fö/bar.bag in a scranroot leads to

$ marv scan
2018-09-06 13:51:59,298 INFO marv.collection.xyz added <Dataset e6yofhkj67bojx2trtcpqry4xa bar>
Traceback (most recent call last):
  File "/home/marv/marv-18-07/venv/bin/marv", line 11, in <module>
    load_entry_point('marv-cli', 'console_scripts', 'marv')()
  File "/home/marv/marv-18-07/marv-robotics/code/marv-cli/marv_cli/__init__.py", line 118, in cli
    marv(auto_envvar_prefix='MARV')
  File "/home/marv/marv-18-07/venv/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/marv/marv-18-07/venv/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/marv/marv-18-07/venv/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/marv/marv-18-07/venv/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/marv/marv-18-07/venv/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/cli.py", line 471, in marvcli_scan
    create_app().site.scan(dry_run)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/site.py", line 406, in scan
    collection.scan(scanroot, dry_run)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/collection.py", line 497, in scan
    self._add_batch(log, batch)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/collection.py", line 571, in _add_batch
    row, fields, relfields = self.render_listing(dataset)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/collection.py", line 684, in render_listing
    value = transform(value)
  File "/home/marv/marv-18-07/marv-robotics/code/marv/marv/collection.py", line 39, in <lambda>
    'string[]': lambda lst: [unicode(x) for x in lst or []],
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)