Skip to content

Import fails with `incompatible character encodings: ASCII-8BIT and UTF-8`

Summary

Importing a GitLab project export that includes some non-UTF8 characters leads to an error incompatible character encodings: ASCII-8BIT and UTF-8. This occurred on 11.2.3. The project the customer is imported is from users in Asia-Pacific, many of which are Korean so it's likely that Korean characters are an issue. Unfortunately we don't know exactly which character. We can do further debugging with the customer and potentially get a copy of their export if we need it.

IIRC, we have an issue somewhere that bad issues/MRs from an import shouldn't fail the entire import, but should be logged for review later. That would certainly help here, although hopefully we can account for this type of situation and handle it gracefully.

@jameslopez Any thoughts?

Logs:

==> /var/log/gitlab/gitlab-rails/production.log <==
Import/Export error raised on /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_export/project_tree_restorer.rb:33:in `rescue in restore': incompatible character encodings: ASCII-8BIT and UTF-8
Import/Export backtrace: /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer/common_mark/html.rb:12:in `block in code_block'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:74:in `block'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer/common_mark/html.rb:6:in `code_block'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:40:in `render'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer/html_renderer.rb:4:in `render'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:24:in `block in out'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:18:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:18:in `out'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:20:in `block (2 levels) in out'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/node.rb:60:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:20:in `block in out'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:18:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:18:in `out'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:49:in `document'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer/html_renderer.rb:8:in `document'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer.rb:34:in `render'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/commonmarker-0.17.8/lib/commonmarker/renderer/html_renderer.rb:4:in `render'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/markdown_engines/common_mark.rb:40:in `render'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/markdown_filter.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline/filter.rb:127:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:131:in `block in perform_filter'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:169:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:130:in `perform_filter'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:116:in `block (2 levels) in call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `inject'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:115:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:169:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.4/lib/html/pipeline.rb:113:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/pipeline/base_pipeline.rb:21:in `block (2 levels) in singleton class'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:108:in `render_result'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in render_result'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:34:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `render_result'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:139:in `block in cacheless_render'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:103:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:138:in `cacheless_render'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in cacheless_render'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:34:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `cacheless_render'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:50:in `cacheless_render_field'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `block in cacheless_render_field'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:34:in `measure'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:159:in `cacheless_render_field'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:85:in `block in refresh_markdown_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:82:in `map'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cache_markdown_field.rb:82:in `refresh_markdown_cache'