Skip to content

Upgrading from 8.13.3 to 8.14.4 with invalid utf-8 in commit messages causes the migration to fail

I use the omnibus package, one of my repositories is a potentially bastardised linux kernel, one of the names has invalid utf-8 in a commit message.

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run

    [execute] rake aborted!
              StandardError: An error has occurred, all later migrations canceled:
              
              source sequence is illegal/malformed utf-8/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20161124141322_migrate_process_commit_worker_jobs.rb:62:in `block in up'
              /opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/sidekiq-4.2.1/lib/sidekiq.rb:94:in `block in redis'

So it's failing at JSON.dumps(payload), I know nothing about ruby, so forgive the copy/paste solution from another issue, but I tried cleaning it up:

        hash = {
          id: commit.oid,
          message: commit.message.encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"),
          parent_ids: commit.parent_ids,
          authored_date: commit.author[:time],
          author_name: commit.author[:name].encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"),
          author_email: commit.author[:email].encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"),
          committed_date: commit.committer[:time],
          committer_email: commit.committer[:email].encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?"),
          committer_name: commit.committer[:name].encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "?")
        }

That takes a very long time to run, I suppose the method of cleaning strings is poor. After that I got a new error:

wrong argument type Hash (expected String)/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/rugged-0.24.0/lib/rugged/repository.rb:99:in `lookup'

Edit: Running reconfigure again after stopping/starting gitlab got past the 'Hash' error.

I'm not sure how accurate this is, my installation was/is in a partially upgraded state.

dev@dev:~$ sudo gitlab-rake gitlab:env:info RAILS_ENV=production


System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:1.13.6
Rake Version:   10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:        8.14.4
Revision:       3ea6c8d
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://dev.touchstar.co.uk
HTTP Clone URL: http://dev.touchstar.co.uk/some-group/some-project.git
SSH Clone URL:  git@dev.touchstar.co.uk:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        4.0.3
Repository storage paths:
- default:      /home/git/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git