mysql-postgresql-converter fails on converting
Summary
I have a new web server and therefore want to switch from the 'install from source' to the 'omnibus' installation.
On my old server I have installed gitlab from source in version 10.0.1 and on the new one the omnibus package in 10.0.1.
On the old server I followed the instructions from here (Option 2) until the conversion of the mysql database to postgresql (see here step # Convert gitlabhq_production.mysql) where I got the error
# sudo -u git -H python mysql-postgresql-converter/db_converter.py gitlabhq_production.mysql db/database.sql Line 1688 (of 3554: 47.50%) [46 tables] [21 inserts] [ETA: 0 min 0 sec]Traceback (most recent call last): File "mysql-postgresql-converter/db_converter.py", line 256, in <module> parse(sys.argv[1], sys.argv[2]) File "mysql-postgresql-converter/db_converter.py", line 73, in parse line = line.decode("utf8").strip().replace(r"\\", "WUBWUBREALSLASHWUB").replace(r"\0", "").replace(r"\'", "''").replace("WUBWUBREALSLASHWUB", r"\\") File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True)
The line 1688 contains the content of the merge_request_diff_commits and the data that is not parsable is the sha-column content that is stored as varbinary.
If I remove this line the same error occurs on line 2718, where the content of the push_event_payloads table is stored and the two columns comment_from and comment_to are stored as blob.
Steps to reproduce
Follow the instructions from
here (Option 2) until the conversion of the mysql database to postgresql
to here step # Convert gitlabhq_production.mysql.
What is the current bug behavior?
The mysql-postgresql-converter/db_converter.py script fails on converting varbinary and blob data.
What is the expected correct behavior?
The varbinary and blob data should be parsed correctly to the corresponding postgresql counterparts OR some steps should be provided to convert the mysql types into something that can be parsed by the script.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 14.04 Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.5.2 Bundler Version:1.14.6 Rake Version: 12.0.0 Redis Version: 2.8.4 Git Version: 2.14.1 Sidekiq Version:5.0.4 Go Version: go1.8.3 linux/amd64GitLab information Version: 10.0.1 Revision: 2417795bf3 Directory: /var/www/gitlab/gitlab DB Adapter: mysql2 URL: https://gitlab.mydomain.de HTTP Clone URL: https://gitlab.mydomain.de/some-group/some-project.git SSH Clone URL: git@gitlab.mydomain.de:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.9.0 Repository storage paths:
- default: /var/www/gitlab/repositories Hooks: /var/www/gitlab/gitlab-shell/hooks Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.9.0 ? ... OK (5.9.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/1 ... ok 1/2 ... ok 3/3 ... ok 4/4 ... ok 6/5 ... ok 6/6 ... ok 7/7 ... ok 7/8 ... ok 4/9 ... ok 3/10 ... ok 4/11 ... ok 4/12 ... ok 4/13 ... ok 4/14 ... ok 4/16 ... ok 8/17 ... ok 8/18 ... ok 2/23 ... repository is empty 2/25 ... ok 3/31 ... ok 8/32 ... ok 14/38 ... ok 14/39 ... repository is empty 4/40 ... ok 8/42 ... ok 15/43 ... ok 2/45 ... ok 15/46 ... ok 15/48 ... ok 15/49 ... ok 15/50 ... ok 15/51 ... ok 15/52 ... ok 8/53 ... ok 7/55 ... ok 8/57 ... ok 2/58 ... repository is empty 8/62 ... ok 8/63 ... ok 15/64 ... ok 17/65 ... ok 17/66 ... ok 20/67 ... ok 20/68 ... ok 20/69 ... ok 15/70 ... ok 22/74 ... ok 2/79 ... ok 8/80 ... ok 8/81 ... ok 15/82 ... ok 8/83 ... ok 2/84 ... repository is empty 2/85 ... repository is empty 2/86 ... repository is empty 2/87 ... repository is empty 2/89 ... repository is empty 2/90 ... repository is empty 2/91 ... ok 15/92 ... ok 15/93 ... ok 15/94 ... ok 15/95 ... ok 26/96 ... ok 8/97 ... ok 2/98 ... ok 17/99 ... ok 2/100 ... ok 2/101 ... ok 17/102 ... ok 15/105 ... ok 15/106 ... ok 3/109 ... ok 3/110 ... ok 3/111 ... ok 3/112 ... ok 3/113 ... ok 17/114 ... ok 15/115 ... ok 15/116 ... ok 3/117 ... ok 2/118 ... repository is empty 15/119 ... ok 17/120 ... ok 15/121 ... ok 17/122 ... ok 15/123 ... ok 8/124 ... ok 3/125 ... ok 31/126 ... ok 15/127 ... repository is empty 17/128 ... ok 15/129 ... ok 17/130 ... ok 15/132 ... ok 15/133 ... ok 15/134 ... ok 15/135 ... ok Running /var/www/gitlab/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/www/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... no Try fixing it: Re-download the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Projects have namespace: ... 5/1 ... yes 1/2 ... yes 3/3 ... yes 4/4 ... yes 6/5 ... yes 6/6 ... yes 7/7 ... yes 7/8 ... yes 4/9 ... yes 3/10 ... yes 4/11 ... yes 4/12 ... yes 4/13 ... yes 4/14 ... yes 4/16 ... yes 8/17 ... yes 8/18 ... yes 2/23 ... yes 2/25 ... yes 3/31 ... yes 8/32 ... yes 14/38 ... yes 14/39 ... yes 4/40 ... yes 8/42 ... yes 15/43 ... yes 2/45 ... yes 15/46 ... yes 15/48 ... yes 15/49 ... yes 15/50 ... yes 15/51 ... yes 15/52 ... yes 8/53 ... yes 7/55 ... yes 8/57 ... yes 2/58 ... yes 8/62 ... yes 8/63 ... yes 15/64 ... yes 17/65 ... yes 17/66 ... yes 20/67 ... yes 20/68 ... yes 20/69 ... yes 15/70 ... yes 22/74 ... yes 2/79 ... yes 8/80 ... yes 8/81 ... yes 15/82 ... yes 8/83 ... yes 2/84 ... yes 2/85 ... yes 2/86 ... yes 2/87 ... yes 2/89 ... yes 2/90 ... yes 2/91 ... yes 15/92 ... yes 15/93 ... yes 15/94 ... yes 15/95 ... yes 26/96 ... yes 8/97 ... yes 2/98 ... yes 17/99 ... yes 2/100 ... yes 2/101 ... yes 17/102 ... yes 15/105 ... yes 15/106 ... yes 3/109 ... yes 3/110 ... yes 3/111 ... yes 3/112 ... yes 3/113 ... yes 17/114 ... yes 15/115 ... yes 15/116 ... yes 3/117 ... yes 2/118 ... yes 15/119 ... yes 17/120 ... yes 15/121 ... yes 17/122 ... yes 15/123 ... yes 8/124 ... yes 3/125 ... yes 31/126 ... yes 15/127 ... yes 17/128 ... yes 15/129 ... yes 17/130 ... yes 15/132 ... yes 15/133 ... yes 15/134 ... yes 15/135 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... Exception: undefined method
run_command' for SystemCheck::App::RubyVersionCheck:Class Did you mean? run_commands Git version >= 2.7.3 ? ... Exception: undefined methodrun_command' for SystemCheck::App::GitVersionCheck:Class Did you mean? run_commands Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1506286259 sudo mv /var/www/gitlab/.ssh/authorized_keys.lock ~/gitlab-check-backup-1506286259 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 20Checking GitLab ... Finished