Fix import/export database errors
Fixes protected branches errors when importing a project including them
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/21295
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/21799
-
CHANGELOG entry added - Tests
-
Added for this feature/bug -
All builds are passing
-
-
Conform by the style guides -
Branch has no merge conflicts with master
(if you do - rebase it please) -
Squashed related commits together
Merge request reports
Activity
Added 1 commit:
- a5d7bc9d - fixed rubocop warning, bumped I/E version up
Added 1 commit:
- 2bafd2b3 - fixed version checker, only complaining about export version being newer
Added 1 commit:
- 7a3554b6 - fix import integration spec
Added 203 commits:
-
7a3554b6...4d042afe - 199 commits from branch
master
- 04b88088 - added spec for protected branches issue and fix
- 3d4b6908 - fixed rubocop warning, bumped I/E version up
- 5b3d2c6e - fixed version checker, only complaining about export version being newer
- 8bb999f7 - fix import integration spec
Toggle commit list-
7a3554b6...4d042afe - 199 commits from branch
Added 1 commit:
- 21dd293c - spec trying to replicate labels issue
Added 1 commit:
- 8c0b9e26 - spec replicating issue with milestones. Also fixed version checker spec
Added 1 commit:
- 0cc667f4 - updated project.json with more milestone testing
Added 1 commit:
- af82a618 - fixed JSON for spec
Added 1 commit:
- ff37dcef - fixed issue to do with labels/milestones shared across multiple MRs and issues
Marked the task Conform by the style guides as completed
Is there a workaround or this import issue? I'm getting this on all the projects I try to import. Can I edit the JSON on the export to remove the protected_branches attribute to bypass the import errors? I don't care about losing that information.
Edited by Michael Deering@mdeering what is the exact error you are getting ?
Added 1 commit:
- 196b1756 - loose pipeline validation on importing, so it does not fail when there are missi…
Added 236 commits:
-
196b1756...b46c703b - 227 commits from branch
master
- f765a44a - added spec for protected branches issue and fix
- a9a00c1a - fixed rubocop warning, bumped I/E version up
- 1c467e05 - fixed version checker, only complaining about export version being newer
- f4807561 - spec trying to replicate labels issue
- 4213e10d - spec replicating issue with milestones. Also fixed version checker spec
- dadb2163 - updated project.json with more milestone testing
- 9c0fa9ee - fixed JSON for spec
- 1d11f266 - fixed issue to do with labels/milestones shared across multiple MRs and issues
- f946b490 - loose pipeline validation on importing, so it does not fail when there are missi…
Toggle commit list-
196b1756...b46c703b - 227 commits from branch
Added 1 commit:
- 1f4f98ad - fix import integration spec
Added 15 commits:
-
1f4f98ad...89af76ed - 5 commits from branch
master
- 5d80b049 - added spec for protected branches issue and fix
- 8b68d3b7 - fixed rubocop warning, bumped I/E version up
- a9c5b188 - fixed version checker, only complaining about export version being newer
- 49ce2255 - spec trying to replicate labels issue
- 9a7e56c7 - spec replicating issue with milestones. Also fixed version checker spec
- 32dc3e54 - updated project.json with more milestone testing
- e9d9858c - fixed JSON for spec
- d9dd5389 - fixed issue to do with labels/milestones shared across multiple MRs and issues
- 20095a52 - loose pipeline validation on importing, so it does not fail when there are missi…
- 54697531 - fix import integration spec
Toggle commit list-
1f4f98ad...89af76ed - 5 commits from branch
Added 1 commit:
- bdbc5d84 - squashed - Fix DB exceptions raised importing some specific projects.
Added 1 commit:
- eca99979 - added changelog
GitLab Community Edition 8.11.4 b871b762
Here is relevant stack trace from the logs also: https://gitlab.com/mdeering/gitlab-ce/snippets/25568
Edited by Michael DeeringAdded 32 commits:
-
eca99979...ad599eb6 - 30 commits from branch
master
- ac06c51f - squashed - Fix DB exceptions raised importing some specific projects.
- edada502 - added changelog
-
eca99979...ad599eb6 - 30 commits from branch
Added 1 commit:
- 69d1ffdb - fix changelog
@mdeering this fix will solve that issue, should definitely be in 8.12. Otherwise, you can workaround it removing the protected_branches part from the JSON, and bundle the tar.gz again...
Added 1 commit:
- 1677aae8 - trying to fix random CI build issue
Added 30 commits:
-
1677aae8...c61ebeda - 26 commits from branch
master
- 52202c1c - squashed - Fix DB exceptions raised importing some specific projects.
- ff40986b - added changelog
- 9b84c21b - fix changelog
- 0ce1ef72 - trying to fix random CI build issue
Toggle commit list-
1677aae8...c61ebeda - 26 commits from branch
Added 1 commit:
- 2fb33836 - update failing spec
Added 1 commit:
- 7276d303 - fixed tricky mySQL issue
Added 32 commits:
Toggle commit listMilestone changed to %8.12
Marked the task CHANGELOG entry added as completed
Added 1 commit:
- 12369941 - squashed - Fix DB exceptions raised importing some specific projects.
Marked the task Squashed related commits together as completed
Reassigned to @smcgivern
@smcgivern can you review this one? Thanks!!
170 unless events.blank? 171 existing_object.assign_attributes(events: events) 172 end 173 166 174 existing_object 167 175 else 168 176 relation_class.new(parsed_relation_hash) 169 177 end 170 178 end 171 179 end 180 181 def existing_object 182 @existing_object ||= 183 begin 184 finder_hash = parsed_relation_hash.slice(*FINDER_ATTRIBUTES) 185 existing_object = relation_class.find_or_create_by(finder_hash) actually, this is not right whoops. I was gonna say it should match on colour, project_id and title as those are unique, but double-checking the validation again looks like
color
is not required to be unique. I'll update!
24 24 end 25 25 26 26 def verify_version!(version) 27 if Gem::Version.new(version) > Gem::Version.new(Gitlab::ImportExport.version) 28 raise Gitlab::ImportExport::Error.new("Import version mismatch: Required <= #{Gitlab::ImportExport.version} but was #{version}") 27 if Gem::Version.new(version) != Gem::Version.new(Gitlab::ImportExport.version) Do we need to update the docs to say which versions are compatible with each other? https://gitlab.com/help/user/project/settings/import_export.md
This has changed so no version is compatible with each other, as this could lead to issues (at least, right now). I think is still useful to have this versioning and not just an integer, as it will indicate if we do a major refactor in the future, etc... At the moment we've been just fixing bugs so only the patch Z version of X.Y.Z has been increased... More useful development-wise than for compatibility, at least for now...
@jameslopez looks good, I just really have a question about versioning
Reassigned to @jameslopez
Added 28 commits:
-
12369941...f7b64a46 - 27 commits from branch
master
- 2d40b462 - squashed - Fix DB exceptions raised importing some specific projects.
-
12369941...f7b64a46 - 27 commits from branch
Added 1 commit:
- 8b69cd48 - Fixed label color issue and added Import/Export versioning table
Reassigned to @smcgivern
@smcgivern feel free to check this again :)
- Resolved by Achilleas Pipinellis
@jameslopez I'm happy, pick an endboss
Reassigned to @jameslopez
Reassigned to @axil
Thanks Sean! Assigning to @axil now, to double check the change in the docs :) Thanks!
Then please assign to @rymai for a final review :)
Edited by James LopezAdded 200 commits:
-
8b69cd48...4bdcbc85 - 198 commits from branch
master
- 65a4c832 - squashed - Fix DB exceptions raised importing some specific projects.
- eed59d0a - Fixed label color issue and added Import/Export versioning table
-
8b69cd48...4bdcbc85 - 198 commits from branch
172 end 173 166 174 existing_object 167 175 else 168 176 relation_class.new(parsed_relation_hash) 169 177 end 170 178 end 171 179 end 180 181 def existing_object 182 @existing_object ||= 183 begin 184 finder_hash = parsed_relation_hash.slice(*FINDER_ATTRIBUTES) 185 existing_object = relation_class.find_or_create_by(finder_hash) 186 # Done in two steps, as MySQL behaves differently than PostgreSQL using 187 # the +find_or_create_by+ method and does not return the ID the second time. @jameslopez Ok, fair enough! Do you think this is a Rails/AR issue and should be raised as such in their issue tracker?
@rymai actually, I didn't spend time digging into the rails code, but when I fixed the issue in https://gitlab.com/gitlab-org/gitlab-ce/commit/7276d3039d156f3fbd4abb744ef3a4e0adb4da31 - what first came to my mind was the
parsed_relation_hash
may have some timestamps such ascreated_at
- and this will differ most likely in postgreSQL / mySQL as I think they have different timestamp, and what was a match for postgresql may not match somehow in mySQL.I'm not sure this was the exact issue though, and perhaps it's worth investigating more... I'm actually curious now so I may dig into it a bit more after my holiday... I think we could leave this as it is for now though as seemed to be fine with MySQL and postgreSQL.
Before https://gitlab.com/gitlab-org/gitlab-ce/commit/7276d3039d156f3fbd4abb744ef3a4e0adb4da31 the test for two issues having the same label didn't work in mySQL, so it wasn't "finding" the record that it just created. While all spec were passing in postgreSQL....
Edited by James Lopez
Added 10 commits:
-
eed59d0a...06438c74 - 8 commits from branch
master
- 80c7d944 - squashed - Fix DB exceptions raised importing some specific projects.
- 15d7526e - Fixed label color issue and added Import/Export versioning table
-
eed59d0a...06438c74 - 8 commits from branch
Reassigned to @jameslopez
Mentioned in issue #21295 (closed)
Added 266 commits:
-
15d7526e...1d51bc7d - 264 commits from branch
master
- e74b7d66 - squashed - Fix DB exceptions raised importing some specific projects.
- 3c7feaf3 - Fixed label color issue and added Import/Export versioning table
-
15d7526e...1d51bc7d - 264 commits from branch
Added 1 commit:
- 0a3d2d7c - fix changelog
@rymai assigning this back to you - failing spec seems unrelated as it's failing in master as well...
Reassigned to @rymai
Mentioned in commit 1e72de66
Mentioned in issue #23730 (closed)