Skip to content
GitLab
Next
Closed
Issue created by Yorick Peterse@yorickpeterseContributor

Final changes for a single codebase for CE and EE

We are getting very close to having a single codebase for both CE and EE, with only a few tasks remaining:

  1. QA images have to be updated so they include lib/gitlab.rb, config/initializers/0_inject_enterprise_edition_module.rb, and ActiveSupport
  2. https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31456 needs to be restored
  3. https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14942 needs to be merged
  4. gitlab-qa#400 (closed) needs to be solved
  5. We need to do a final check of all the differences, just in case new ones have been introduced without this being noticed https://gitlab.com/gitlab-org/gitlab-ee/issues/6680
  6. The pipelines for https://gitlab.com/gitlab-org/gitlab-foss need to pass, which may require some additional changes in CE and EE
  7. We update GDK to support the new setup, if any changes are necessary at all
  8. Resolve differences at top-level files (e.g. Gemfile)

Plan

On August 9, 2019 we (@yorickpeterse, @rymai, @rspeicher, and @marin) had a call on the steps we need to take to move to a single codebase, and in particular about how to move the projects around. In this call we decided to take the following approach:

GitLab EE becomes the canonical repository, and GitLab CE will become the FOSS only repository. GitLab EE will be renamed to just "GitLab", and the URL would be renamed to gitlab-org/gitlab. GitLab CE will be renamed to GitLab FOSS, and the URL will be renamed to gitlab-org/gitlab-foss.

Existing EE issues will have a certain EE label applied to indicate that they cover an EE specific feature.

All issues from CE will be moved to EE by a bot. This bot will also post a comment on every issue explaining why the issue is being moved. All CE MRs will be closed, again with a comment explaining why.

Rationale

By moving development into EE (instead of CE or a separate project) we retain Git log data. The amount of tooling changes necessary also remains minimal. Finally, CE already is FOSS only, and this approach ensures that it does not suddenly contain EE code (with the FOSS-only code being in a different repository).

Having to move a large number of CE issues over is unfortunate, but this process can be automated. GitLab also takes care of redirecting users, so the impact should be minimal.

Finally, this approach does not require any deployment changes for GitLab.com, and can be performed incrementally. This gives us more time, instead of having to rush all of the work in the next two weeks.

Timeline

August 26 - August 30

  1. Communicate internally and externally what our plan is for moving development and issues around.
  2. Write a script that applies an EE label to existing EE issues: https://gitlab.com/snippets/1884816
  3. Close CE merge requests that are at least one year old, as a start
  4. Move CE issues that are at least one year old, as a start
  5. Notify CE merge requests (via a comment) that they will be closed after September 6
  6. Announce on the team call
  7. Announce on the engineering week in review

September 9 - September 13

  1. Write a script that automatically closes any newly created CE issues, explaining the author why => gitlab-org/quality/triage-ops#268 (closed) (Quality)
    • We can not disable the issues tracker, as this will break redirects.
  2. Update the default issue and MR template to inform users not to create MRs and issues in the CE project
  3. Move all remaining CE issues into EE
  4. Close CE MRs with no activity in 3 months
  5. Notify remaining CE MRs that they will be closed in the near future
  6. Update documentation about EE being the canonical repository
  7. Extend the merge train so that it can sync stable branches to the FOSS repository, starting with a minimal version
    • The FOSS merge can be triggered using a CI trigger. In GitLab we will set up a CI job that runs when pushing to a stable branch, and this job will trigger a run of the merge train.
    • gitlab-com/gl-infra/delivery#468 (closed)
  8. Build Omnibus package from the current temporary gitlab-foss repository (that we will remove in favour of renaming gitlab-ee)

We do not yet rename the projects to reduce the stress of deploying with these new names during release week.

September 16 - September 20

  1. Test the renaming of projects on staging.gitlab.com
  2. Remove container repositories from GitLab CE
    • Paste the rename code below into a Rails console
    • Run rename_project(gitlab_ce, user, 'gitlab-foss') in a Rails console
  3. Remove container repositories from GitLab EE
    • Paste the rename code below into a Rails console
    • Run rename_project(gitlab_ee, user, 'gitlab') in a Rails console
  4. Rename the GitLab EE project to just "GitLab", and change the URL from gitlab-org/gitlab-ee to gitlab-org/gitlab
  5. Rename the GitLab CE project to "GitLab FOSS", change the URL from gitlab-org/gitlab-ce to gitlab-org/gitlab-foss, and update the project description to clarify the new purpose of the repository
  6. Merge merge-train!26 (merged)
  7. Test automatic moving of issues
  8. Sync gitlab-org/gitlab to gitlab-org/gitlab-foss manually using merge train
  9. Update chatops to use the new project names: gitlab-com/chatops!77 (merged)
  10. Update release tools to use the new project names: release-tools!712 (merged)
  11. Update the deployer to use the new project names
  12. Update https://gitlab.com/gitlab-org/quality/triage-ops to use the new project names: gitlab-org/quality/triage-ops!300 (merged)
  13. Update https://gitlab.com/gitlab-org/gitlab-insights to use the new project names: gitlab-insights!161 (merged)
  14. Update Periscope to use the new project names (if needed: not sure because the graphs are probably built based on projects ID)
  15. Communicate both internally and externally that these projects have been renamed
  16. Handle any systems that are not able to handle any redirects GitLab may produce (e.g. we may have to change some monitoring systems)
  17. Update Omnibus to use all these new URLs omnibus-gitlab!3513 (merged)
  18. Update gitlab-docs to use all these new URLs gitlab-docs#417 (closed)
  19. Update https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/generators/releases.rb and https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/generators/direction.rb to point to gitlab-org/gitlab only: gitlab-com/www-gitlab-com@6bb96aef
  20. Update contribution docs to cover community contribution workflows

Rename script:

module Projects
  class UpdateService < BaseService
    def validate!
      unless valid_visibility_level_change?(project, params[:visibility_level])
        raise ValidationError.new(s_('UpdateProject|New visibility level not allowed!'))
      end

      if changing_default_branch?
        raise ValidationError.new(s_("UpdateProject|Could not set the default branch")) unless project.change_head(params[:default_branch])
      end
    end
  end
end

module Projects
  class AfterRenameService
    def first_ensure_no_registry_tags_are_present
      # noop
    end
  end
end

gitlab_ce = Project.find_by_full_path('gitlab-org/gitlab-ce')
gitlab_ee = Project.find_by_full_path('gitlab-org/gitlab-ee')
user = User.find_by_username('yorickpeterse')

def rename_project(project, user, path)
  project.container_repositories.destroy_all
  Projects::UpdateService.new(project, user, path: path).execute
end

Security workflow for a single codebase

Until December 2019, the workflow is mostly the same. Developers submit their security MRs to the security equivalents of both gitlab-org/gitlab-foss and gitlab-org/gitlab. If they are submitting an MR for a release based on the single codebase, they only submit their MR to gitlab-org/gitlab, and the Merge Train will sync their changes to the gitlab-org/gitlab-foss.

Should we manage to move the workflow to .com before December, the process might change a bit.

Summary

Edited by Yorick Peterse