This part of [Scaling the Security Release process on GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/168) Following definitions are used throughout this epic, be sure to familiarize yourself with them before continue reading: * The [GitLab security terminology](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/process.md#terminology) * Projects terminology: * Canonical: `gitlab.com/gitlab-org/gitlab` * Security: `gitlab.com/gitlab-org/security/gitlab` * Build: `dev.gitlab.org/gitlab/gitlab-ee` ## Problem statement Currently, when performing a critical or regular [security selease](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/process.md) auto-deploy processes are paused. This implies that **no deployments to GitLab.com** are executed until the Security Release is finished. The purpose of this epic is to find and implement a way for the auto-deploy and Security Release processes to be free-flowing: One shouldn't block the other. ## DRI's * `@mayra-cabrera` * `@rspeicher` ## Exit criteria Auto-deploy processes should continue flowing while performing Security Releases. Particularly, when a Security Release is performed: * Auto-deploy branches on GitLab.com will continue to be created at the same pace. * Continous deployment on GitLab.com will continue rolling at the same pace. * GitLab.com should be protected 24/7 by including unpublished security fixes into the continuous deployment. * Ensuring that the security fixes are only available to the public when the Security Release is published and not leaked before. ## Proposal: Include security fixes as part of the auto-deploy Merge security merge requests targeting `master` once they're ready to be merged and deployed them to our different environments along with regular fixes. To achieve this, we'd need to modify: * Our infrastructure setup between Canonical, Security and Build, the auto-deploy process and the security release process ### Modifications to the infrastructure between repositories Currently, our Infrastructure between Canonical, Security, and Build is set to: * Protected branches from `Canonical` are synced to `Security` via push mirror * Branches from `Security` are synced to `Build` via push mirror. ```mermaid graph LR classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px classDef security fill:#e086bd,stroke:#333,stroke-width:1px subgraph Canonical c-master(master) c-auto-deploy(12-8-auto-deploy-20200120) c-stable(12-8-stable) c-feature(feature/some-new-feature) end subgraph Security s-master(master) s-auto-deploy(12-8-auto-deploy-20200120) s-stable(12-8-stable) end subgraph Build b-master(master) b-auto-deploy(12-8-auto-deploy-20200120) b-stable(12-8-stable) end c-master -->|push mirror| s-master s-master -->|push mirror| b-master c-stable -->|push mirror| s-stable s-stable -->|push mirror| b-stable c-auto-deploy -->|push mirror | s-auto-deploy s-auto-deploy -->|push mirror | b-auto-deploy class c-master,c-stable,c-auto-deploy canonical class b-master,b-stable,b-auto-deploy canonical class s-master,s-stable,s-auto-deploy security ``` To achieve a free-flowing process between Security Releases and Auto-deploy processes, our infrastructure needs to change to: * Security `master` is updated two different ways: * When it doesn't diverge, through push mirroring * When diverge, through merge-train. * Auto-deploy branches only exist on Security * [SAME] protected branches from Canonical are synced to Security via push mirror. * [SAME] branches from Security are synched to `Build` via push mirror ```mermaid graph LR classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px classDef security fill:#e086bd,stroke:#333,stroke-width:1px subgraph Canonical c-master(master) c-stable(13-2-stable) c-feature(feature/some-new-feature) end subgraph Security s-master(master) s-auto-deploy(13-2-auto-deploy-20200710) s-stable(13-2-stable) end subgraph Build b-master(master) b-auto-deploy(13-2-auto-deploy-20200710) b-stable(13-2-stable) end c-master -->|merge| s-master s-master -->|push mirror| b-master c-stable -->|push mirror| s-stable s-stable -->|push mirror| b-stable s-auto-deploy -->|push mirror | b-auto-deploy class c-master,c-stable,c-auto-deploy canonical class b-master,b-stable,b-auto-deploy canonical class s-master,s-stable,s-auto-deploy security ``` ### Modifications to the auto-deploy processes Auto-deploy processes should be performed on Security only. This will allow us to: * Deploy security fixes to our different environments * Prevent leaking security fixes that are not ready to be published ### Modifications to the Security Release process **Outside of the Security Release process** 1. When security merge requests targeting `master`, associated to the Security Release Tracking Issue, are ready they're merged and deploy to our different environments along with regular fixes **During the Security Release process** 1. Backports are merged. 1. Packages and blog posts are prepared as usual 1. Branches between Canonical and Security are synced. ## Boards **Completed** | Topic | Status | | ------ | ------ | | [Validate merge requests before they're merged](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/839) | :white_check_mark: | | [Analyze time-to-merge security merge requests targeting `master`](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/989) | :white_check_mark: | | [Experiment: Early merge of security merge requests targeting `master`](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1018) | :white_check_mark: | | [Change the way security implementation issues are closed](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1106) | :white_check_mark: | | [Create auto-deploy branches on Security](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1026) | :white_check_mark: | | [Deal with a main branch divergence between Canonical and Security mirrors](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/298) | :white_check_mark: | | [Allow for security merge requests to trigger Pipelines for Merged Results](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1107) | :white_check_mark: | | [Modify the way release-tools processes security merge requests](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1122) | :white_check_mark: | ## References * [Security Releases + Auto deploy: Playlist](https://www.youtube.com/playlist?list=PL05JrBw4t0KpSknwk7nCud5Qa9xK4eY6S) ## Follow ups - [Security Release Technical debt](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/305) - [Create security releases on demand](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/961) Original proposals: <details> <summary>Former proposal 1</summary> ## Goals * Developer must be able to do regular workflow on security (merging merge request to master once approved by maintainers) * Security master needs to be deployed continuously on GitLab.com, so we can have GitLab.com protected 24/7 * We should only publish security commits until they are included in a Security Release * Publishing security commits => merging them into Canonical * Because both Canonical branches and Security branches (prefixed with `security/`) are synced to Build, we can perform a regular or security release at any time; one does not block the other. ## Process ### Setup * Specific protected branches from Canonical (`master`, `X-Y-stable`, `X-Y-auto-deploy-YYYYMMDD`) should be synced to Security via pull mirror * Prefixing all branches with security/ (e.g., `master` on Canonical becomes `security/master` on Security). * Security pushes `security/` branches to Build ### Workflow ```mermaid graph LR classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px classDef security fill:#e086bd,stroke:#333,stroke-width:1px subgraph Canonical c-master(master) c-stable(12-3-stable) c-feature(feature/some-new-feature) c-bug(15015-some-bug-fix) end subgraph Security s-master(security/master) s-stable(security/12-3-stable) s-auto-deploy(security/12-3-auto-deploy-20190911) end subgraph Build b-master(master) b-stable(12-3-stable) bs-stable(security/12-3-stable) bs-auto-deploy(security/12-3-auto-deploy-20190911) end subgraph release-tools rt-auto-deploy[auto_deploy:prepare] end c-master -.->|pull mirror with prefix| s-master c-master -->|push mirror| b-master c-stable -.->|pull mirror with prefix| s-stable c-stable -->|push mirror| b-stable s-stable -->|push mirror| bs-stable s-auto-deploy -->|push mirror| bs-auto-deploy rt-auto-deploy -.-> s-auto-deploy class c-master,c-stable,c-auto-deploy canonical class b-master,b-stable,b-auto-deploy canonical class s-master,s-stable,s-auto-deploy security class bs-master,bs-stable,bs-auto-deploy security ``` #### Picking into auto-deploy process on Canonical 1. Developer open merge request against canonical master. * Merge request follows the review process * Merge request is merged 1. Developers add ~"Pick into auto deploy" label 1. Release tools: * Fetches merged merge requests that have ~”Pick into auto-deploy” * Brings them into `security/X-Y-auto-deploy-YYYYMMDD` branch * Because of our setup, Security auto-deploy branch is automatically pushed to Build. **Notes:** * Note that there is no longer an auto-deploy branch in Canonical. All changes intended for auto-deploy -- both Canonical changes and Security fixes against `master` -- get picked into the _Security_ auto-deploy branch. This way, our servers always have the latest fixes, security or otherwise. #### Picking into auto-deploy process on Security 1. Developer open merge request against Security `security/master` and adds ~unpublished label. * Merge request follows the review process * Merge request is merged 2. Developer adds ~"Pick into auto-deploy" label 3. Developer prepares the backports 4. Release-tools: * Fetches merged merge requests with ~"Pick into auto-deploy" * Brings them into `Security/X-Y-auto-deploy-YYYYMMDD` branch * Because of our setup, Security auto-deploy branch is automatically pushed to Build **Notes** * Build always has the latest security fixes from Security and regular fixes from Canonical. * Security master contains all the changes from Canonical master and all the security-related fixes #### Publishing security fixes **Alternative 1: Use the ~"unpublished" label** ~”unpublished” label designates which security merge requests have been merged, deployed to production (ideally), but not released to the public (not included in a security or patch release). This label is added when a merge request is opened against `security/master` and the related issue is included in the current security issue. When we’re ready to publish the security release: 1. Release-tools fetches the merge request list from the current security issue 2. Gathers the list of security merge requests. 3. Cherry-picks the merged merge requests with ~unpublished label into Canonical master **Alternative 2: Use the current security issue as a SSOT instead of the label** ## Open questions * In case we merge stable into master at the moment of publishing, we also need to sync from Build to Canonical at the moment of publishing * On publish, security-12-3-stable gets merged into 12-3-stable and then pushed to Canonical. Push mirror handles push to Build * :warning: If a Security MR moves a file (`ee/lib/some_file.rb` goes to `lib/some_file.rb`), then gets merged into `security/master`, it's possible that development has continued to happen on that file in Canonical `master` _at the old location_ (`ee/lib/some_file.rb`). * What happens when merge-train tries to reconcile those two branches? </details> <details> <summary>Former proposal 2</summary> ### Proposal A. Merge protected branches 1. On `Security` * We prefix protected branches with `security/`, so `master` becomes `security/master`, `12-8-stable` becomes `security/12-8-stable`, `12-8-auto-deploy-20200120` becomes `security/12-8-auto-deploy-20200120`. 2. When performing a security release: * We disable mirroring from `Canonical` to `Security` * We merge protected branches from `Canonical` to `Security` every `X` minutes. 3. When we're ready to publish: * We merge protected branches from `Security` into `Canonical` and push to all remotes. ```mermaid graph LR classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px classDef security fill:#e086bd,stroke:#333,stroke-width:1px subgraph Canonical c-master(master) c-stable(12-8-stable) c-auto-deploy(12-8-auto-deploy-20200120) c-feature(feature/some-new-feature) end subgraph Security s-master(security/master) s-stable(security/12-8-stable) s-auto-deploy(security/12-8-auto-deploy-20200120) end subgraph Build b-master(master) b-stable(12-8-stable) b-auto-deploy(12-8-auto-deploy-20200120) end c-master -->|merge| s-master s-master -->|push mirror| b-master c-stable -->|merge| s-stable s-stable -->|push mirror| b-stable c-auto-deploy -->|merge| s-auto-deploy s-auto-deploy -->|push mirror | b-auto-deploy class c-master,c-stable,c-auto-deploy canonical class b-master,b-stable,b-auto-deploy canonical class s-master,s-stable,s-auto-deploy security ``` ### Proposal B. Selective mirroring. 1. On `Security` * We prefix protected branches with `security/`, so `master` becomes `security/master`, `12-8-stable` becomes `security/12-8-stable`, `12-8-auto-deploy-20200120` becomes `security/12-8-auto-deploy-20200120`. 2. When performing a security release: * We activate "Selective mirroring": This one keeps pushing changes to the remotes even if these ones diverged. 3. When we're ready to publish: * We merge protected branches from `Security` into `Canonical` and push to all remotes. ```mermaid graph LR classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px classDef security fill:#e086bd,stroke:#333,stroke-width:1px subgraph Canonical c-master(master) c-auto-deploy(12-8-auto-deploy-20200120) c-stable(12-8-stable) c-feature(feature/some-new-feature) end subgraph Security s-master(security/master) s-auto-deploy(security/12-8-auto-deploy-20200120) s-stable(security/12-8-stable) end subgraph Build b-master(master) b-auto-deploy(12-8-auto-deploy-20200120) b-stable(12-8-stable) end c-master -->|push mirror| s-master s-master -->|push mirror| b-master c-stable -->|push mirror| s-stable s-stable -->|push mirror| b-stable c-auto-deploy -->|push mirror | s-auto-deploy s-auto-deploy -->|push mirror | b-auto-deploy class c-master,c-stable,c-auto-deploy canonical class b-master,b-stable,b-auto-deploy canonical class s-master,s-stable,s-auto-deploy security ``` </details> ## Status 2020-09-28 An early merge window is opened ahead of the security release during which Security fixes are merged as soon as they're ready and released to GitLab.com. A new process removes much of the backport overhead for engineers.