Background (Internal doc): https://docs.google.com/document/d/1i4c5DaU0juJxGgVisu7_n1w8qvQHZQJ_GXGzSt6X8Bw/edit#heading=h.19v39r7xn099 - We need so shore up mix deployment tests to ensure stability of GitLab.com - We need a new environment, and retire existing staging, declare bankruptcy with existing staging - We need new environment that is able to mirror production (both in topology, as well as the type of load they are receiving) Pairing infra epic in `gitlab-com` https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/559 * Slack https://gitlab.slack.com/archives/C02BA0YKRUH * Google Doc https://docs.google.com/document/d/1KDp_4RI1U6O6aCfOhH99NriU4IJx0qcZjQx-B2e0WUE/edit# * Kick-off https://www.youtube.com/watch?v=Rj3WdVKWiic ## Overview The problem happens when changes are introduced in GitLab-Canary that are incompatible with GitLab-Stable-Production. These incompatibilities will mostly occurs in asynchronous processes where data is persisted in some format. Either the method of writing the data or the method of reading the data will have changed, creating inconsistency and possible failure between different versioned GitLab nodes. This occurs when a feature is implemented that fails to follow the [expand and contract pattern](https://docs.gitlab.com/ee/development/multi_version_compatibility.html#requirements-for-zero-downtime-upgrades) detailed in our documentation. ### Urgency We now have 2 high severity incidents that could be caught with mixed deployments. 1. CI build tokens backwards compatibility https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3441 * Immediate corrective action completed with mix deployment deemed for future iterations https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10546 1. 500 errors - each UNION query must have the same number of columns https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5171 ### Mixed Deployment Requirements We have a plan outlined below with a multi layered testing and additional training on observing change patterns that will trigger this. 1. Some lightweight smoke test that performs migrations while the current version is running, and then tries some basic tests with the new version. 1. Mimic our deploys to production by making staging mirror the exact same procedure, preferably with similar data set. 1. Separate environments that test the mixed deployment case. (optional if we have 1 and 2) 1. Ensure new developers are consistently trained to observe the `expand/contract` pattern mentioned above ### More capable Staging Requirements * Move us closer to testing needs of Quality and Development teams * Moving us closer to testing in a production-like environment * Admin testing access * Current staging has customer data which is a blocker for giving more access to development and quality teams. * Testing Different paid tiers * Democratizing testing and better test data * Fresh and familiar data for GitLab team members in `gitlab-org` in an ideal staging * Better access to test accounts and wider permissions * Other facets like Geo ## Plan Setup **Staging-Canary** to catch mix deployments while working on another Staging based on reference architecture in parallel. (**staging-ref**) Failures in **staging-ref** will not block deployments initially until we are sure that signal to noise ratio is clear. **Phase 1** ``` -> staging-ref (new) -> QA / gstg-migrations -> gstg-cny (new) -> QA (new) -> gstg -> QA -> gprd-migrations -> gprd-cny -> QA -> gprd ``` Next we will move to use all QA on all Staging to block deployments and ensure quality and reliability. **Phase 2** ``` -----> staging-ref -> QA ------> / \ gstg-migrations -> gstg-cny -> QA -> gstg -> QA -> gprd-migrations -> gprd-cny -> QA -> gprd ``` * Mix-deployment track (Add **Staging-Canary** to **Existing Staging**) * [x] Setup `gstg-cny` @pguinoiseau https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14043 * [x] Wire in `release-tools` for `gstg-cny` @pguinoiseau https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14045 * [x] Validate mixed-deployment flow @zeffmorgan https://gitlab.com/gitlab-org/gitlab/-/issues/338970 * Test that has cooking routing capabilities to validate client views on incompatible versions. * [x] New QA pipeline for Staging Canary similar to what we [have on Canary/Prod using a specific cookie](https://gitlab.com/gitlab-org/quality/canary/-/blob/master/.gitlab-ci.yml#L26) https://gitlab.com/gitlab-org/gitlab/-/issues/340639 * [x] Refactor Gitlab QA gem for [report issues](https://gitlab.com/gitlab-org/gitlab-qa/-/issues/649) * [x] Ensure that new mix-deployment test is not lost in the noise and monitoring is setup to catch this signal * [x] [Refactor pipeline-common CI config to facilitate mixed deployment environment testing](https://gitlab.com/gitlab-org/gitlab/-/issues/344541) * [x] [Add Staging Canary QA pipeline](https://gitlab.com/gitlab-org/gitlab/-/issues/340639) configuration (complete and approved, dependent on a test change being merged now) * [x] Announcements for changes * [ ] [Refactor test for mixed deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/344213) environment new pipeline deployment * [x] Add [debug/triage documentation](https://gitlab.com/gitlab-org/gitlab/-/issues/341752) for mixed deployment environment tests * [x] Update existing staging infrastructure and remove flaky tests cause of underlying infrastructure @zeffmorgan https://gitlab.com/gitlab-org/gitlab/-/issues/338972 * [x] [Analyze potential mixed deployment environment failure points](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1047) * [x] [Refactor mixed-deployment jobs to explicitly limit domains](https://gitlab.com/gitlab-org/quality/pipeline-common/-/issues/19) * [x] [Implement API-based canary cookie-switching functionality to enable more testing types](https://gitlab.com/gitlab-org/gitlab/-/issues/343654) * [x] [Refactor UI canary cookie switching](https://gitlab.com/gitlab-org/gitlab/-/issues/343653) * [x] Explore adding new tests for mixed deployment specifically @zeffmorgan https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1047 * [x] Provide chatops support for Staging Canary environment https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14291 * [x] Improve load testing for staging-canary @niskhakova https://gitlab.com/gitlab-org/gitlab/-/issues/338978 * New Staging track (**staging-ref**) * [x] Identify GCP provisioning pool and provide access to the team here @pguinoiseau https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14044 * [x] Setup 10K reference architecture `staging-ref` @niskhakova https://gitlab.com/gitlab-org/gitlab/-/issues/338973 * [x] Set up SSL for staging-ref environment - https://gitlab.com/gitlab-org/gitlab/-/issues/343363 * [x] Create test data and test accounts @niskhakova https://gitlab.com/gitlab-org/gitlab/-/issues/338974 * [x] Expand test data and test accounts requirements from Development and quality teams @ibaum https://gitlab.com/gitlab-org/gitlab/-/issues/338976 * [x] Wire in `release-tools` @pguinoiseau @niskhakova https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14046 * Add GitLab-QA regression tests to Staging Ref: * [x] Support GitLab-QA pipeline for the new Staging Ref @niskhakova - https://gitlab.com/gitlab-org/gitlab/-/issues/338977 * [x] Trigger Staging Ref QA pipeline during deployment process @pguinoiseau @niskhakova - https://gitlab.com/gitlab-org/gitlab/-/issues/343936 - (_requires_ https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14046) * [x] Create access tokens for Staging Ref automatically @niskhakova - https://gitlab.com/gitlab-org/gitlab/-/issues/343921 * [x] Provide GitLab team access to Staging Ref @niskhakova - https://gitlab.com/gitlab-org/gitlab/-/issues/341605 * [x] Set up SMTP for staging-ref environment @niskhakova - https://gitlab.com/gitlab-org/gitlab/-/issues/340554 * [x] Provide chatops support for Staging Ref environment @pguinoiseau https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14292 * [x] Add documentation about Staging Ref environment @niskhakova https://gitlab.com/gitlab-org/gitlab/-/issues/340323 * [x] Set staging-ref to blocking deployments @pguinoiseau @niskhakova https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14294 - decision not to implement this, see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14294#note_794062778 ### Next iteration for staging-ref <details><summary>Click to expand</summary> We will use `staging-ref` to rinse-repeat fast as we include more testing features e.g. High Level: * [x] Geo * [x] Global search - configured [in iteration#1](https://about.gitlab.com/handbook/engineering/infrastructure/environments/staging-ref/#environment-information) Staging Ref iteration#2 work: * [x] Configure monitoring/alerts same as production https://gitlab.com/gitlab-com/runbooks @pguinoiseau https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14047 * Configure Geo for Staging Ref environment * [x] Gather feedback and plan work for Staging Ref Geo - https://gitlab.com/gitlab-org/gitlab/-/issues/350741 * [x] Setup Geo site for Staging Ref @niskhakova https://gitlab.com/gitlab-org/gitlab/-/issues/350869 * [x] Set up GitLab-QA pipeline for the new Staging Ref Geo site https://gitlab.com/gitlab-org/gitlab/-/issues/350870 * Configure CustomersDot portal for Staging Ref * [x] Add Paid Tiers on Staging Ref - https://gitlab.com/gitlab-org/gitlab/-/issues/342150 * [x] Link QA test session reports to Release tasks - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2168 * [ ] Deliver requested Test data from Development teams for new Staging - https://gitlab.com/groups/gitlab-org/-/epics/7020 * [ ] Set up Incoming email for Staging-Ref environment - https://gitlab.com/gitlab-org/gitlab/-/issues/348970 Outstanding non-critical work for Staging Ref that will be done future iterations https://gitlab.com/groups/gitlab-org/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=staging-improvements&label_name%5B%5D=Future&first_page_size=20 The goal of **staging-ref** track is to get us on a faster track. We also want to limit the number of staging and if a new staging env is created it should be short lived. To achieve this, anything that is needed to setup staging should be automated, baked into our tooling efficiency. </details> ## Issue creation locations * For infrastructure tasks https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues * For Testing capabilities requirements (tests accounts, test data, tests coverage) https://gitlab.com/gitlab-org/gitlab/-/issues * Gaps in GET https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit ## Issue Boards * For Development/ Quality tasks - https://gitlab.com/groups/gitlab-org/-/boards/3450576?label_name[]=staging-improvements * For Infrastructure tasks - https://gitlab.com/groups/gitlab-com/gl-infra/-/boards/3142168?label_name%5B%5D=staging-improvements ## Staffing Allocation | Individual | Role | | ------ | ------ | | @niskhakova | Senior Software Engineer in Test | | @stanhu | Engineering Fellow | | @zeffmorgan | Senior Software Engineer in Test | | @ibaum | Senior Engineer, Geo | | @pguinoiseau | Senior SRE, Infrastructure | | @vincywilson | Acting QEM, Enablement | ## Overall Summary ### Staging Ref Summary Staging Ref is a sandbox environment used for pre-production testing of the latest Staging Canary code with full access to the environment and control over data ([1](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14044), [2](https://gitlab.com/gitlab-org/gitlab/-/issues/338973), [3](https://gitlab.com/gitlab-org/gitlab/-/issues/341605)). This environment has been built with the great collaboration of Infrastructure and Quality teams. * Staging Ref deployment runs parallel to Staging Canary deployment. [Deployer](https://ops.gitlab.net/gitlab-com/gl-infra/deployer) triggers a job in [Staging-Ref GET Config](https://ops.gitlab.net/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref) to update the environment. It also supports ChatOps commands to pause deployments or enable/disable feature flags. * https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14046, https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14292, https://gitlab.com/gitlab-org/gitlab/-/issues/350482 * GitLab QA end-to-end tests are run after each deployment https://ops.gitlab.net/gitlab-org/quality/staging-ref/-/pipelines * https://gitlab.com/gitlab-org/gitlab/-/issues/338977, https://gitlab.com/gitlab-org/gitlab/-/issues/343936, https://gitlab.com/gitlab-org/gitlab/-/issues/343921 * Staging ref is configured with SSL Certificates, Google OAuth, Mailgun SMTP, Outgoing email, Advanced Search, Paid Plans, Sentry and Snowplow * https://gitlab.com/gitlab-org/gitlab/-/issues/343363, https://gitlab.com/gitlab-org/gitlab/-/issues/340554, https://gitlab.com/gitlab-org/gitlab/-/issues/342150, https://gitlab.com/gitlab-org/gitlab/-/issues/348970, https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/576 * Environment monitoring is available at [staging-ref folder](https://dashboards.gitlab.net/d/Fyic5Wanz/server-performance?orgId=1), it's configured using `prometheus-stack` and connected to existing .com Thanos cluster. * https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14047, https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/594 * Environment can be rebuilt if required with minimal manual tasks * https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/12, https://gitlab.com/gitlab-org/gitlab/-/issues/343921 * Geo is setup on Staging Ref where Staging Ref US site (primary) [10k Cloud Native Hybrid Reference Architecture](https://docs.gitlab.com/ee/administration/reference_architectures/10k_users.html#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) environment and Staging Ref EU site (secondary) [3k Reference Architecture](https://docs.gitlab.com/ee/administration/reference_architectures/3k_users.html) full Omnibus environment * https://gitlab.com/gitlab-org/gitlab/-/issues/350741, https://gitlab.com/gitlab-org/gitlab/-/issues/350869, https://gitlab.com/gitlab-org/gitlab/-/issues/350870, https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/11, https://gitlab.com/gitlab-org/gitlab/-/issues/352506, https://gitlab.com/gitlab-org/gitlab/-/issues/352496 * Staging Ref is currently pre-configured with Test data that was identified as part of this effort * https://gitlab.com/gitlab-org/gitlab/-/issues/338974, https://gitlab.com/gitlab-org/gitlab/-/issues/344427). The ongoing work is tracked in https://gitlab.com/groups/gitlab-org/-/epics/7020 Full documentation can be found at: https://about.gitlab.com/handbook/engineering/infrastructure/environments/staging-ref/ Since the moment Staging Ref became available in January 2022, there have been [multiple examples](https://gitlab.com/gitlab-org/gitlab/-/issues/350744#usage-examples) when it's been used by GitLab team members. Some of them - `gitlab-sshd` daemon testing, Gitaly queue limits, UX team members used Staging Ref to test features that required Admin access. ### Staging Canary Summary Staging Canary is an improved staging environment bringing GitLab another step closer to enabling testing in a production-like environment to discover issues quickly. The Infrastructure and Quality teams collaborated to setup the Staging Canary environment and test automation within it. * Staging Canary adds to the original Staging environment for pre-production testing, mimicking our current canary/production deployment. This gives us an opportunity to uncover issues that may be missed by testing code in isolated environments as opposed to when mixed versions are utilizing the same shared components such as databases, etc. * https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2089, https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2087, https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/608 * Mixed deployment testing occurs with each deployment, covering thoroughly investigated risk factors. Many areas of risk are introduced in mixed version deployment scenarios, such as our Canary deployments. Testing that targets this mixed environment assists us in mitigating these risks. * https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1047 * Staging Canary Quality project is configured to enable testing mixed version deployments but also enables us to target only the new deployment if desired. * https://gitlab.com/gitlab-org/quality/staging-canary, https://gitlab.com/gitlab-org/quality/staging-canary/-/merge_requests/1, https://gitlab.com/gitlab-org/quality/staging-canary/-/merge_requests/2, https://gitlab.com/gitlab-org/gitlab/-/issues/344541 * Quality tests are triaged in the Quality pipeline triage rotation with coverage and debugging instructions added to help us identify mixed version deployment issues and correct them before hitting production. * https://gitlab.com/gitlab-org/gitlab/-/issues/341752, https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/93884 * Reporting is enabled to differentiate errors within Staging Canary from other pipelines, enabling teams to quickly react as with our other pipelines. * https://gitlab.com/gitlab-org/gitlab-qa/-/issues/649 Documentation for the Staging Canary environment can be found at: https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary Staging Canary has been successfully operating to block deployments since the official pipeline re-order was announced in March 2022. ## Future Improvements Future improvements to Staging Ref and Staging Canary are tracked in https://gitlab.com/groups/gitlab-org/-/issues/?sort=updated_desc&state=opened&label_name%5B%5D=staging-improvements&label_name%5B%5D=Future&first_page_size=20 ## Thank you!!! This was a great, collaborative effort and many thanks go out to all those who helped make this huge effort happen! :handshake: :rocket: