Skip to content
GitLab Next
Closed
Created by Mek Stittri@meksMaintainer

Requesting volunteers to own feature areas to validate on staging after GCP migration failover

From the last discussion in #330 (comment 72461420) with @grzesiek and @andrewn

We will need help with the rest of engineering to test the following features. Currently, we have volunteers from the production and geo teams but this is not enough #330 (comment 71582338) However, not all of the engineers involved in the GCP migration has the adequate setup to run all these tests. It's more effective to delegate these areas to the original teams who wrote the features to verify since they should already have these setup on staging and production as part of the development workflow.

@dhavens @tommy.morgan can both of you volunteer a member on your team to help us out with this, please ?

This directly helps with our GCP migration efforts. Please understand that this is one of the most important initiatives for the company and we appreciate all the help from everyone.

README

If you are volunteering, please make yourself available to join the GCP Migration Rehearsal call. Ping @andrewn or @meks for an invite. The next one is on Friday, May 25 6:00am – 8:00am Pacific Time

The staging migration rehearsal steps can be found here https://gitlab.com/gitlab-com/migration/blob/master/.gitlab/issue_templates/failover.md

  • In this failover plan there is a section for QA during Blackout which is when the tests in the test plan will take place.
  • Link to the section for QA during Blackout
  • Link to the test plan

Note: Any setup for validating the features has to be completed BEFORE the failover. This is so that the state gets carried over to the new env in GCP from staging. We want to shorten the Black out period as much as we can.

Test plan

We aim to keep the test plan as lightweight yet effective as possible. For manual clicking through validations, we are keeping it free form. Just enough description to tell people what to test helps fuzzes the workflow up. We are more likely to catch more critical bugs that way. This is a page taken from the book How Google Tests Software. For running command in steps please feel free to add more information as sub-bullet items in your respective areas.

Note: As of 2018-05-25, the team decided to move to Google Sheets to avoid mid-air update collisions during the critical phase of the test run.

During the Blackout

Tests that run during the Blackout are high priority. These are the areas that needs to be validated before we cross the point of no return (rollback of migration)

  1. Repository Operations - @mkozono
  2. pushing to protected branch (provide access when it should and protect when it should)
  3. ssh
  4. forking
  5. Access existing LFS object
  6. Create new LFS object
  7. Uploads - @reprazent
  8. Access existing upload
  9. Create new upload
  10. Artifacts - @mkozono
  11. Access existing artifacts
  12. Create new artifacts
  13. Basic API tests - @rymai
  14. Using access tokens to perform API requests:
    1. Generate a new API token if needed at https://staging.gitlab.com/profile/personal_access_tokens and save it at ~/.gitlab-token-qa
    2. export PROJECT_PATH=$(date "+qa-api-tests-%Y-%m-%d-%H-%M-%S")
    3. POST: curl --request POST --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" --data "name=$PROJECT_PATH" --data "path=$PROJECT_PATH" https://staging.gitlab.com/api/v4/projects
    4. POST: curl --request POST --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" --data "branch=master" --data "content=Hello world" --data "commit_message=Add README.md" https://staging.gitlab.com/api/v4/projects/gitlab-qa%2F$PROJECT_PATH/repository/files/README%2Emd
    5. GET: curl --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" https://staging.gitlab.com/api/v4/projects/gitlab-qa%2F$PROJECT_PATH/repository/files/README%2Emd\?ref\=master
    6. DELETE: curl --request DELETE --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" --data "branch=master" --data "commit_message=Remove README.md" https://staging.gitlab.com/api/v4/projects/gitlab-qa%2F$PROJECT_PATH/repository/files/README%2Emd
    7. GET: curl --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" https://staging.gitlab.com/api/v4/projects/gitlab-qa%2F$PROJECT_PATH/repository/tree => []
    8. DELETE: curl --request DELETE --header "PRIVATE-TOKEN: $(cat ~/.gitlab-token-qa)" https://staging.gitlab.com/api/v4/projects/gitlab-qa%2F$PROJECT_PATH => {"message":"202 Accepted"}
    9. Using session cookie to access API: once logged-in, visit https://staging.gitlab.com/api/v4/user
  15. Some basic validation for Webhooks - @fjsanpedro
  16. add new webhooks
  17. Trigger webhooks
  18. Issue boards - @felipe_artur
  19. create a list on group issue board
  20. move issue between lists on group issue board
  21. scope a group issue board
  22. create a list on project issue board
  23. move issue between lists on project issue board
  24. scope a project issue board
  25. CI/CD Workflow (Basic flow) - @bikebilly @DylanGriffith
  • Coverage: project templates / GKE integration / k8s integration / ingress and prometheus integration / Auto DevOps basic flow
  1. Setup
    1. create a new project based on the Rails template
    2. create a new k8s cluster on GKE using the GitLab interface
    3. install helm, ingress, prometheus
    4. enable Auto DevOps and configure the domain
    5. At this point, a pipeline will be executed automatically.
  2. Checks
    1. all the jobs in the pipeline succeeded
    2. a deployment to production happened
    3. default page is visible at the given url for production
    4. performances metrics are shown
  3. GitLab Service Desk - @stanhu (PRODUCTION ONLY)
  4. incoming emails
  5. AlertManager Alerts - @dawsmith on behalf of Production Team
  6. AlertManager

After the Blackout

Tests that run after the Blackout covers functionality that can be addressed after we are on GCP.

  1. CI/CD Workflow (Advanced flow) - @bikebilly @DylanGriffith
  • Coverage: runner integration / staging deployments / incremental rollout deployments / multi-pod deployments / deploy boards / web terminals
  1. Setup
    1. install runner on the cluster
    2. disable shared runners in project settings
    3. run a new manual pipeline adding the following variables:
    4. STAGING_ENABLED -> 1
    5. PRODUCTION_REPLICAS -> 10
    6. INCREMENTAL_ROLLOUT_ENABLED -> 1
    7. after the staging job ended, trigger the rollout 10% manual action
    8. after the rollout 10% job ended, trigger the rollout 100% manual action
    9. click on the web terminal button and execute id on the remote shell
  2. Checks
    1. all the jobs in the pipeline succeeded
    2. a deployment to staging happened
    3. default page is visible at the given url for staging
    4. manual actions are available to do incremental rollout (10%, 25%, 50%, 100%)
    5. deploy boards show 1 pod for staging
    6. deploy boards show 1 new pod and 9 old pods after rollout 10% job ended
    7. deploy boards show 10 pods after rollout 100% job ended
    8. check for the output of the id command in the web terminal
  3. Email (Production only for outgoing email)
  4. Does outgoing notification continue to work from new sending hosts DKIM, SPF (PRODUCTION ONLY) - @toon
  5. Do incoming emails update notes - @toon
  6. Create an issue from email @jprovaznik
  7. Wiki Operations - @digitalmoksha
  8. Access and modify an existing wiki
  9. Create new wiki
  10. Clone wiki repository
  11. Pages - @toon (PRODUCTION ONLY)
  12. Access existing page
  13. Create new page
  14. Access from Azure
  15. Access from GCP
  16. Mirror - @vsizov
  17. Push
  18. Pull
  19. Backups - @dawsmith on behalf of Production Team
  20. Redis
  21. Postgres
  22. Git Data
  23. Kubernetes monitoring - @bjk-gitlab
  24. TBD
  25. File locking @rdavila
  26. Lock/Unlock files through the UI
  27. Lock/Unlock files through git lfs lock
  28. List locked files through git lfs locks
  29. CI/CD for external repo - @jamedjo
  30. TBD
  31. CI/CD for GitHub - @jamedjo
  32. TBD
  33. Multi-project pipelines Follow up issue: #516 (closed)
  34. TBD
  35. AutoDevOps workflow Follow up issue: #516 (closed)
  36. Canary deployments needs owner outline test areas
    1. TBD

Once we have owners to these points I will update the main test plan template. Which can be found here https://gitlab.com/gitlab-com/migration/blob/master/.gitlab/issue_templates/test_plan.md

/cc @rymai @andrewn @nick.thomas @mkozono @grzesiek

Edited by Mek Stittri
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information