Acceptance Testing: SSH::SSHUploadPack

~Conversation: #218 (closed)

See the Migration Process documentation for more information on the Acceptance Testing stage of the process.

Feature Toggle Environment Variable: :gitaly_ssh_upload_pack

---

1. Preparation

• Routes: what routes use this migration?
  • git clone & git fetch
• Sentry:
  • Ensure that all gitaly_migrate issues in the GitLab.com tracker are either assigned or resolved: https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+is%3Aunassigned+gitaly_migrate
  • Ensure that all issues in the Gitaly Production tracker are either assigned or resolved: https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+is%3Aunassigned
• Grafana
  • Link to the Gitaly Feature Status dashboard (edit accordingly): https://performance.gitlab.net/dashboard/db/gitaly-feature-status?var-method=SSHUploadPack&refresh=5m&orgId=1
• Kibana
  • Based on routes listed above, provide a Kibana short-url link to incoming requests to that route. https://log.gitlap.com/goto/c96f572778e1bf65d02adcc52771a58a
  • Provide a Kibana short-url link to Gitaly logs related to this endpoint https://log.gitlap.com/goto/f881d37785050594a76b2c27eb9a0e64

2. Development and Staging Trial

• Enable on dev.gitlab.org:
  • ssh into dev.gitlab.org and enable the feature running by running Feature.get('gitaly_FEATURE_NAME').enable on a rails console.
  • Perform some testing and leave the feature enabled
• Enable on staging.gitlab.com in #development
  • Perform some testing and leave the feature enabled

3. Mid Impact Trial

• Set Gitaly to 50% using the command !feature-set gitaly_FEATURE_NAME 50 in #production
• Leave running for at least 24 hours
• Monitor sentry, grafana and kibaba links above, every few hours
  • On usual activity, disable trial with !feature-set gitaly_FEATURE_NAME false in #production

4. Full Impact Trial

• Set Gitaly to 100% using the command !feature-set gitaly_FEATURE_NAME 50 in #production
• Leave running for at least a week
• Monitor sentry, grafana and kibaba links above daily
  • On usual activity, disable trial with !feature-set gitaly_FEATURE_NAME false in #production

5. Outcome

• Report on the outcome of the tests (leave comments on this issue)

  • Did the migration perform as expected?
  • Did the code have reasonable performance characteristics?
  • Did error rates jump to an unacceptable level?

• Additional Runbook information required?

  • If so, was it added? (link to MR)

• Prometheus Alerts Added

  • If so, was it added? (link to MR)