CommitService::ListCommitsByOid Acceptance Testing
~Conversation: #670 (closed)
See the Migration Process documentation for more information on the Acceptance Testing stage of the process.
Details
-
Feature Toggle Name:
gitaly_list_commits_by_oid
-
GRPC Service:
CommitService::ListCommitsByOid
-
Required Gitaly Version:
v0.56.0
-
Required GitLab Version:
v10.4
1. Preparation
-
Routes: what routes use this migration? Pipelines#index
2. Development Trial
Check Dev Server Versions
-
Gitaly: Gitaly Dev Version Tracker Dashboard -
GitLab: https://dev.gitlab.org/help
dev.gitlab.org
:
Enable on -
!feature-set gitaly_list_commits_by_oid true
in#dev-gitlab
Monitor (initially )
-
Monitor Grafana feature dashboard on dev: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - ListCommitsByOid invocations, last hour for unusual activity
- ListCommitsByOid errors, last hour for unusual activity
-
Check for errors in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.CommitService %2FListCommitsByOid) -
Check for errors in GitLab Dev Sentry
3. Staging Trial
Check Staging Server Versions
staging.gitlab.com
Enable on -
!feature-set gitaly_list_commits_by_oid true
in#development
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on staging: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - ListCommitsByOid invocations, last hour for unusual activity
- ListCommitsByOid errors, last hour for unusual activity
-
Check for errors in Gitaly Staging Sentry -
Check for errors in GitLab Staging Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using !feature-set gitaly_list_commits_by_oid false
in#development
otherwise leave running and proceed to next stage.
4. Production Server Version Check
-
Gitaly: Gitaly Production Version Tracker Dashboard -
GitLab: https://gitlab.com/help
5. Initial Impact Check
-
Create an issue in the infrastructure tracker: Create issue now -
Set Gitaly to 1% using the command !feature-set gitaly_list_commits_by_oid 1
in#production
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - ListCommitsByOid invocations, last hour for unusual activity
- ListCommitsByOid errors, last hour for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_list_commits_by_oid false
in#production
otherwise leave running and proceed to next stage.
6. Low Impact Trial
-
Set Gitaly to 5% using the command !feature-set gitaly_list_commits_by_oid 5
in#production
Then leave running while monitoring for at least 2 hours.
Monitor (at least every 20 minutes)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=ListCommitsByOid &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [ListCommitsByOid invocations, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [ListCommitsByOid errors, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.CommitService %2FListCommitsByOid ) -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_list_commits_by_oid false
in#production
otherwise leave running and proceed to next stage.
7. Mid Impact Trial
-
Set Gitaly to 50% using the command !feature-set gitaly_list_commits_by_oid 50
in#production
Then leave running while monitoring for at least 24 hours.
Monitor (at least every few hours)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=ListCommitsByOid &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [ListCommitsByOid invocations, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [ListCommitsByOid errors, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.CommitService %2FListCommitsByOid ) -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_list_commits_by_oid false
in#production
otherwise leave running and proceed to next stage.
8. Full Impact Trial
-
Set Gitaly to 100% using the command !feature-set gitaly_list_commits_by_oid 100
in#production
Then leave running while monitoring for at least 1 week.
Monitor (at least every day)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=ListCommitsByOid &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [ListCommitsByOid invocations, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [ListCommitsByOid errors, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:ListCommitsByOid %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.CommitService %2FListCommitsByOid ) -
Check for errors in GitLab Sentry
Success?
-
Close this issue and mark the ~Conversation as ~"Migration:Opt-In"
Edited by Kim Carlbäcker