Acceptance Testing: RenameNamespace
~Conversation: #385 (closed)
See the Migration Process documentation for more information on the Acceptance Testing stage of the process.
Details
-
Feature Toggle Name:
gitaly_remove_namespace
-
GRPC Service:
NamespaceService::RenameNamespace
-
Required Gitaly Version:
v0.43.0
-
Required GitLab Version:
v10.1
1. Preparation
-
Routes: what routes use this migration? - Only renaming a Group/User
2. Development Trial
Check Dev Server Versions
-
Gitaly: Gitaly Dev Version Tracker Dashboard -
GitLab: https://dev.gitlab.org/help
dev.gitlab.org
:
Enable on -
!feature-set gitaly_remove_namespace true
in#dev-gitlab
Monitor (initially )
-
Monitor Grafana feature dashboard on dev: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=RenameNamespace &var-job=gitaly-dev&refresh=5m) -
Inspect logs in ELK: - [RenameNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Dev Sentry](https://sentry.gitlap.com/gitlab/devgitlaborg-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Dev Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remove_namespace false
in#dev-gitlab
otherwise leave running and proceed proceed to next stage.
3. Staging Trial
Check Staging Server Versions
staging.gitlab.com
Enable on -
!feature-set gitaly_rename_namespace true
in#development
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on staging: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=RenameNamespace &var-job=gitaly-nfs-staging&refresh=5m) -
Inspect logs in ELK: - [RenameNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20hostname:nfs5'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20hostname:nfs5%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Staging Sentry](https://sentry.gitlap.com/gitlab/staginggitlabcom-gitaly/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Staging Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using !feature-set gitaly_remove_namespace false
in#development
otherwise leave running and proceed to next stage.
4. Production Server Version Check
-
Gitaly: Gitaly Production Version Tracker Dashboard -
GitLab: https://gitlab.com/help
5. Initial Impact Check
-
Create an issue in the infrastructure tracker: [Create issue now](https://gitlab.com/gitlab-com/infrastructure/issues/new?issue[title]=Testing%20of%20Gitaly%20Feature%20gitaly_remove_namespace &issue[description]=https%3A%2F%2Fgitlab.com%2Fgitlab-org%2Fgitaly%2Fissues%2FACCEPTANCE_TEST_ISSUE_NUMBER%0A%0A%2Flabel%20 %Milestone 2017-06-16change) -
Set Gitaly to 1% using the command !feature-set gitaly_rename_namespace 1
in#production
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=RenameNamespace &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [RenameNamespace invocations, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last hour](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-1h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remove_namespace false
in#production
otherwise leave running and proceed to next stage.
6. Low Impact Trial
-
Set Gitaly to 5% using the command !feature-set gitaly_rename_namespace 5
in#production
Then leave running while monitoring for at least 2 hours.
Monitor (at least every 20 minutes)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=RenameNamespace &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [RenameNamespace invocations, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last 2 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-2h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remove_namespace false
in#production
otherwise leave running and proceed to next stage.
7. Mid Impact Trial
-
Set Gitaly to 50% using the command !feature-set gitaly_rename_namespace 50
in#production
Then leave running while monitoring for at least 24 hours.
Monitor (at least every few hours)
-
Monitor Grafana feature dashboard on production: [Gitaly Feature Status Dashboard](https://performance.gitlab.net/dashboard/db/gitaly-feature-status?from=now-12h&to=now&orgId=1&var-method=RenameNamespace &var-job=gitaly-production&refresh=5m) -
Inspect logs in ELK: - [RenameNamespace invocations, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last 24 hours](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-24h,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remove_namespace false
in#production
otherwise leave running and proceed to next stage.
8. Full Impact Trial
-
Set Gitaly to 100% using the command !feature-set gitaly_remove_namespace 100
in#production
Then leave running while monitoring for at least 1 week.
Monitor (at least every day)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - [RenameNamespace invocations, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
- [RenameNamespace errors, last 7 days](https://log.gitlap.com/app/kibana#/discover?_a=%28index%3A'gitaly-*'%2Cquery%3A%28query_string%3A%28query%3A'grpc.method:RenameNamespace %20AND%20NOT%20hostname:dev%20AND%20NOT%20grpc.code:OK%20AND%20message:finished'%29%29%29&_g=%28refreshInterval:%28display:Off,pause:!f,value:0%29,time:%28from:now-7d,mode:quick,to:now%29%29) for unusual activity
-
Check for errors in [Gitaly Sentry](https://sentry.gitlap.com/gitlab/gitaly-production/?query=is%3Aunresolved+grpc.method%3A%2Fgitaly.NamespaceService %2FRenameNamespace ) -
Check for errors in GitLab Sentry
Success?
-
Close this issue and mark the ~Conversation as ~"Migration:Opt-In"
Edited by Zeger-Jan van de Weg