Acceptance Testing RepsitoryService::HasLocalBranches
~Conversation: #616 (closed)
See the Migration Process documentation for more information on the Acceptance Testing stage of the process.
Details
-
Feature Toggle Name:
gitaly_has_local_branches
-
GRPC Service:
RepsitoryService::HasLocalBranches
-
Required Gitaly Version:
v0.45.1
-
Required GitLab Version:
v10.2
1. Preparation
-
Routes:
A bunch of routes use this call since has_local_branches?
is one of the conditions tested to check if the repository is empty. I think it should be sufficient to test if /[path-to-project]
, the project landing page, works correctly, since it's definitely heavily used, and relies on this call.
2. Development Trial
Check Dev Server Versions
-
Gitaly: Gitaly Dev Version Tracker Dashboard -
GitLab: https://dev.gitlab.org/help
dev.gitlab.org
:
Enable on -
!feature-set gitaly_has_local_branches true
in#dev-gitlab
Then leave running while monitoring and performing some testing through web, api or SSH.
Monitor (initially )
-
Monitor Grafana feature dashboard on dev: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last hour for unusual activity
- GRPC_METHOD_NAME errors, last hour for unusual activity
-
Check for errors in Gitaly Dev Sentry -
Check for errors in GitLab Dev Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set GITALY_FEATURE_NAME false
in#dev-gitlab
otherwise leave running and proceed proceed to next stage.
3. Staging Trial
Check Staging Server Versions
staging.gitlab.com
Enable on -
!feature-set GITALY_FEATURE_NAME true
in#development
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on staging: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last hour for unusual activity
- GRPC_METHOD_NAME errors, last hour for unusual activity
-
Check for errors in Gitaly Staging Sentry -
Check for errors in GitLab Staging Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using !feature-set GITALY_FEATURE_NAME false
in#development
otherwise leave running and proceed to next stage.
4. Production Server Version Check
-
Gitaly: Gitaly Production Version Tracker Dashboard -
GitLab: https://gitlab.com/help
5. Initial Impact Check
-
Create an issue in the infrastructure tracker: Create issue now -
Set Gitaly to 1% using the command !feature-set GITALY_FEATURE_NAME 1
in#production
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last hour for unusual activity
- GRPC_METHOD_NAME errors, last hour for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set GITALY_FEATURE_NAME false
in#production
otherwise leave running and proceed to next stage.
6. Low Impact Trial
-
Set Gitaly to 5% using the command !feature-set gitaly_has_local_branches 5
in#production
Then leave running while monitoring for at least 2 hours.
Monitor (at least every 20 minutes)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last 2 hours for unusual activity
- GRPC_METHOD_NAME errors, last 2 hours for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set GITALY_FEATURE_NAME false
in#production
otherwise leave running and proceed to next stage.
7. Mid Impact Trial
-
Set Gitaly to 50% using the command !feature-set GITALY_FEATURE_NAME 50
in#production
Then leave running while monitoring for at least 24 hours.
Monitor (at least every few hours)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last 24 hours for unusual activity
- GRPC_METHOD_NAME errors, last 24 hours for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set GITALY_FEATURE_NAME false
in#production
otherwise leave running and proceed to next stage.
8. Full Impact Trial
-
Set Gitaly to 100% using the command !feature-set gitaly_has_local_branches 100
in#production
Then leave running while monitoring for at least 1 week.
Monitor (at least every day)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - GRPC_METHOD_NAME invocations, last 7 days for unusual activity
- GRPC_METHOD_NAME errors, last 7 days for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Success?
-
Close this issue and mark the ~Conversation as ~"Migration:Opt-In"