Acceptance testing: git remotes pointing to external sources
~Conversation: #554 (closed)
Previously ~Blocked on https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16150 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16331
An example of an 'external source' would be a GitHub repository that is being imported into GitLab.
Details
-
Feature Toggle Name:
gitaly_remote_add_remote
,gitaly_remote_remove_remote
-
GRPC Service:
RemoteService::AddRemote
,RemoteService::RemoveRemote
-
Required Gitaly Version:
v0.64.0
-
Required GitLab Version:
v10.4
1. Preparation
-
Routes: what routes use this migration? - POST
import/github
- POST
2. Development Trial
Check Dev Server Versions
-
Gitaly: Gitaly Dev Version Tracker Dashboard -
GitLab: https://dev.gitlab.org/help
dev.gitlab.org
:
Enable on -
!feature-set gitaly_remote_add_remote true
in#dev-gitlab
-
!feature-set gitaly_remote_remove_remote true
in#dev-gitlab
Then leave running while monitoring and performing some testing through web, api or SSH.
Monitor (initially )
-
Monitor Grafana feature dashboard on dev: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on dev: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last hour for unusual activity
- RemoveRemote invocations, last hour for unusual activity
- AddRemote errors, last hour for unusual activity
- RemoveRemote errors, last hour for unusual activity
-
Check for errors in Gitaly Dev Sentry -
Check for errors in Gitaly Dev Sentry -
Check for errors in GitLab Dev Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_add_remote false
in#dev-gitlab
otherwise leave running and proceed proceed to next stage. -
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_remove_remote false
in#dev-gitlab
otherwise leave running and proceed proceed to next stage.
3. Staging Trial
Check Staging Server Versions
staging.gitlab.com
Enable on -
!feature-set gitaly_remote_add_remote true
in#development
-
!feature-set gitaly_remote_remove_remote true
in#development
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on staging: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on staging: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last hour for unusual activity
- RemoveRemote invocations, last hour for unusual activity
- AddRemote errors, last hour for unusual activity
- RemoveRemote errors, last hour for unusual activity
-
Check for errors in Gitaly Staging Sentry -
Check for errors in Gitaly Staging Sentry -
Check for errors in GitLab Staging Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using !feature-set gitaly_remote_add_remote false
in#development
otherwise leave running and proceed to next stage. -
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately using !feature-set gitaly_remote_remove_remote false
in#development
otherwise leave running and proceed to next stage.
4. Production Server Version Check
-
Gitaly: Gitaly Production Version Tracker Dashboard -
GitLab: https://gitlab.com/help
5. Initial Impact Check
-
Create an issue in the infrastructure tracker: Create issue now -
Create an issue in the infrastructure tracker: Create issue now -
Set Gitaly to 1% using the command !feature-set gitaly_remote_add_remote 1
in#production
-
Set Gitaly to 1% using the command !feature-set gitaly_remote_remove_remote 1
in#production
Then leave running while monitoring for at least 15 minutes while performing some testing through web, api or SSH.
Monitor (at least every 5 minutes, preferably real-time)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last hour for unusual activity
- RemoveRemote invocations, last hour for unusual activity
- AddRemote errors, last hour for unusual activity
- RemoveRemote errors, last hour for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_add_remote false
in#production
otherwise leave running and proceed to next stage. -
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_remove_remote false
in#production
otherwise leave running and proceed to next stage.
6. Low Impact Trial
-
Set Gitaly to 5% using the command !feature-set gitaly_remote_add_remote 5
in#production
-
Set Gitaly to 5% using the command !feature-set gitaly_remote_remove_remote 5
in#production
Then leave running while monitoring for at least 2 hours.
Monitor (at least every 20 minutes)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last 2 hours for unusual activity
- RemoveRemote invocations, last 2 hours for unusual activity
- AddRemote errors, last 2 hours for unusual activity
- RemoveRemote errors, last 2 hours for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_add_remote false
in#production
otherwise leave running and proceed to next stage. -
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_remove_remote false
in#production
otherwise leave running and proceed to next stage.
7. Mid Impact Trial
-
Set Gitaly to 50% using the command !feature-set gitaly_remote_add_remote 50
in#production
-
Set Gitaly to 50% using the command !feature-set gitaly_remote_remove_remote 50
in#production
Then leave running while monitoring for at least 24 hours.
Monitor (at least every few hours)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last 24 hours for unusual activity
- RemoveRemote invocations, last 24 hours for unusual activity
- AddRemote errors, last 24 hours for unusual activity
- RemoveRemote errors, last 24 hours for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Continue?
-
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_add_remote false
in#production
otherwise leave running and proceed to next stage. -
On unexpectedly high calls rates, error rates, CPU activity, etc, disable trial immediately with !feature-set gitaly_remote_remove_remote false
in#production
otherwise leave running and proceed to next stage.
8. Full Impact Trial
-
Set Gitaly to 100% using the command !feature-set gitaly_remote_add_remote 100
in#production
-
Set Gitaly to 100% using the command !feature-set gitaly_remote_remove_remote 100
in#production
Then leave running while monitoring for at least 1 week.
Monitor (at least every day)
-
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Monitor Grafana feature dashboard on production: Gitaly Feature Status Dashboard -
Inspect logs in ELK: - AddRemote invocations, last 7 days for unusual activity
- RemoveRemote invocations, last 7 days for unusual activity
- AddRemote errors, last 7 days for unusual activity
- RemoveRemote errors, last 7 days for unusual activity
-
Check for errors in Gitaly Sentry -
Check for errors in Gitaly Sentry -
Check for errors in GitLab Sentry
Success?
-
Close this issue and mark the ~Conversation as ~"Migration:Opt-In"