GitLab.com KAS early-adopters (beta) programme
What is the kas address for GitLab.com
wss://kas.gitlab.com
What
Rollout the :kubernetes_agent_on_gitlab_com
feature flag
Owners
- Team: ~"group::configure"
- Most appropriate slack channel to reach out to:
#s_configure
- Best individual to reach out to: @tkuah
Expectations
What are we expecting to happen?
We can deploy KAS service, and only projects in the allowlist will be allowed to use this feature
What might happen if this goes wrong?
- Clients who are not on the allowlist will see similar in their
agentk
logs:
│ {"level":"warn","time":"2021-02-07T20:06:24.920Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = PermissionDenied desc = forbidden"} │
│ {"level":"error","time":"2021-02-07T20:06:25.387Z","msg":"GetObjectsToSynchronize.Recv failed","project_id":"tkuah/kas-agent-test","error":"rpc error: code = PermissionDenied desc = forbidden"}
KAS service degrades, or fails (https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/kas/README.md)
What can we monitor to detect problems with this?
https://dashboards.gitlab.net/d/kas-main/kas-overview?orgId=1
Beta groups/projects
If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
NOTE: Only public, and GitLab Premium projects (~180K) are eligible
-
GitLab internal testing projects -
Beta list of users
Success criteria to rollout to GA
-
Rate limiting for KAS agents is active - KAS / GitLab.com handles progressively increased load well.
Roll Out Steps
-
Enable on staging for selected project ( /chatops run feature set --project=tkuah/kas-agent-test kubernetes_agent_on_gitlab_com true --staging
-
Enable on staging ( /chatops run feature set kubernetes_agent_on_gitlab_com true --staging
) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab kubernetes_agent_on_gitlab_com true
) -
Progressive rollout 1% (up to ~1,800 GitLab Premium projects), /chatops run feature set kubernetes_agent_on_gitlab_com 1 --actors
-
Progressive rollout 10% (up to ~18,000 GitLab Premium projects), /chatops run feature set kubernetes_agent_on_gitlab_com 10 --actors
-
Progressive rollout 20% -
Progressive rollout 50% -
Progressive rollout 100% -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#production
mention@sre-oncall
and@release-managers
. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production
(/chatops run feature set kubernetes_agent_on_gitlab_com true
) -
Cross post chatops Slack command to #support_gitlab-com
(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #production
channel
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set --project=gitlab-org/gitlab kubernetes_agent_on_gitlab_com false
Edited by Viktor Nagy (GitLab)