Fine tune limits, requests, replicas, and puma settings for API service traffic in kubernetes
This issue will be to discuss how we will tune K8s and Puma for API traffic, when we move production workloads to the Kubernetes cluster.
We should be basing this on our experience from #1067 (closed) and collect the necessary API performance data.
Analysis
- workhorse memory usage: Thanos
- webservice memory usage: Thanos
- TODO: Other memory observation capabilities
- nginx memory usage: Thanos
- workhorse CPU usage: Thanos
- webservice CPU usage: Thanos
Testing Impact
- Saturation: Dashboard This should go down
- Apdex: Dashboard This should stay at or above our current status quo
- Observation of node state and pod state after the changes, currently this is done manually by observing the % of requests/limits/hpa state on a per deployment/node
- All of this done using 2 methods:
kubectl describe <object>
- https://console.cloud.google.com/home/dashboard
- All of this done using 2 methods:
Goals
- Prior to the migration we were running
36
c2-standard-16
VMs in the API fleet - After the migration with over-provisioning we are running between
80
and100
c2-standard-16
nodes in the API pool
The API fleet was under-provisioned prior to the migration, we would like to aim for a 20-50% increase in VMs which would be between 40-50 c2-standard-16
nodes.
Metrics to track for compute resource consumption
-
Nodes per region (
us-east1
cluster is canary API) -
Pods per region (
us-east1
cluster is canary API) - NGINX pods per region
-
default
node pool nodes per region (Thedefault
node pool is where we are running NGINX pods)
Timeline
- 2021-05-25: gitlab-com/gl-infra/k8s-workloads/gitlab-com!886 (merged) This increased the
targetAverageValue
for webservice API HPA from1600m
to2200m
. - 2021-05-26: gitlab-com/gl-infra/k8s-workloads/gitlab-com!883 (merged) This increased the
targetMemoryUtilizationPercentage
for NGINX from 50% to 70%- Expected results:
-
Decrease in the number of pods (we only saw a drop on canary)
-
- Expected results:
- 2021-05-27: gitlab-com/gl-infra/k8s-workloads/gitlab-com!890 (merged) This increases the
targetMemoryUtilizationPercentage
for NGINX from 70 to 75%, globally and increases the memory requests onus-east1-b
and canary from 500MB to 1GB.- Expected results:
-
We should see a small decrease in the number of pods in all envs -
We should see a bigger decrease in the number of pods in us-east1-b
-
We should see a decrease in the number of nodes in the default
node pool inus-east1-b
-
With fewer pods in us-east1-b
we should see our CPU utilization closer to our requests console link -
Applied to all clusters with gitlab-com/gl-infra/k8s-workloads/gitlab-com!890 (merged) - Decreased our node count in the
default
pool from 23 to 13 nodes
-
- Expected results:
- 2021-05-27: gitlab-com/gl-infra/k8s-workloads/gitlab-com!893 (merged) This increases the cpu HPA target for API and lowers the min replicas in a single cluster
- Expected results:
-
We should see a noticeable decrease in the number of pods in us-east1-b
-
We should also see a reduction in the number of nodes in us-east1-b
compared to other regions -
CPU should be closer to our requests console link
-
- Expected results:
- 2021-05-31 13:30 UTC - We needed to revert the previous change as it led to degraded latencies and alerts for the cny workhorse APDEX (incident)
- 2021-06-01 10:23 UTC - Lower the API minReplicas count on cny only from 5 to 3: gitlab-com/gl-infra/k8s-workloads/gitlab-com!901 (merged)
- Expected Results:
-
Lowered min count of Pods at low traffic - we never got below 8 Pods, which indicates that we need to increase the HPA CPU target if we want to run with less pods.
-
Lowered min count of Nodes at low traffic - dito
-
No APDEX drops during deployments -
No Ruby thread contention saturation
-
- Expected Results:
- 2021-06-01 - Raise the CPU target on cny only from 2200 (55%) to 2800 (70%) (instead of 3200 (80%): gitlab-com/gl-infra/k8s-workloads/gitlab-com!902 (merged)
- Expected Results:
-
CPU should be closer to our requests -
Lowered count of Pods -
Lowered min count of Nodes -
No APDEX drop -
No Ruby thread contention saturation - There is a moderate increase, but mostly staying below 80%.
-
- Expected Results:
- 2021-06-02 - Apply the same to us-east1-b gprd cluster: gitlab-com/gl-infra/k8s-workloads/gitlab-com!906 (merged)
- Expected Results:
-
CPU should be closer to our requests -
Lowered count of Pods -
Lowered count of Nodes -
No considerable APDEX drop -
No considerable Ruby thread contention saturation
-
- Expected Results:
- 2021-06-03 13:21 UTC - Apply the same to all clusters: gitlab-com/gl-infra/k8s-workloads/gitlab-com!909 (merged)
- Expected Results:
-
CPU should be closer to our requests -
Lowered count of Pods Interestingly we first saw pods and nodes spike up when changing the CPU target and then go down after that.
-
Lowered count of Nodes -
No considerable APDEX drop -
No considerable Ruby thread contention saturation
-
- Expected Results:
- 2021-06-04 02:18 - Changed from 2800 to 2600 CPU target in grpd and cny because of big APDEX drop during canary deploy: gitlab-com/gl-infra/k8s-workloads/gitlab-com!912 (merged)
- 2021-06-04 10:15 - set minReplicas to 16 for API cny to reduce node scaling
- this is keeping us at a minimum node count of 6, which should eliminate node scaling mostly in cny.
- 2021-06-07 - Reduce CPU requests on cny/single cluster: #1592 (comment 594818884)
- Expected Results:
-
Lowered count of Nodes (more pods per node) -
Higher number of API Pods per Node Thanos -
Less node scaling during deployments -
No considerable APDEX drop -
No considerable Ruby thread contention saturation
-
- Expected Results:
- 2021-06-08 - Reduce CPU requests on cny/single cluster: #1592 (comment 595214437)
- Expected Results:
-
Lowered count of Nodes (more pods per node) -
Higher number of API Pods per Node Thanos -
Less node scaling during deployments -
No considerable APDEX drop -
No considerable Ruby thread contention saturation
-
- Expected Results:
- 2021-06-09 - Reduce CPU requests on cny; increase worker count on a single cluster: #1592 (comment 597182353)
- Expected Results:
-
Lowered count of Nodes in canary (more pods per node) -
Observe APDEX -
Observe Ruby thread contention saturation -
Higher utilization of RAM on cluster b
-
- Expected Results:
- 2021-06-16 - Manual tuning of the API service in cluster b
- We reduced node count down to 17 during load, and pod count down to HPA minimums: production#4896 (comment 603253603)
- Saw a reduction in Pods and Nodes; however during the next day, heavy traffic reduced apdex and saturated memory
- 2021-06-17 - lower average CPU target to schedule more Pods
- Conversation: #1592 (comment 604071340)
Results
Start
API
- We had our minimum Pod counts set to 48, but we were constantly running above 50 during low traffic periods, though we hovered around the upper 60's during higher load times. This provided us with 840 workers. This is 30% more than what we were running VM's.
- Our node pool was using anywhere between 35 and 45 nodes per zone, which is 70% more than what we were running on VM's.
NGINX
- Once we sent traffic to the API, we hit our Pod Maximum of 100 immediately.
- This drove up the default node pool node counts by a marginal amount, roughly 3 nodes per zone
End
API
- We tweaked worker counts per Pod and various resource configurations which has allowed us to run at minimum 27 Pods, and we've seen high load utilize around 45 pods. This provides us with around still 810 workers, but resources are more well utilized.
- Our node pools are now vary between 14 and 25 nodes. So at worst we run 50% more node and during off peak times, we run 6 more nodes than we previously used too
NGINX
- We now hover anywhere between 16 and 25 pods pending load
- Node counts hover around 3 or 4 per zone
Dashboard Snippet
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 8,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.2.0",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "count(kube_pod_labels{label_type=~\"api\"}) by (region,type)",
"interval": "",
"legendFormat": "{{region}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "API Pod Counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"hiddenSeries": false,
"id": 6,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.2.0",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "count(kube_pod_labels{label_app=\"nginx-ingress\"}) by (region,type)",
"interval": "",
"legendFormat": "{{region}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "NGINX Ingress Controller Pod Counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 8
},
"hiddenSeries": false,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.2.0",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "count(kube_node_labels{label_type=~\"api\"}) by (region,type)",
"interval": "",
"legendFormat": "{{region}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "API Node Counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fieldConfig": {
"defaults": {
"custom": {}
},
"overrides": []
},
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 8
},
"hiddenSeries": false,
"id": 4,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "7.2.0",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "count(kube_node_labels{label_type=~\"default\"}) by (region,type)",
"interval": "",
"legendFormat": "{{region}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Default Node Counts",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": false,
"schemaVersion": 26,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "2021-05-21T00:00:00.000Z",
"to": "2021-05-21T23:59:59.000Z"
},
"timepicker": {},
"timezone": "",
"title": "New dashboard",
"uid": null,
"version": 0
}
Edited by John Skarbek