kubeadm: increase Etcd election-timeout/heartbeat-interval (align with RKE2)

What does this MR do and why?

This MR aims to solve the issues that have been encountered with the latency of etcd that can be observed on management clusters during the runs of the CI. The values which are directly targeted are heartbeat-interval and election-timeout as the current values are not suited for our CI infrastructure. WAL fsync p95 (and p99 also) latency can be seen in most of the ASCII graphs found under management-cluster-dump/prometheus-extracts .

At the moment, on kadm etcd is using the default values with heartbeat-interval=100 and election-timeout=1000 which are designed more for low-latency local networks and disks. This MR sets the timeout to 5000 and the heartbeat-interval to 500 in order to allow for a more robust approach over the latency issue.

Additional point:

After a short test that I have run today, I have managed to recreate the spikes of kadm mgmt cluster upgrade locally. They did not lead to a timeout as in CI, but the behavior seem to be the same (oY expressed in seconds):

image.png

Source issue: #4579

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu
  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse
  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu
  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu
  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse
  • ☁️ capo 🚀 rke2 🐧 suse
  • ☁️ capo 🚀 rke2 🐧 leapmicro
  • ☁️ capo 🚀 kadm 🐧 ubuntu
  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor
  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu
  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu
  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse
  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu
  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.6.x 🛠️ ha,misc 🐧 ubuntu
  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu
  • ☁️ capo 🚀 rke2 🛠️ misc 🐧 ubuntu 🟢 mgmt:harbor 🔴 neuvector
  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao 🐧 suse
  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag
  • ☁️ capm3 🚀 rke2 🐧 suse
  • ☁️ capm3 🚀 kadm 🐧 ubuntu
  • ☁️ capm3 🚀 ck8s 🐧 ubuntu
  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu
  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse
  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu
  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse
  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse
  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha,misc 🐧 suse
  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse
  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu
  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na
  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5
  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

Global config for deployment pipelines

  • autorun pipelines

  • allow failure on pipelines

  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Tiberiu Mihai

Merge request reports

Loading
Loading