Have CI power off all real-bmh hosts and reset their BMCs with sylvactl before deploying

What does this MR do and why?

A BMC restart operation is needed at times for one of the real-bmh CI servers to get Ready (had 3 x ☁capm3 🚀rke2 🏗️real-bmh 🛠ha 🐧suse failures with dell-r360-165 stuck in Inspecting, e.g.). I was initially proposing a curl BMC reset for that particular server before @jonathang pointed out sylvactl's ability to interact with BMC and suggested to run this for all BM servers.
We also hide the server BMC IP and username from the publicly exposed logs.

Test coverage

CI run went fine and for this part it logged:

$ # Apply private values # collapsed multi-line command
:
===============================
Resetting BMC for dell-r660-168
Shutting down system System.Embedded.1...
waiting for system System.Embedded.1 to shut down
System System.Embedded.1 is now shut down
Resetting manager iDRAC.Embedded.1...
Resetting manager iDRAC.Embedded.1 with GracefulRestart...
Manager iDRAC.Embedded.1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl20g11-166
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl360g11-169
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl360g11-170
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for le-sr250v3-167
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for le-sr630v3-171
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for dell-r360-165
Shutting down system System.Embedded.1...
waiting for system System.Embedded.1 to shut down
System System.Embedded.1 is now shut down
Resetting manager iDRAC.Embedded.1...
Resetting manager iDRAC.Embedded.1 with GracefulRestart...
Manager iDRAC.Embedded.1 reset command successful with GracefulRestart
All BMCs are back

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon Meaning Available values
☁️ Infra Provider capd, capo, capm3
🚀 Bootstrap Provider kubeadm (alias kadm), rke2, okd, ck8s
🐧 Node OS ubuntu, suse, na, leapmicro
🛠️ Deployment Options Deployment option list and description
🎬 Pipeline Scenarios Available scenario list and description
🟢 Enabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🔴 Disabled units Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type
🏗️ Target platform Can be used to select specific deployment environment Available platform list and description
⚡ Pipeline control autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way
  • 🎬 preview ☁️ capd 🚀 kadm 🐧 ubuntu

  • 🎬 preview ☁️ capo 🚀 rke2 🐧 suse

  • 🎬 preview ☁️ capm3 🚀 rke2 🐧 ubuntu

  • ☁️ capd 🚀 kadm 🛠️ light-deploy 🐧 ubuntu

  • ☁️ capd 🚀 rke2 🛠️ light-deploy 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 leapmicro

  • ☁️ capo 🚀 kadm 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🐧 ubuntu 🟢 neuvector,mgmt:harbor

  • ☁️ capo 🚀 rke2 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 kadm 🎬 wkld-k8s-upgrade 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 rolling-update-no-wkld 🛠️ ha 🐧 suse

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade 🛠️ ha 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🎬 sylva-upgrade-from-1.6.x 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capo 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capo 🚀 rke2 🛠️ misc 🐧 ubuntu 🟢 mgmt:harbor 🔴 neuvector

  • ☁️ capo 🚀 rke2 🛠️ ha,misc,openbao 🐧 suse

  • ☁️ capo 🚀 rke2 🐧 suse 🎬 upgrade-from-prev-tag

  • ☁️ capm3 🚀 rke2 🐧 suse

  • ☁️ capm3 🚀 kadm 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🛠️ external-openbao-no-tls 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🛠️ external-openbao 🐧 suse

  • ☁️ capm3 🚀 kadm 🛠️ external-openbao,openbao-namespace 🐧 ubuntu

  • ☁️ capm3 🚀 ck8s 🐧 ubuntu

  • ☁️ capm3 🚀 kadm 🎬 rolling-update-no-wkld 🛠️ ha,misc 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 wkld-k8s-upgrade 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2 🎬 upgrade-from-prev-release-branch 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🛠️ misc,ha 🐧 suse

  • ☁️ capm3 🚀 rke2 🎬 sylva-upgrade 🛠️ ha,misc 🐧 suse

  • ☁️ capm3 🚀 kadm 🎬 rolling-update 🛠️ ha 🐧 suse

  • ☁️ capm3 🚀 ck8s 🎬 rolling-update 🛠️ ha 🐧 ubuntu

  • ☁️ capm3 🚀 rke2|okd 🎬 no-update 🐧 ubuntu|na

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-from-release-1.5

  • ☁️ capm3 🚀 rke2 🐧 suse 🎬 upgrade-to-main

  • ☁️ capm3 🚀 rke2 🛠️ ha 🐧 suse 🏗️ real-bmh

Global config for deployment pipelines

  • autorun pipelines
  • allow failure on pipelines
  • record sylvactl events

Notes:

  • Enabling autorun will make deployment pipelines to be run automatically without human interaction
  • Disabling allow failure will make deployment pipelines mandatory for pipeline success.
  • if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited by Bogdan-Adrian Burciu

Merge request reports

Loading
Loading