Have CI power off all real-bmh hosts and reset their BMCs with sylvactl before deploying
What does this MR do and why?
A BMC restart operation is needed at times for one of the real-bmh CI servers to get Ready (had 3 x ☁capm3 🚀rke2 🏗️real-bmh 🛠ha 🐧suse failures with dell-r360-165 stuck in Inspecting, e.g.). I was initially proposing a curl BMC reset for that particular server before @jonathang pointed out sylvactl's ability to interact with BMC and suggested to run this for all BM servers.
We also hide the server BMC IP and username from the publicly exposed logs.
Related reference(s)
Test coverage
CI run went fine and for this part it logged:
$ # Apply private values # collapsed multi-line command
:
===============================
Resetting BMC for dell-r660-168
Shutting down system System.Embedded.1...
waiting for system System.Embedded.1 to shut down
System System.Embedded.1 is now shut down
Resetting manager iDRAC.Embedded.1...
Resetting manager iDRAC.Embedded.1 with GracefulRestart...
Manager iDRAC.Embedded.1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl20g11-166
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl360g11-169
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for hp-dl360g11-170
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for le-sr250v3-167
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for le-sr630v3-171
Shutting down system 1...
waiting for system 1 to shut down
System 1 is now shut down
Resetting manager 1...
Resetting manager 1 with GracefulRestart...
Manager 1 reset command successful with GracefulRestart
===============================
Resetting BMC for dell-r360-165
Shutting down system System.Embedded.1...
waiting for system System.Embedded.1 to shut down
System System.Embedded.1 is now shut down
Resetting manager iDRAC.Embedded.1...
Resetting manager iDRAC.Embedded.1 with GracefulRestart...
Manager iDRAC.Embedded.1 reset command successful with GracefulRestart
All BMCs are backCI configuration
Below you can choose test deployment variants to run in this MR's CI.
Click to open to CI configuration
Legend:
| Icon | Meaning | Available values |
|---|---|---|
| Infra Provider | capd, capo, capm3 |
|
| Bootstrap Provider | kubeadm (alias kadm), rke2, okd, ck8s |
|
| Node OS | ubuntu, suse, na, leapmicro |
|
| Deployment Options | Deployment option list and description | |
| Pipeline Scenarios | Available scenario list and description | |
| Enabled units | Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type |
|
| Disabled units | Any available units name, by default apply to management and workload cluster. Can be prefixed by mgmt: or wkld: to be applied only to a specific cluster type |
|
| Target platform | Can be used to select specific deployment environment Available platform list and description | |
| Pipeline control | autorun, manual or blocking. Can be used to override global config and start a deployment pipeline the required way |
-
🎬 preview☁️ capd🚀 kadm🐧 ubuntu -
🎬 preview☁️ capo🚀 rke2🐧 suse -
🎬 preview☁️ capm3🚀 rke2🐧 ubuntu -
☁️ capd🚀 kadm🛠️ light-deploy🐧 ubuntu -
☁️ capd🚀 rke2🛠️ light-deploy🐧 suse -
☁️ capo🚀 rke2🐧 suse -
☁️ capo🚀 rke2🐧 leapmicro -
☁️ capo🚀 kadm🐧 ubuntu -
☁️ capo🚀 kadm🐧 ubuntu🟢 neuvector,mgmt:harbor -
☁️ capo🚀 rke2🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capo🚀 kadm🎬 wkld-k8s-upgrade🐧 ubuntu -
☁️ capo🚀 rke2🎬 rolling-update-no-wkld🛠️ ha🐧 suse -
☁️ capo🚀 rke2🎬 sylva-upgrade🛠️ ha🐧 ubuntu -
☁️ capo🚀 rke2🎬 sylva-upgrade-from-1.6.x🛠️ ha,misc🐧 ubuntu -
☁️ capo🚀 rke2🛠️ ha,misc🐧 ubuntu -
☁️ capo🚀 rke2|okd🎬 no-update🐧 ubuntu|na -
☁️ capo🚀 rke2🛠️ misc🐧 ubuntu🟢 mgmt:harbor🔴 neuvector -
☁️ capo🚀 rke2🛠️ ha,misc,openbao🐧 suse -
☁️ capo🚀 rke2🐧 suse🎬 upgrade-from-prev-tag -
☁️ capm3🚀 rke2🐧 suse -
☁️ capm3🚀 kadm🐧 ubuntu -
☁️ capm3🚀 rke2🛠️ external-openbao-no-tls🐧 ubuntu -
☁️ capm3🚀 rke2🛠️ external-openbao🐧 suse -
☁️ capm3🚀 kadm🛠️ external-openbao,openbao-namespace🐧 ubuntu -
☁️ capm3🚀 ck8s🐧 ubuntu -
☁️ capm3🚀 kadm🎬 rolling-update-no-wkld🛠️ ha,misc🐧 ubuntu -
☁️ capm3🚀 rke2🎬 wkld-k8s-upgrade🛠️ ha🐧 suse -
☁️ capm3🚀 kadm🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capm3🚀 rke2🎬 upgrade-from-prev-release-branch🛠️ ha🐧 suse -
☁️ capm3🚀 rke2🛠️ misc,ha🐧 suse -
☁️ capm3🚀 rke2🎬 sylva-upgrade🛠️ ha,misc🐧 suse -
☁️ capm3🚀 kadm🎬 rolling-update🛠️ ha🐧 suse -
☁️ capm3🚀 ck8s🎬 rolling-update🛠️ ha🐧 ubuntu -
☁️ capm3🚀 rke2|okd🎬 no-update🐧 ubuntu|na -
☁️ capm3🚀 rke2🐧 suse🎬 upgrade-from-release-1.5 -
☁️ capm3🚀 rke2🐧 suse🎬 upgrade-to-main -
☁️ capm3🚀 rke2🛠️ ha🐧 suse🏗️ real-bmh
Global config for deployment pipelines
- autorun pipelines
- allow failure on pipelines
- record sylvactl events
Notes:
- Enabling
autorunwill make deployment pipelines to be run automatically without human interaction - Disabling
allow failurewill make deployment pipelines mandatory for pipeline success. - if both
autorunandallow failureare disabled, deployment pipelines will need manual triggering but will be blocking the pipeline
Be aware: after configuration change, pipeline is not triggered automatically.
Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.