Collect Node Level logs like SSH,AUTH,AUDIT

Closes #2177 (closed)

The purpose of this MR is to introduce HostTailer pods to collect Node level logs like ssh, auth and audit. These HostTailer pods also configured as per OS and uses nodeSelector to target node of specific OS so that it could accurately target right path and services to collect logs.

For example there is difference in ssh service name i.e sshd in opensuse and ssh in ubuntu. The way we collect auth logs also varies. For example in case of ubuntu it can be fetched from /var/log/auth/auth.log and in case of opensuse it is /var/log/messages. To adjust to these mix mode this MR is introducing different type of HostTailer resource which is configured as per OS i.e opensuse and ubuntu and uses nodeSelector to target nodes of specific OS type so that right HostTailer pods lands on right node installed with specific OS. This way hosttailer pods will always land on right node and work correctly without going to CrashLoopBackOff.( Which was the issue due to absence of specific service or path in particular OS)

Note: This capability to land pod on nodes of specific OS is due to presense of os labels in nodes intoduced by MR(sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster!802 (merged))

From grafana we can filter out logs like as follows

Filtered sshd_logs

Parsed sshd logs

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	capd, capo, capm3
🚀	Bootstrap Provider	kubeadm (alias kadm), rke2
🐧	Node OS	ubuntu, suse
🛠️	Deployment Options	light-deploy, oci, ha, misc
🎬	Pipeline Scenarios	rolling-update, mgmt-rolling-update, k8s-upgrade, sylva-upgrade, sylva-upgrade-from-x.x.X, simple-update, preview, nightly

• 🎬preview ☁️capd 🚀kadm 🐧ubuntu 🛠️oci

• 🎬preview ☁️capo 🚀rke2 🐧suse

• 🎬preview ☁️capm3 🚀rke2 🐧ubuntu

• ☁️capd 🚀kubeadm 🛠️light-deploy 🐧ubuntu

• ☁️capd 🚀rke2 🛠️oci,light-deploy 🐧suse

• ☁️capo 🚀rke2 🛠️logging 🐧suse

• ☁️capo 🚀kadm 🛠️oci 🐧ubuntu

• ☁️capo 🚀rke2 🎬rolling-update 🛠️ha 🐧ubuntu

• ☁️capo 🚀kadm 🎬k8s-upgrade 🐧ubuntu

• ☁️capo 🚀rke2 🎬mgmt-rolling-update 🛠️ha,misc 🐧suse

• ☁️capo 🚀rke2 🎬sylva-upgrade 🛠️misc,ha,logging 🐧ubuntu

• ☁️capo 🚀rke2 🎬sylva-upgrade 🛠️misc,ha,logging 🐧suse

• ☁️capo 🚀rke2 🛠️misc,ha,logging 🐧ubuntu

• ☁️capo 🚀rke2 🛠️misc,ha,logging 🐧suse

• ☁️capm3🚀rke2 🛠️misc,ha,logging 🐧suse

• ☁️capm3🚀rke2 🛠️misc,ha,logging 🐧ubuntu

• ☁️capm3🚀kadm 🛠️misc,ha,logging 🐧suse

• ☁️capm3🚀kadm 🛠️misc,ha,logging 🐧ubuntu

• ☁️capm3🚀rke2 🛠️ha,logging 🐧suse

Global config for deployment pipelines

• autorun pipelines
• allow failure on pipelines
• record sylvactl events

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.