Update Sylva-elements (main) (minor)

💡 This MR is a special MR arising in the context of the for of release-1.

tools/deps-project-tool.py has been used to generate release branches in sylva-elements repos, and fresh new x.(Y+1).0 tags have been created there on the same commit to which sylva-core was pointing via an x.y.Z tag.

Hence this MR introduces no code, config or any behavioral change.

More context: Creating a New Release Branch for Sylva core


This MR contains the following updates:

Package Type Update Change
container-images/sylva-toolbox minor v1.10.10 → v1.11.0
gitlab.com/sylva-projects/sylva-elements/heat-operator Kustomization minor 0.5.2 → 0.6.0
gitlab.com/sylva-projects/sylva-elements/misc-controllers-suite Kustomization minor v1.4.3 → v1.5.0
gitlab.com/sylva-projects/sylva-elements/sylva-units-operator Kustomization minor v1.4.5 → v1.5.0
gitlab.com/sylva-projects/sylva-elements/workload-cluster-operator Kustomization minor v0.14.6 → v0.15.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas.git minor 1.5.0 → 1.6.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/keycloak-user-management.git minor 0.4.0 → 0.5.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/nmstate-resources.git minor 1.1.0 → 1.2.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/os-image-server.git minor 2.8.1 → 2.9.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/rancher-roles-management.git minor 1.3.0 → 1.4.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-alertmanager-resources.git minor 0.4.0 → 0.5.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster.git minor 0.15.22 → 0.16.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-dashboards.git minor 1.0.2 → 1.1.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-junos-config.git minor 0.1.0 → 0.2.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-logging-flows.git minor 0.4.1 → 0.5.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git minor 0.4.2 → 0.5.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-snmp-resources.git minor 0.4.0 → 0.5.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules.git minor 0.5.4 → 0.6.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sync-openstack-images.git minor 0.9.4 → 0.10.0
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/workload-team-defs.git minor 0.8.7 → 0.9.0
registry.gitlab.com/sylva-projects/sylva-elements/container-images/ci-image minor v1.10.4 → v1.11.0
registry.gitlab.com/sylva-projects/sylva-elements/container-images/ci-image/ci-playwright-image image minor v1.10.4 → v1.11.0
registry.gitlab.com/sylva-projects/sylva-elements/container-images/kube-job minor v1.7.3 → v1.8.0
registry.gitlab.com/sylva-projects/sylva-elements/heat-operator minor 0.5.2 → 0.6.0
registry.gitlab.com/sylva-projects/sylva-elements/heat-operator Kustomization minor 0.5.2 → 0.6.0
registry.gitlab.com/sylva-projects/sylva-elements/misc-controllers-suite minor v1.4.3 → v1.5.0
registry.gitlab.com/sylva-projects/sylva-elements/misc-controllers-suite Kustomization minor v1.4.3 → v1.5.0
registry.gitlab.com/sylva-projects/sylva-elements/sylva-units-operator minor v1.4.5 → v1.5.0
registry.gitlab.com/sylva-projects/sylva-elements/sylva-units-operator Kustomization minor v1.4.5 → v1.5.0
registry.gitlab.com/sylva-projects/sylva-elements/workload-cluster-operator minor v0.14.6 → v0.15.0
registry.gitlab.com/sylva-projects/sylva-elements/workload-cluster-operator Kustomization minor v0.14.6 → v0.15.0
sylva-library minor 0.10.2 → 0.11.0

⚠️ Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.


Release Notes

sylva-projects/sylva-elements/container-images/sylva-toolbox (container-images/sylva-toolbox)

v1.11.0: sylva-toolbox: v1.11.0

Compare Source

Merge Requests integrated in this release

4 merge requests were integrated in this repo between v1.10.9 and v1.11.0. These notes don't account for the MRs merged in secondary repos.

Sylva-units framework

  • Update dependency sylva-projects/sylva-elements/sylvactl to v0.15.5 !703 renovate
  • Update dependency sylva-projects/sylva-elements/sylva-units-operator to v1.4.5 !697 renovate

Logging

Internal tooling

Contributors

1 person contributed.

Jonathan Gayvallet

sylva-projects/sylva-elements/heat-operator (gitlab.com/sylva-projects/sylva-elements/heat-operator)

v0.6.0: heat-operator: 0.6.0

Compare Source

Merge Requests integrated in this release

16 merge requests were integrated in this repo between 0.5.1 and 0.6.0. These notes don't account for the MRs merged in secondary repos.

Other dependency upgrades
  • Update module k8s.io/apimachinery to v0.35.7 !367
  • Update module github.com/go-logr/logr to v1.4.4 !369
  • Update GO modules dependencies to v0.35.7 !371
  • Update GO modules dependencies !357
  • Update GO modules dependencies !374
CI
Internal tooling
Contributors

1 person contributed.

Loic Nicolle

sylva-projects/sylva-elements/misc-controllers-suite (gitlab.com/sylva-projects/sylva-elements/misc-controllers-suite)

v1.5.0: misc-controllers-suite: v1.5.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between v1.4.2 and v1.5.0. These notes don't account for the MRs merged in secondary repos.

Internal tooling
Contributors

1 person contributed.

Loic Nicolle

sylva-projects/sylva-elements/sylva-units-operator (gitlab.com/sylva-projects/sylva-elements/sylva-units-operator)

v1.5.0: sylva-units-operator: v1.5.0

Compare Source

Merge Requests integrated in this release

6 merge requests were integrated in this repo between v1.4.4 and v1.5.0. These notes don't account for the MRs merged in secondary repos.

Other dependency upgrades
  • Update module k8s.io/apimachinery to v0.35.9 !675
  • Update pre-commit hook crate-ci/typos to v1.50.2 !677
CI
Internal tooling
Contributors

1 person contributed.

Loic Nicolle

sylva-projects/sylva-elements/workload-cluster-operator (gitlab.com/sylva-projects/sylva-elements/workload-cluster-operator)

v0.15.0: workload-cluster-operator: v0.15.0

Compare Source

Merge Requests integrated in this release

11 merge requests were integrated in this repo between v0.14.5 and v0.15.0. These notes don't account for the MRs merged in secondary repos.

Sylva-units framework
Workload cluster
  • Ensure overriding values do not override the one defined by the CR !460 (issues: #​47)
Other dependency upgrades
CI
Internal tooling
Contributors

2 people contributed.

Loic Nicolle, Vladimir Braquet

sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas.git)

v1.6.0: capo-contrail-bgpaas: 1.6.0

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between 1.1.3 and 1.6.0. These notes don't account for the MRs merged in secondary repos.

Networking

  • fix missing heatstack fields !145

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.0 !144 renovate

Contributors

1 person contributed.

Thomas Morin

sylva-projects/sylva-elements/helm-charts/keycloak-user-management (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/keycloak-user-management.git)

v0.5.0: keycloak-user-management: 0.5.0

Compare Source

Merge Requests integrated in this release

4 merge requests were integrated in this repo between 0.3.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Change policy for roleIdRef !17

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.2 renovate !14 !15
  • Update dependency to-be-continuous/gitleaks to v2.11.0 !16 renovate

Contributors

1 person contributed.

Bogdan Antohe

sylva-projects/sylva-elements/helm-charts/nmstate-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/nmstate-resources.git)

v1.2.0

Compare Source

sylva-projects/sylva-elements/helm-charts/os-image-server (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/os-image-server.git)

v2.9.0: os-image-server: 2.9.0

Compare Source

Merge Requests integrated in this release

9 merge requests were integrated in this repo between 2.7.2 and 2.9.0. These notes don't account for the MRs merged in secondary repos.

Other dependency upgrades

  • Update Helm chart sylva-library to v0.10.1 !209

CI

Contributors

0 person contributed.

sylva-projects/sylva-elements/helm-charts/rancher-roles-management (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/rancher-roles-management.git)

v1.4.0: rancher-roles-management: 1.4.0

Compare Source

Merge Requests integrated in this release

7 merge requests were integrated in this repo between 1.1.4 and 1.4.0. These notes don't account for the MRs merged in secondary repos.

Cluster API

  • Replace additionalProperties with unevaluatedProperties !26 (issues: #​7)

Other

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.5 renovate !21 !24 !27 !29
  • Update dependency to-be-continuous/gitleaks to v2.11.0 !22 renovate

Contributors

2 people contributed.

Priya Goyal, Samuel Bartel

sylva-projects/sylva-elements/helm-charts/sylva-alertmanager-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-alertmanager-resources.git)

v0.5.0: sylva-alertmanager-resources: 0.5.0

Compare Source

Merge Requests integrated in this release

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.37
    • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.34 !59 renovate

    • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.35 !60 renovate

    • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.36 !62 renovate

    • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.37 !68 renovate

  • Update dependency renovate-bot/renovate-runner to v20
    • Update dependency renovate-bot/renovate-runner to v19.94.0 !61 renovate

    • Update dependency renovate-bot/renovate-runner to v19.107.1 !64 renovate

    • Update dependency renovate-bot/renovate-runner to v19.111.4 !66 renovate

    • Update dependency renovate-bot/renovate-runner to v20 !67 renovate

  • Update dependency to-be-continuous/gitleaks to v2.7.0 !63 renovate

Other

  • Update VictorOps entity_display_name format !65
  • Add snmp-notifier helm chart !69

Contributors

Alin H

sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster.git)

v0.16.0: sylva-capi-cluster: 0.16.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between 0.15.21 and 0.16.0. These notes don't account for the MRs merged in secondary repos.

Cluster API

  • rke2: remove RKE2 installation binaries after installation !1120 rke2 (issues: #​330)

RKE2

Contributors

1 person contributed.

Thomas Morin

sylva-projects/sylva-elements/helm-charts/sylva-dashboards (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-dashboards.git)

v1.1.0: sylva-dashboards: 1.1.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between 1.0.1 and 1.1.0. These notes don't account for the MRs merged in secondary repos.

Other

Contributors

1 person contributed.

Dragos Gerea

sylva-projects/sylva-elements/helm-charts/sylva-junos-config (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-junos-config.git)

v0.2.0

Compare Source

sylva-projects/sylva-elements/helm-charts/sylva-logging-flows (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-logging-flows.git)

v0.5.0: sylva-logging-flows: 0.5.0

Compare Source

Merge Requests integrated in this release

77 merge requests were integrated in this repo between 0.0.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Add 'home' field to Chart.yaml !70
  • Add license file !88

CI

Contributors

2 people contributed.

Loic Nicolle, Stephen Oresanya

sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git)

v0.5.0: sylva-prometheus-rules: 0.5.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between 0.4.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Rancher: Detect if fleet agent is still present !163

Contributors

1 person contributed.

Bogdan Antohe

sylva-prometheus-rules

Generate PrometheusRule objects for consumption by Prometheus

Overview

There are two mechanisms that control which rules are deployed

  1. createRules selects which directories are considered
  2. optional_rules selects which files in those directories are added to the Configmap

Rules overrides

.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/

If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other clusters are specified

This allows for rule overriding. Example:

createRules:
  allclusters: true
  management-cluster: true
alert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml

alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml
  • First the PrometheusRule with the flux, minio and health-alerts name from management-cluster are created.
  • Then health-alerts and dummy from allcluster are parsed. Since health-alerts is already applied from mananagement-cluster it will not be applied again. dummy will be applied since it doesn't override anything

This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster

Rules activation

.Values.optional_rules controls which rules are enabled for optional components

Details about rules
alert-rules/allclusters/goldpinger.yaml
Alert Name For Severity Type Description
Goldpinger_Node_Unhealthy 5m critical network Goldpinger reports unhealthy nodes: "{{ $labels.goldpinger_instance }}"
Goldpinger_Cluster_Health_Unhealthy 5m critical network Goldpinger reports overall cluster health failure on node "{{ $labels.goldpinger_instance }}". This indicates a pod-to-pod connectivity failure or failure to reach goldpinger service or cluster VIP on port 6443.
alert-rules/allclusters/junos-exporter.yaml
Alert Name For Severity Type Description
JunOS_Down 5m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Down_All_in_Group 5m critical network All targets in cluster "{{ $labels.cluster_name }}" in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Rebooted 0m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" recently rebooted.
JunOS_Alarms_Major_Present 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Major alarms present. Value: {{ humanize $value }} alarms
JunOS_Alarms_Minor_Present 5m info network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Minor alarms present. Value: {{ humanize $value }} alarms
JunOS_BGP_Flapping 5m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}"" has flapping BGP sessions. Value: {{ humanize $value }}
JunOS_BGP_not_Up 5m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" BGP session with "{{ $labels.ip }}" is DOWN.
JunOS_Interface_Receive_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped incoming packets on {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 10m. Value: {{ humanize $value }}
JunOS_Interface_Receive_Errors 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for incoming packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Transmit_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in 10 minutes. Value: {{ humanize $value }}
JunOS_Interface_Transmit_Errors 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Physical_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has physical layer errors on "{{ $labels.name }}" ( {{ $labels.description }} ) interface.
JunOS_Interface_Oper_Status_Down 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Operational status on "{{ $labels.name }}" ( {{ $labels.description }} ) interface changed to Down.
JunOS_Power_Source_Down 5m info network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Power source "{{ $labels.item }}" status not OK. Value: {{ $labels.status }} status
JunOS_Routing_Engine_Status_NOK 30m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" routing engine {{ $labels.slot }} status not OK for more than 30m.
JunOS_Routing_Engine_High_CPU_Utilization 30m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" CPU usage for routing engine {{ $labels.slot }} is greater than 90% for more than 30m. Value: {{ humanize $value }}%
JunOS_Interface_Rate_Limit_Packets_Drop 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets due to rate limit on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Queues_Packets_Drop 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Flapping 0m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" interface "{{ $labels.name }}" ( {{ $labels.description }} ) in the last hour. Value: {{ humanize $value }} resets
JunOS_IPSec_Tunnels_Down 5m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has {{ $value }} IPSec tunnels DOWN out of {{ printf junos_ipsec_configured_tunnels{alias="%s"} $labels.alias
JunOS_High_Memory_Heap_Utilization 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" memory heap utilization has exceeded the threshold of 60% for more than 5 minutes. Value: {{ humanize $value }}%
alert-rules/allclusters/snmp-cisco-ncs.yaml
Alert Name For Severity Type Description
SNMP_Cisco_NCS_SNMP_get_failed 30m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_NCS_High_CPU_Utilization 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU utilization is greater than 90% for more than 30m. Value: {{ $value }}%
SNMP_Cisco_NCS_Interface_Flapping 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_NCS_SNMP_get_returned_too_few_results 1h info network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Oper_Status_Down 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_NCS_FRU_Power_Oper_Status_not_OK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_NCS_SensorStatus_Celsius_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_voltsDC_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_amperes_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dBm_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_other_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_rpm_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_unknown_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dB_NOK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_mplsL3VpnVrf_Oper_And_Admin_Status_Mismatch 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational and Admin status mismatch on "{{ $labels.mplsL3VpnVrfName }}".
alert-rules/allclusters/snmp-cisco-nexus.yaml
Alert Name For Severity Type Description
SNMP_Cisco_Nexus_SNMP_get_failed 30m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_Nexus_SNMP_get_returned_too_few_results 1h info network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Flapping 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Oper_Status_Down 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_Nexus_FRU_Power_Oper_Status_not_OK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_Nexus_High_CPU_Utilization 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU usage is greater than 90% for more than 30m. Value: {{ $value }}
SNMP_Cisco_Nexus_High_Memory_Utilization 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Memory usage is greater than 60% for more than 5m. Value: {{ $value }}
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Rcvr_Status_not_OK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg receive status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgRcvrStatus }} status
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Send_Status_not_OK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg send status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgSendStatus }} status
SNMP_Cisco_Nexus_Port_Storm_Control_Active 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Port Storm Control is active on interface "{{ $labels.ifDescr }}" in state "{{ $labels.cpscStatus }}"
SNMP_Cisco_Nexus_bgpPeerState_not_OK 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] bgpPeers connection to "{{ $labels.bgpPeerRemoteAddr }}" is in non established "{{ $labels.bgpPeerState }} state.
alert-rules/allclusters/snmp-dell-idrac.yaml
Alert Name For Severity Type Description
SNMP_DELL_iDRAC_globalSystemStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }}
SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC.
SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK 5m critical hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_controllerRollUpStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}.
SNMP_DELL_iDRAC_controllerComponentStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskState_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}.
SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}).
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL 5m critical hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_virtualDiskState_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}.
SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}.
SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}).
alert-rules/allclusters/snmp-fortinet-fortigate.yaml
Alert Name For Severity Type Description
SNMP_Fortinet_FortiGate_SNMP_get_failed 30m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Fortinet_FortiGate_SNMP_get_returned_too_few_results 1h info network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Flapping 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Drops 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Errors 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Oper_Status_Down 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Fortinet_FortiGate_HA_State_Not_Synchronized 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has "{{ $labels.fgHaStatsSyncStatus }}" for HA Index "{{ $labels.fgHaStatsIndex }}".
SNMP_Fortinet_FortiGate_HA_State_Changed 0m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA State has changed in the last 10 minutes.
SNMP_Fortinet_FortiGate_PowerSupply_Status_Alarm_Active 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power Supply "{{ $labels.fgHwSensorEntDescr }}" status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_Hardware_Sensor_Alarm_Active 5m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Hardware Sensor "{{ $labels.fgHwSensorEntDescr }} status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_High_CPU_Usage 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High CPU Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Memory_Usage 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Memory Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Disk_Usage 30m warning network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Disk Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_HaStatsSyncStatus_NOK 5m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Sync Status on unit "{{ $labels.fgHaStatsHostname }}" for more then 5m.
SNMP_Fortinet_FortiGate_HaStatus_Changed 0m error network Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Status changed to "{{ $labels.fgVdEntHaState }}" in the last 30m.
alert-rules/allclusters/snmp-hp-cpq.yaml
Alert Name For Severity Type Description
SNMP_HP_CPQ_Overall_Health_NOK 5m critical hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}"
SNMP_HP_CPQ_Event_Log_Condition_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}"
SNMP_HP_CPQ_CPU_Health_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}"
SNMP_HP_CPQ_Thermal_Condition_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}"
SNMP_HP_CPQ_Power_Supply_Condition_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}"
SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}"
SNMP_HP_CPQ_Controller_Overall_Condition_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}" status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}". This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator.
SNMP_HP_CPQ_iLO_LicenseKey_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation.
alert-rules/allclusters/snmp-lenovo-xcc.yaml
Alert Name For Severity Type Description
SNMP_Lenovo_XCC_systemHealthStat_NOK 5m critical hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }}
SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }}
SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }}
SNMP_Lenovo_XCC_memoryHealthStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }}
SNMP_Lenovo_XCC_fanHealthStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }}
SNMP_Lenovo_XCC_voltHealthStatus_NOK 5m warning hardware Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }}
alert-rules/management-cluster/flux.yaml
Alert Name For Severity Type Description
Flux_Kustomization_Failing 15m warning deployment Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_Kustomization_Failing_Cluster 60m warning deployment Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_HelmRelease_Failing 15m warning deployment Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Source_Failing 15m warning deployment Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Resource_Suspended 2h warning deployment Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended.
alert-rules/management-cluster/harbor.yaml
Alert Name For Severity Type Description
Harbor_Component_Status_NOK 5m warning tools Harbor component "{{ $labels.component }}" status is DOWN.
alert-rules/management-cluster/keycloak.yaml
Alert Name For Severity Type Description
Keycloak-CNPG_WAL_Disk_Usage_High 5m warning tools WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB
Keycloak-CNPG_Replica_Lagging_Behind_Primary 5m warning tools CNPG replica slot {{ $labels.slot_name }} on {{ $labels.pod }} is lagging {{ $value
alert-rules/management-cluster/loki.yaml
Alert Name For Severity Type Description
Loki-Write_PVC_Disk_Usage_Warning 5m warning logging Loki write PVC "{{ $labels.persistentvolumeclaim }}" in namespace "{{ $labels.namespace }}" disk usage is above 70% (current value: {{ printf "%.1f" $value }}%).
Loki-Write_PVC_Disk_Usage_Critical 5m critical logging Loki write PVC "{{ $labels.persistentvolumeclaim }}" in namespace "{{ $labels.namespace }}" disk usage is above 85% (current value: {{ printf "%.1f" $value }}%).
alert-rules/management-cluster/minio.yaml
Alert Name For Severity Type Description
MinIO_Cluster_Health_Status_NOK 5m critical monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK.
MinIO_Cluster_Health_Status_Unknown 5m critical monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages.
MinIO_Cluster_Disk_Offline 5m critical monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline.
MinIO_Cluster_Disk_Space_Usage 5m warning monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 20%.
MinIO_Cluster_Disk_Space_Usage 5m critical monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%.
MinIO_Cluster_Tolerance 5m critical monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes.
MinIO_Nodes_Offline 5m warning monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes.
alert-rules/management-cluster/rancher.yaml
Alert Name For Severity Type Description
Rancher_Token_Expires_Soon 1h warning rancher Token "{{ $labels.token_internal_name }}" for user "{{ $labels.user_email }}" expires in less than 10 days.
Rancher_Fleet_Agent_Detected 5m critical rancher Fleet Agent pod should be disabled starting with version 2.15.x, please check chart value fleet.bootstrap.localAgentDisabled
alert-rules/management-cluster/thanos.yaml
Alert Name For Severity Type Description
Thanos-Compact_Multiple_Running 5m warning monitoring More than one Thanos Compact instance is running. Current number of instances: {{ $value }}.
Thanos-Compact_Halted 5m critical monitoring Thanos Compact has failed to run and now is halted.
Thanos-Compact_Compaction_Failures_Rate_High 15m warning monitoring Thanos Compact is failing to execute {{ $value
Thanos-Compact_Bucket_Operation_Failures_Rate_High 15m warning monitoring Thanos Compact Bucket is failing to execute {{ $value
Thanos-Compact_Has_Not_Run 5m warning monitoring Thanos Compact has not uploaded anything for 24 hours.
Thanos-Query_Store_Endpoints_Missing 5m critical monitoring Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly.
Thanos-Query_HTTP_Request_Query_Error_Rate_High 5m critical monitoring Thanos Query is failing to handle {{ $value
Thanos-Query_HTTP_Request_QueryRange_Error_Rate_High 5m critical monitoring Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Server_Error_Rate_High 5m warning monitoring Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Client_Error_Rate_High 5m warning monitoring Thanos Query is failing to send {{ $value
Thanos-Query_Endpoint_DNS_Lookup_Failure_Rate_High 15m warning monitoring Thanos Query has {{ $value
Thanos-Query_Endpoint_Groups_DNS_Lookup_Failure_Rate_High 15m warning monitoring Thanos Query has {{ $value
Thanos-Query_Instant_Latency_High 10m critical monitoring Thanos Query has a 99th percentile latency of {{ $value }} seconds for instant queries.
Thanos-Query_Range_Latency_High 10m critical monitoring Thanos Query has a 99th percentile latency of {{ $value }} seconds for range queries.
Thanos-Query_Overload 15m warning monitoring Thanos Query has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support.
Thanos-Receive_HTTP_Request_Error_Rate_High 5m critical monitoring Thanos Receive is failing to handle {{ $value
Thanos-Receive_HTTP_Request_Latency_High 10m critical monitoring Thanos Receive has a 99th percentile latency of {{ $value }} seconds for requests.
Thanos-Receive_Replication_Failures_Rate_High 5m warning monitoring Thanos Receive is failing to replicate {{ $value
Thanos-Receive_Forward_Request_Failures_Rate_High 5m info monitoring Thanos Receive is failing to forward {{ $value
Thanos-Receive_Hashring_File_Refresh_Failures_Rate_High 15m warning monitoring Thanos Receive is failing to refresh hashring file, {{ $value
Thanos-Receive_Config_Reload_Failure 5m warning monitoring Thanos Receive has not been able to reload hashring configurations.
Thanos-Receive_No_Upload 3h critical monitoring Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage.
Thanos-Receive_Limits_Config_Reload_Failure 5m warning monitoring Thanos Receive has not been able to reload the limits configuration.
Thanos-Receive_Bucket_Operation_Failures_Rate_High 15m warning monitoring Thanos Receive Bucket is failing to execute {{ $value
Thanos-Store_GRPC_Error_Rate_High 5m warning monitoring Thanos Store is failing to handle {{ $value
Thanos-Store_Series_Gate_Latency_High 10m warning monitoring Thanos Store has a 99th percentile latency of {{ $value }} seconds for store series gate requests.
Thanos-Store_Bucket_Operation_Failures_Rate_High 15m warning monitoring Thanos Store Bucket is failing to execute {{ $value
Thanos-Store_Objstore_Operation_Latency_High 10m warning monitoring Thanos Store Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations.
Thanos-Store_Block_Drop_Rate_high 10m warning monitoring Thanos Store is evicting blocks from its in-memory cache at a high rate. This may increase query latency and indicate that the index cache size is insufficient for your workload.
Thanos-Ruler_Queue_is_Dropping_Alerts 5m critical monitoring Thanos Rule {{ $labels.pod }} is failing to queue alerts.
Thanos-Ruler_Sender_is_Failing_Alerts 5m critical monitoring Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager.
Thanos-Ruler_Rule_Evaluation_Failures_Rate_High 5m critical monitoring Thanos Rule {{ $labels.pod }} is failing to evaluate rules.
Thanos-Ruler_Rule_Evaluation_Warnings_Rate_High 15m info monitoring Thanos Rule {{ $labels.pod }} has high number of evaluation warnings.
Thanos-Ruler_Rule_Evaluation_Latency_High 5m warning monitoring Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}.
Thanos-Ruler_GRPC_Error_Rate_High 5m warning monitoring Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value
Thanos-Ruler_Config_Reload_Failure 5m info monitoring Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration.
Thanos-Ruler_Query_DNS_Lookup_Failure_Rate_High 15m warning monitoring Thanos Ruler {{ $labels.pod }} has {{ $value
Thanos-Ruler_Alertmanager_DNS_Failure_Lookup_Rate_High 15m warning monitoring Thanos Rule {{ $labels.pod }} has {{ $value
Thanos-Ruler_No_Evaluation_For_10_Intervals 5m info monitoring Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval.
Thanos-Ruler_No_Rule_Evaluations 5m critical monitoring Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes.
Thanos-Rule_Bucket_Operation_Failures_Rate_High 15m warning monitoring Thanos Rule Bucket is failing to execute {{ $value
Thanos-Component_Compact_is_Down 5m critical monitoring Thanos Compact has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Query_is_Down 5m critical monitoring Thanos Query has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_QueryFrontend_is_Down 5m critical monitoring Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Receive_is_Down 5m critical monitoring Thanos Receive has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Rule_is_Down 5m critical monitoring Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Store_is_Down 5m critical monitoring Thanos Store has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Block_Meta_Sync_Failures 10m critical monitoring Thanos "{{ $labels.container }}" has failed to fetch or parse some block metadata from object storage in the last 10 minutes. This may cause missing data in queries or failed compactions.
sylva-projects/sylva-elements/helm-charts/sylva-snmp-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-snmp-resources.git)

v0.5.0: sylva-snmp-resources: 0.5.0

Compare Source

Merge Requests integrated in this release

4 merge requests were integrated in this repo between 0.3.3 and 0.5.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Add CPU metrics to Cisco NCS module !117

CI

  • Update dependency to-be-continuous/gitleaks to v2.10.0 renovate !119 !120
  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.0 !121 renovate

Contributors

1 person contributed.

Alin H

sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules.git)

v0.6.0: sylva-thanos-rules: 0.6.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between 0.5.3 and 0.6.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Update the Kubernetes rules targeting only Sylva workloads !166

Contributors

1 person contributed.

Alin H

sylva-thanos-rules

Objective

Generate ConfigMap object for consumption by Thanos Ruler

Rules should be individually addressable so they can be modified, deleted and new rules be added.

Rule format

The values.yaml format allows for individual rule addressing (dictionary entries under groups.<group_name>.rules)

Rules are configured in values.yaml; the syntax is close to the Prometheus rule format

Each entry under rules: is a rule:

  • for recording rules (_type: record) the key represents the alertname+rule index separated by "|". The rule index is used for rule ordering so recording rules that use other recording rules' results are evaluated in the correct order. Recording rules are numerically sorted in ascending order.
  • for alerting rules there is no rule index. The key will be the final alert name. Alerting rules are alphabetically sorted. Alerting rules come after recording rules so they use the latest results.
fileName.yml: # file name that will be visible inside thanos ruler
  groups:
    groupName: # group name. groups are processed in parallel
      interval: 1m
      rules:
        ruleName|ruleIndex: # mandatory syntax for recording rules
          _type: record
          expr: vector(1)
          ...
        ruleName: # alerting rules only need a rule name
          _type: alert
          expr: vector(1)
          ...
    anothergroupName: # repeat for other group in the same file
    ...
anotherfileName.yml: # another file containing groups/rules
  ...

Rule configuration

Example section from values.yaml


# ...
node_exporter.yml: # file name
  groups:
    Node CPU rules: # group name
      interval: 1m # group interval
      rules:
        instance:node_cpus:count|1: # recording rule. name: instance:node_cpus:count, index 1 so it's the first rule
          _type: record
          expr: ...
        instance_mode:node_cpu_seconds:rate5m|2: # recording rule. name: instance_mode:node_cpu_seconds:rate5m, index 2 so it comes after instance:node_cpus:count
          _type: record
          expr: ...
        Node-Recently_Rebooted: # alerting rule
          _type: alert          # rule type
          # all the other fields follow the standard Prometheus rule format
          expr: (time() - node_boot_time_seconds) < 1800
          for: 0m
          labels:
            severity: 'warning'
            sylva_alert: 'true'
            type: 'system'
          annotations:
            description: 'Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" has been rebooted in the last 30 minutes. Value: {{ humanizeDuration $value }} uptime.'
            runbook_url: ''
            summary: 'Node recently rebooted'

# ...

Recording rules (_type: record) require one extra step - determining which index to encode in the key name to ensure that the rule will be placed in the right order for evaluation.

Rule list

Details about rules
_helper_kubernetes_metadata.yml
Alert Name For Severity Type Description
k8s-Metamonitoring_configuration_error_kube_namespace_labels 45m error deployment Metric "kube_namespace_labels" from cluster "{{ $labels.capi_cluster_name }}" is not exposed by "kube-state-metrics".
k8s-Metamonitoring_configuration_error_rancher_project_info 45m error deployment Metric "rancher_project_info" from the management cluster is not exposed by "kube-state-metrics".
clusters_state_rules.yml
Alert Name For Severity Type Description
Sylva_cluster_Prometheus_not_Sending_Data_management 45m critical deployment Prometheus server from the management cluster has not sent data in the last 45m.
Sylva_cluster_Prometheus_not_Sending_Data 5m critical deployment Prometheus server from cluster "{{ $labels.capi_cluster_name }}" in namespace "{{ $labels.capi_cluster_namespace }}" has not sent data in the last 45m.
Sylva_clusters_different_number 45m critical deployment Some cluster is not properly provisioned in Rancher, check all clusters to see if cattle-agent is properly deployed
Sylva_clusters_metric_absent 45m error deployment Metric "capi_cluster_info" from the management cluster is not exposed by "kube-state-metrics".
etcd.yml
Alert Name For Severity Type Description
etcd-Members_Down 5m warning etcd etcd in cluster "{{ $labels.cluster }}" members are down.
etcd-Members_Insufficient 5m warning etcd etcd in cluster "{{ $labels.cluster }}" has insufficient members. Value: {{ $value }}
etcd-Members_No_Leader 5m warning etcd etcd in cluster "{{ $labels.cluster }}" member {{ $labels.instance }} has no leader.
etcd-High_Number_of_Leader_Changes 5m warning etcd etcd in cluster "{{ $labels.cluster }}" {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.
etcd-gRPC_High_Number_of_Failed_Requests low-bar 10m warning etcd
etcd-gRPC_High_Number_of_Failed_Requests high-bar 5m critical etcd
etcd-Members_Communication_Slow 10m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" member communication is taking too long. Value: {{ $value }}s.
etcd-High_Number_of_Failed_Proposals 15m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has proposal failures within the last 30 minutes. Value: {{ $value }}
etcd-High_Fsync_Duration 10m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile fync duration. Value: {{ $value }}s
etcd-High_Commit_Duration 10m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile commit duration. Value: {{ $value }}s
etcd-High_Apply_Duration 10m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile apply duration. Value: {{ $value }}s
etcd-High_Range_Duration 10m warning etcd etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile range requests duration. Value: {{ $value }}s
etcd-HTTP_High_Error_Rate low-bar 10m warning etcd
etcd-HTTP_High_Error_Rate high-bar 10m warning etcd
kubernetes_capacity.yml
Alert Name For Severity Type Description
k8s-Cluster_CPU_Overcommitted 5m warning k8s Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable CPUs. Node failures may cause Pods to be unschedulable due to lack of resources
k8s-Cluster_Memory_Overcommitted 5m warning k8s Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable Memory. Node failures may cause Pods to be unschedulable due to lack of resources
k8s-Cluster_Too_Many_Pods 15m warning k8s Kubernetes cluster "{{ $labels.cluster }}" number of pods over 90% of Pod number limit. Value: {{ humanize $value }}%
k8s-Node_Too_Many_Pods 15m warning k8s Kubernetes cluster "{{ $labels.cluster }}" node {{ $labels.node }} number of pods over 90% of Pod number limit. Value: {{ humanize $value }}%
k8s-Kube_Quota_Almost_Full 15m warning k8s Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value
k8s-Kube_Quota_Exceeded 15m error k8s Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value
kubernetes_cluster_components.yml
Alert Name For Severity Type Description
k8s-Version_Mismatch 4h warning k8s Kubernetes cluster "{{ $labels.cluster }}" has different versions of Kubernetes components running. Value: {{ $value }}
k8s-Client_Errors 15m warning k8s Kubernetes cluster "{{ $labels.cluster }}" API server client "{{ $labels.instance }}" job "{{ $labels.job }}" is experiencing errors. Value: {{ printf "%0.0f" $value }}%
k8s-Client_Certificate_Expiration low-bar 5m warning k8s
k8s-Client_Certificate_Expiration high-bar 5m critical k8s
k8s-API_Global_Error_Rate_High 15m warning k8s Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for over 3% of requests. Value: {{ humanize $value }}%
k8s-API_Error_Rate_High 15m warning k8s Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for 10% of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}. Value: {{ humanize $value }}%
k8s-Aggregated_API_Errors 15m warning k8s Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has reported errors. It has appeared unavailable {{ $value
k8s-Aggregated_API_Down 15m warning k8s Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has been only {{ $value
k8s-API_Endpoint_Down 15m error k8s Kubernetes API endpoint {{ $labels.instance }} in cluster {{ $labels.cluster }} is unreachable.
k8s-API_Down 15m critical k8s Kubernetes API in cluster {{ $labels.cluster }} is unreachable.
k8s-API_Terminated_Requests 15m warning k8s Kubernetes API in cluster {{ $labels.cluster }} has terminated {{ $value
kubernetes_jobs.yml
Alert Name For Severity Type Description
k8s-CronJob_Status_Failed 5m warning k8s CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" failed. Last job has failed multiple times. Value: {{ $value }}
k8s-CronJob_Taking_Too_Long 0m warning k8s CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" is taking too long to completes - is over deadline. Value: {{ humanizeDuration $value }}
k8s-Job_not_Completed 15m warning k8s Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" is taking more than 12h to complete.
k8s-Job_Failed 15m warning k8s Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" failed to complete. Removing failed job after investigation should clear this alert.
kubernetes_nodes.yml
Alert Name For Severity Type Description
k8s-Node_Kubelet_Down 5m critical k8s Kubelet on {{ $labels.node }} in cluster "{{ $labels.cluster }}" is not reachable
k8s-Node_Status_OutOfDisk 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is almost out of disk space
k8s-Node_Status_MemoryPressure 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under memory pressure.
k8s-Node_Status_DiskPressure 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under disk pressure
k8s-Node_Status_PIDPressure 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under PID pressure
k8s-Node_Status_NotReady 5m error k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has been not been Ready for more than an hour
k8s-Node_Status_NetworkUnavailable 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has NetworkUnavailable condition.
k8s-Node_Status_Ready_flapping 5m warning k8s Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" readiness status changed {{ $value }} times in the last 15 minutes.
kubernetes_pods.yml
Alert Name For Severity Type Description
k8s-Pod_Status_not_Ready 15m warning k8s Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in a non-ready state for longer than 15 minutes.
k8s-Pod_Status_OOMKilled 0m error k8s Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" has been restarted due to "OOMKilled" reason in the last hour. Value: {{ humanize $value }}
k8s-Pod_Status_Crashlooping 15m warning k8s Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }}
k8s-Pod_Init_Container_Status_Crashlooping 15m warning k8s Init Container from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }}
k8s-Pod_Container_Status_Waiting 1h warning k8s Container "{{ $labels.container }}" from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in waiting state for longer than 1 hour.
k8s-Statefulset_Replicas_not_Ready 15m warning k8s Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }}
k8s-Statefulset_Generation_Mismatch 15m warning k8s Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" generation for does not match. This indicates that the StatefulSet has failed but has not been rolled back
k8s-Statefulset_Update_not_Rolled_Out 15m warning k8s Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" update has not been rolled out
k8s-Statefulset_Replicas_Mismatch 15m warning k8s Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_statefulset_replicas{statefulset="%s", cluster="%s"} $labels.statefulset $labels.cluster
k8s-Statefulset_Replicas_not_Updated 15m warning k8s Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment
k8s-ReplicaSet_Replicas_Mismatch 15m warning k8s ReplicaSet {{ $labels.namespace }}/{{ $labels.replicaset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas
k8s-Deployment_Replicas_not_Ready 15m warning k8s Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }}
k8s-Deployment_Replicas_Mismatch 15m warning k8s Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_deployment_spec_replicas{deployment="%s", cluster="%s"} $labels.deployment $labels.cluster
k8s-Deployment_Generation_Mismatch 15m warning k8s Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" generation does not match expected one
k8s-Deployment_Replicas_not_Updated 15m warning k8s Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment
k8s-Deployment_Rollout_Stuck 15m warning k8s Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" is not progressing for longer than 15 minutes.
k8s-Daemonset_Rollout_Stuck 15m warning k8s Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has less than 100% of desired pods scheduled and ready. Value: {{ humanize $value }}%
k8s-Daemonset_not_Scheduled 15m warning k8s Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has unscheduled pods. Value: {{ humanize $value }}
k8s-Daemonset_Misscheduled 15m warning k8s Daemonset pods {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" are running where they are not supposed to. Value: {{ humanize $value }}
k8s-Daemonset_Generation_Mismatch 15m warning k8s Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" generation does not match expected one
k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_High_CPU low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_High_CPU high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_High_CPU low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_High_CPU high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_High_Memory low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Requests_Too_High_Memory high-bar 15m warning k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_High_Memory low-bar 15m info k8s-info
k8s-Pod_Resource_Allocation_Limits_Too_High_Memory high-bar 15m warning k8s-info
kubernetes_storage.yml
Alert Name For Severity Type Description
k8s-Persistent_Volume_Disk_Space_Usage_High low-bar 5m info k8s
k8s-Persistent_Volume_Disk_Space_Usage_High high-bar 5m warning k8s
k8s-Persistent_Volume_Full_in_4_days 5m info k8s PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" will fill up in 4 days at the current rate of utilization. Value: {{ printf "%0.2f" $value }}% available
k8s-Persistent_Volume_Inodes_Usage_High low-bar 5m info k8s
k8s-Persistent_Volume_Inodes_Usage_High high-bar 5m warning k8s
k8s-Persistent_Volume_Errors 5m warning k8s PersistentVolume "{{ $labels.persistentvolume }}" in cluster "{{ $labels.cluster }}" has status "{{ $labels.phase }}"
k8s-Persistent_Volume_Claim_Orphan 3h info k8s PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" is not used by any pod
kubernetes_storage_ephemeral.yml
Alert Name For Severity Type Description
k8s-Ephemeral_Storage_Container_Usage_at_Limit 5m warning k8s Ephemeral storage usage of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is at {{ $value }}% of the limit.
k8s-Ephemeral_Storage_Container_Usage_Reaching_Limit 15m warning k8s Ephemeral storage limit of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is expected to be reached within 12 hours. Currently, {{ $value }}% is used.
k8s-Ephemeral_Storage_Volume_Filled_Up 5m warning k8s Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" high usage. Value: {{ $value }}%
k8s-Ephemeral_Storage_Volume_Filling_Up 5m warning k8s Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" is expected to be filled up within 12 hours. Currently, {{ $value }}% is used
k8s-Ephemeral_Storage_on_Node_Filling_Up low-bar 5m info k8s
k8s-Ephemeral_Storage_on_Node_Filling_Up high-bar 5m warning k8s
longhorn.yml
Alert Name For Severity Type Description
Longhorn-Volume_Status_Critical 5m error storage Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Faulted" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted
Longhorn-Volume_Status_Warning 5m warning storage Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Degraded" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted
Longhorn-Volume_Status_Unknown 5m warning storage Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Unknown" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted
Longhorn-Node_Storage_Warning 5m warning storage The used storage of node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value
Longhorn-Disk_Storage_Warning 5m warning storage The used storage of disk "{{ $labels.disk }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value
Longhorn-Node_Down 5m error storage There are "{{ $value
Longhorn-Instance_Manager_CPU_Usage_Warning 5m info storage Longhorn instance manager "{{ $labels.instance_manager }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU request at "{{ $value
Longhorn-Node_CPU_Usage_Warning 5m info storage Longhorn node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU capacity at "{{ $value
metallb.yml
Alert Name For Severity Type Description
MetalLB-BGP_Session_Down 5m error network MetalLB speaker {{ $labels.instance }} in cluster "{{ $labels.cluster}}" has BGP session {{ $labels.peer }} down for more than 5 minutes.
MetalLB-BGP_All_Sessions_Down 5m critical network MetalLB in "{{ $labels.cluster}}" all {{ $value }} BGP sessions are down for more than 5 minutes.
MetalLB-Address_Pool_High_Usage 5m info network MetalLB pool "{{ $labels.pool }}" in cluster "{{ $labels.cluster}}" has more than 75% of the total addresses used.
MetalLB-Config_Stale 5m warning network MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" has a stale config.
MetalLB-Config_not_Loaded 5m warning network MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" config not loaded.
minio.yml
Alert Name For Severity Type Description
MinIO_Cluster_Disk_Space_Will_Fill_Up_Soon 6h warning monitoring MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" at the current rate of utilization the available disk space will run out in the next 4 days.
monitoring_stack_components.yml
Alert Name For Severity Type Description
Monitoring-Prometheus_Bad_Config 10m critical monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration.
Monitoring-Prometheus_SD_Refresh_Failure 20m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to refresh SD with mechanism "{{ $labels.mechanism }}".
Monitoring-Prometheus_Kubernetes_List_Watch_Failures 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" service discovery is experiencing failures with LIST/WATCH requests to the Kubernetes API in the last 5 minutes. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Notification_Queue_Running_Full 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" alert notification queue predicted to run full in less than 30 minutes.
Monitoring-Prometheus_Error_Sending_Alerts_to_Some_Alertmanagers 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has {{ printf "%.1f" $value }}% of alerts sent to Alertmanager "{{ $labels.alertmanager }}" affected by errors.
Monitoring-Prometheus_not_Connected_to_Alertmanagers 10m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not connected to any Alertmanagers.
Monitoring-Prometheus_TSDB_Reloads_Failing 4h warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced TSDB reload failures in the last 3h. Value: {{ $value }}
Monitoring-Prometheus_TSDB_Compactions_Failing 4h warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced compaction failures in the last 3h. Value: {{ $value }}
Monitoring-Prometheus_not_Ingesting_Samples 10m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not ingesting samples.
Monitoring-Prometheus_Duplicate_Timestamps 10m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with different values but duplicated timestamps. Value: {{ printf "%.4g" $value }}
Monitoring-Prometheus_Out_of_Order_Timestamps 10m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with timestamps arriving out of order. Value: {{ printf "%.4g" $value }}
Monitoring-Prometheus_Remote_Storage_Failures 15m critical monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send a high number of samples to "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}%
Monitoring-Prometheus_Remote_Write_Behind 15m critical monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write is behind for "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}s
Monitoring-Prometheus_Remote_Write_Desired_Shards 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write desired shards calculation wants to run {{ $value }} shards for queue "{{ $labels.remote_name}}:{{ $labels.url }}", which is more than the max of "{{ printf prometheus_remote_storage_shards_max{instance="%s"} $labels.instance
Monitoring-Prometheus_Rule_Failures 15m critical monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to evaluate rules in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Missing_Rule_Evaluations 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" missed rule group evaluations in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Target_Limit_Hit 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because the number of targets exceeds the configured target_limit. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Label_Limit_Hit 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Scrape_Body_Size_Limit_Hit 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured body_size_limit. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Scrape_Sample_Limit_Hit 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured sample_limit. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_Target_Sync_Failure 5m critical monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" targets failed to sync because invalid configuration was supplied. Value: {{ printf "%.0f" $value }}
Monitoring-Prometheus_High_Query_Load 15m warning monitoring Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" query API has less than 20% available capacity in its query engine for the last 15 minutes.
Monitoring-PrometheusOperator_List_Errors 15m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "List" operations.
Monitoring-PrometheusOperator_Watch_Errors 15m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "Watch" operations.
Monitoring-PrometheusOperator_Sync_Failed 10m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed sync operations for {{ $value }} objects.
Monitoring-PrometheusOperator_Reconcile_Errors 10m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed reconciling operations. Value: {{ $value
Monitoring-PrometheusOperator_Status_Update_Errors 10m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed status update operations. Value: {{ $value
Monitoring-PrometheusOperator_Node_Lookup_Errors 10m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" errors while reconciling Prometheus.
Monitoring-PrometheusOperator_Not_Ready 5m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" is not ready to reconcile resources.
Monitoring-PrometheusOperator_Rejected_Resources 5m warning monitoring Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" rejected "{{ $labels.resource }}" resources. Value: {{ printf "%0.0f" $value }}
Monitoring-Alertmanager_Failed_Reload 10m critical monitoring Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration.
Monitoring-Alertmanager_Members_Inconsistent 15m critical monitoring Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has only found {{ $value }} members of the Alertmanager cluster.
Monitoring-Alertmanager_Failed_to_Send_Alerts 5m warning monitoring Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send {{ $value
Monitoring-Alertmanager_Cluster_Failed_to_Send_Alerts 5m critical monitoring Alertmanager in cluster "{{ $labels.cluster }}" has high notification failure rate to "{{ $labels.integration }}". Value: {{ $value
Monitoring-Alertmanager_Config_Inconsistent 20m critical monitoring Alertmanager instances in cluster "{{ $labels.cluster }}" have different configurations.
node_exporter.yml
Alert Name For Severity Type Description
Node-Recently_Rebooted 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" has been rebooted in the last 30 minutes. Value: {{ humanizeDuration $value }} uptime.
Node-CPU_High_Usage 30m info system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - CPU usage has exceeded the threshold of 90% for more than 30 minutes. Value: {{ humanize $value }}
Node-CPU_High_steal low-bar 15m info system
Node-CPU_High_steal high-bar 15m warning system
Node-CPU_High_iowait low-bar 15m info system
Node-CPU_High_iowait high-bar 15m error system
Node-Memory_Major_Pages_Faults 15m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - memory major page faults are occurring at very high rate. Value: {{ humanize $value }}
Node-Memory_High_Usage low-bar 10m info system
Node-Memory_High_Usage high-bar 10m warning system
Node-Disk_Space_High_Usage low-bar 15m warning system
Node-Disk_Space_High_Usage high-bar 15m error system
Node-Disk_Will_Fill_Up_In_4h 5m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" will fill up in 4 hours at the current rate of utilization. Value: {{ printf `node_filesystem_avail_bytes{fstype=~"ext.*
Node-High_Disk_Inodes_High_Usage low-bar 15m info system
Node-High_Disk_Inodes_High_Usage high-bar 15m warning system
Node-Load_High 15m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - system load per core is above 2 for the last 15 minutes. This might indicate this instance resources saturation and can cause it becoming unresponsive. Value: {{ humanize $value }}
Node-fds_Near_Limit_Process 5m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - "{{ $labels.job }}" has more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }}
Node-fds_Near_Limit 5m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }}
Node-Network_High_Receive_Drop 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has high drop in network reception. Value: {{ humanize $value }}
Node-Network_High_Receive_Errors 30m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered receive errors. Value: {{ humanize $value }}
Node-Network_High_Transmit_Errors 30m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered transmit errors. Value: {{ humanize $value }}
Node-Network_Interface_Flapping 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" changing its up status often. Value: {{ humanize $value }}
Node-Network_Bond_Interface_Misconfigured 5m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" is misconfigured. Check bonding slaves configuration.
Node-Network_Bond_Interface_Down 5m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" has interface(s) down. Value: {{ $value }}
Node-Too_Many_OOM_Kills 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - several OOM kills detected in the past 1h. Value: {{ humanize $value }}. Find out which process by running `dmesg
Node-Clock_Not_Synchronising 10m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - clock is not synchronising. Ensure NTP is configured on this host.
Node-Clock_Skew_Detected 10m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - clock is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host.
Node-Host_Conntrack_Limit 10m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - more than 75% of conntrack entries are used.
Node-EDAC_Correctable_Errors_Detected 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - EDAC Correctable Errors detected.
Node-EDAC_Uncorrectable_Errors_Detected 0m error system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - EDAC Uncorrectable Errors detected.
Node-Filesystem_Device_Error 0m warning system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" filesystem error.
Node-Disk_Queue_Length_High 10m info system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has disk queue length greater than 1 for more than 10 minutes. Value: {{ humanize $value }}
Node-Disk_IO_Time_Weighted_Seconds 10m info system Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has high disk io queue (aqu-sq) for more than 10 minutes. Value: {{ humanize $value }}
sylva-projects/sylva-elements/helm-charts/sync-openstack-images (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sync-openstack-images.git)

v0.10.0: sync-openstack-images: 0.10.0

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between 0.9.3 and 0.10.0. These notes don't account for the MRs merged in secondary repos.

Other dependency upgrades

  • Update Helm chart sylva-library to v0.10.1 !91

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.4 !93 renovate

Contributors

0 person contributed.

sylva-projects/sylva-elements/helm-charts/workload-team-defs (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/workload-team-defs.git)

v0.9.0: workload-team-defs: 0.9.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between 0.8.6 and 0.9.0. These notes don't account for the MRs merged in secondary repos.

Other

  • Condition JWTOIDCAuthEngineRole deployment on Vault being configured for OIDC !184

Contributors

1 person contributed.

Bogdan-Adrian Burciu

sylva-projects/sylva-elements/container-images/ci-image (registry.gitlab.com/sylva-projects/sylva-elements/container-images/ci-image)

v1.11.0: ci-image: v1.11.0

Compare Source

Merge Requests integrated in this release

1 merge request was integrated in this repo between v1.10.3 and v1.11.0. These notes don't account for the MRs merged in secondary repos.

CI
  • Update dependency sylva-projects/sylva-elements/container-images/sylva-toolbox to v1.10.1 !452 renovate
Contributors

0 person contributed.

sylva-projects/sylva-elements/container-images/kube-job (registry.gitlab.com/sylva-projects/sylva-elements/container-images/kube-job)

v1.8.0: kube-job: v1.8.0

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between v1.7.2 and v1.8.0. These notes don't account for the MRs merged in secondary repos.

Other
  • Revert clusterctl version to v1.12.x !274
CI
  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.3 !273 renovate
Contributors

1 person contributed.

Bogdan Antohe

sylva-projects/sylva-elements/helm-charts/sylva-library (sylva-library)

v0.11.0: sylva-library: 0.11.0

Compare Source

Merge Requests integrated in this release

3 merge requests were integrated in this repo between 0.10.1 and 0.11.0. These notes don't account for the MRs merged in secondary repos.

Networking

CI

  • Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.5 renovate !169 !170

Contributors

1 person contributed.

Cristian Manda


Configuration

📅 Schedule: (UTC)

  • Branch creation
    • At any time (no schedule defined)
  • Automerge
    • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

👻 Immortal: This MR will be recreated if closed unmerged. Get config help if that's undesired.


  • If you want to rebase/retry this MR, check this box

This MR has been generated by Renovate Bot Sylva instance.

CI configuration couldn't be handle by MR description. A dedicated comment has been posted to control it.

If no checkbox is checked, a default pipeline will be enabled (capm3, or capo if capo label is set)

Edited by Thomas Morin

Merge request reports

Loading
Loading