Update Sylva-elements (main) (minor)
tools/deps-project-tool.py has been used to generate release branches in sylva-elements repos, and fresh new x.(Y+1).0 tags have been created there on the same commit to which sylva-core was pointing via an x.y.Z tag.
Hence this MR introduces no code, config or any behavioral change.
More context: Creating a New Release Branch for Sylva core
This MR contains the following updates:
⚠️ WarningSome dependencies could not be looked up. Check the Dependency Dashboard for more information.
Release Notes
sylva-projects/sylva-elements/container-images/sylva-toolbox (container-images/sylva-toolbox)
v1.11.0: sylva-toolbox: v1.11.0
Merge Requests integrated in this release
4 merge requests were integrated in this repo between v1.10.9 and v1.11.0. These notes don't account for the MRs merged in secondary repos.
Sylva-units framework
- Update dependency sylva-projects/sylva-elements/sylvactl to v0.15.5 !703 renovate
- Update dependency sylva-projects/sylva-elements/sylva-units-operator to v1.4.5 !697 renovate
Logging
- Update dependency grafana/loki to v3.7.8 !634 area:observability renovate
Internal tooling
Contributors
1 person contributed.
sylva-projects/sylva-elements/heat-operator (gitlab.com/sylva-projects/sylva-elements/heat-operator)
v0.6.0: heat-operator: 0.6.0
Merge Requests integrated in this release
16 merge requests were integrated in this repo between 0.5.1 and 0.6.0. These notes don't account for the MRs merged in secondary repos.
Other dependency upgrades
- Update module k8s.io/apimachinery to v0.35.7 !367
- Update module github.com/go-logr/logr to v1.4.4 !369
- Update GO modules dependencies to v0.35.7 !371
- Update GO modules dependencies !357
- Update GO modules dependencies !374
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.3 renovate !363 !365
- Update sylva-elements/container-images/ci-image container to v1.10.4 renovate !351 !362
- Update docker.io/semgrep/semgrep container to v1.172.0 renovate !348 !366 !368 !370 !372 !373
Internal tooling
Contributors
1 person contributed.
sylva-projects/sylva-elements/misc-controllers-suite (gitlab.com/sylva-projects/sylva-elements/misc-controllers-suite)
v1.5.0: misc-controllers-suite: v1.5.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between v1.4.2 and v1.5.0. These notes don't account for the MRs merged in secondary repos.
Internal tooling
Contributors
1 person contributed.
sylva-projects/sylva-elements/sylva-units-operator (gitlab.com/sylva-projects/sylva-elements/sylva-units-operator)
v1.5.0: sylva-units-operator: v1.5.0
Merge Requests integrated in this release
6 merge requests were integrated in this repo between v1.4.4 and v1.5.0. These notes don't account for the MRs merged in secondary repos.
Other dependency upgrades
- Update module k8s.io/apimachinery to v0.35.9 !675
- Update pre-commit hook crate-ci/typos to v1.50.2 !677
CI
Internal tooling
Contributors
1 person contributed.
sylva-projects/sylva-elements/workload-cluster-operator (gitlab.com/sylva-projects/sylva-elements/workload-cluster-operator)
v0.15.0: workload-cluster-operator: v0.15.0
Merge Requests integrated in this release
11 merge requests were integrated in this repo between v0.14.5 and v0.15.0. These notes don't account for the MRs merged in secondary repos.
Sylva-units framework
- Update dependency gitlab.com/sylva-projects/sylva-elements/sylva-units-operator to v1.4.5 !635 renovate
- Update sylvactl to v0.15.5 !640 release-typepatch renovate
Workload cluster
Other dependency upgrades
- Update GO modules patch to v0.37.1 !653 release-typepatch
- Update GO minor - kubernetes !645 release-typepatch
- Update GO minor - test !646 release-typepatch
CI
- Update GO tools dependencies !648 release-typenone renovate
- add _typos.toml to exclude go.mod and go.sum from typos checks !651
- Update GO tools - codegen !647 release-typenone renovate
Internal tooling
- Regroup fluxcd updates in a single MR !641 area:CI
- Regenerate manifests on codegen updates !650 area:CI
Contributors
2 people contributed.
sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas.git)
v1.6.0: capo-contrail-bgpaas: 1.6.0
Merge Requests integrated in this release
2 merge requests were integrated in this repo between 1.1.3 and 1.6.0. These notes don't account for the MRs merged in secondary repos.
Networking
- fix missing heatstack fields !145
CI
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/keycloak-user-management (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/keycloak-user-management.git)
v0.5.0: keycloak-user-management: 0.5.0
Merge Requests integrated in this release
4 merge requests were integrated in this repo between 0.3.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.
Other
- Change policy for roleIdRef !17
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.2 renovate !14 !15
- Update dependency to-be-continuous/gitleaks to v2.11.0 !16 renovate
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/nmstate-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/nmstate-resources.git)
v1.2.0
sylva-projects/sylva-elements/helm-charts/os-image-server (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/os-image-server.git)
v2.9.0: os-image-server: 2.9.0
Merge Requests integrated in this release
9 merge requests were integrated in this repo between 2.7.2 and 2.9.0. These notes don't account for the MRs merged in secondary repos.
Other dependency upgrades
- Update Helm chart sylva-library to v0.10.1 !209
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.4 renovate !199 !206 !207 !210 !211
- Update dependency to-be-continuous/gitleaks to v2.11.0 renovate !200 !201 !208
Contributors
0 person contributed.
sylva-projects/sylva-elements/helm-charts/rancher-roles-management (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/rancher-roles-management.git)
v1.4.0: rancher-roles-management: 1.4.0
Merge Requests integrated in this release
7 merge requests were integrated in this repo between 1.1.4 and 1.4.0. These notes don't account for the MRs merged in secondary repos.
Cluster API
Other
- add singler-cluster mode support !31 review-focus
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.5 renovate !21 !24 !27 !29
- Update dependency to-be-continuous/gitleaks to v2.11.0 !22 renovate
Contributors
2 people contributed.
sylva-projects/sylva-elements/helm-charts/sylva-alertmanager-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-alertmanager-resources.git)
v0.5.0: sylva-alertmanager-resources: 0.5.0
Merge Requests integrated in this release
CI
-
Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.37
-
Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.34 !59 renovate
-
Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.35 !60 renovate
-
Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.36 !62 renovate
-
Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.0.37 !68 renovate
-
-
Update dependency renovate-bot/renovate-runner to v20
Other
Contributors
sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-capi-cluster.git)
v0.16.0: sylva-capi-cluster: 0.16.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between 0.15.21 and 0.16.0. These notes don't account for the MRs merged in secondary repos.
Cluster API
RKE2
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/sylva-dashboards (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-dashboards.git)
v1.1.0: sylva-dashboards: 1.1.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between 1.0.1 and 1.1.0. These notes don't account for the MRs merged in secondary repos.
Other
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/sylva-junos-config (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-junos-config.git)
v0.2.0
sylva-projects/sylva-elements/helm-charts/sylva-logging-flows (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-logging-flows.git)
v0.5.0: sylva-logging-flows: 0.5.0
Merge Requests integrated in this release
77 merge requests were integrated in this repo between 0.0.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.
Other
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.3 renovate !23 !25 !31 !32 !34 !37 !38 !40 !41 !46 !48 !51 !55 !58 !59 !61 !66 !68 !69 !74 !79 !82 !84 !87 !89 !90 !91 !92 !94 !95 !97 !100 !101 !104
- Update dependency renovate-bot/renovate-runner to v24 renovate !24 !26 !27 !29 !30 !33 !35 !36 !39 !42 !43 !44 !45 !47 !49 !50 !52 !53 !54 !56 !57 !60 !63 !64 !65 !67 !72 !73 !76 !78
- Update dependency to-be-continuous/gitleaks to v2.11.0 renovate !28 !62 !71 !75 !77 !80 !93 !96 !98 !99 !103
Contributors
2 people contributed.
sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git)
v0.5.0: sylva-prometheus-rules: 0.5.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between 0.4.1 and 0.5.0. These notes don't account for the MRs merged in secondary repos.
Other
- Rancher: Detect if fleet agent is still present !163
Contributors
1 person contributed.
sylva-prometheus-rules
Generate PrometheusRule objects for consumption by Prometheus
Overview
There are two mechanisms that control which rules are deployed
createRulesselects which directories are consideredoptional_rulesselects which files in those directories are added to the Configmap
Rules overrides
.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/
If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other
clusters are specified
This allows for rule overriding. Example:
createRules:
allclusters: true
management-cluster: truealert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml
alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml- First the
PrometheusRulewith the flux, minio and health-alerts name frommanagement-clusterare created. - Then health-alerts and dummy from
allclusterare parsed. Since health-alerts is already applied frommananagement-clusterit will not be applied again. dummy will be applied since it doesn't override anything
This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster
Rules activation
.Values.optional_rules controls which rules are enabled for optional components
Details about rules
alert-rules/allclusters/goldpinger.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Goldpinger_Node_Unhealthy | 5m | critical | network | Goldpinger reports unhealthy nodes: "{{ $labels.goldpinger_instance }}" |
| Goldpinger_Cluster_Health_Unhealthy | 5m | critical | network | Goldpinger reports overall cluster health failure on node "{{ $labels.goldpinger_instance }}". This indicates a pod-to-pod connectivity failure or failure to reach goldpinger service or cluster VIP on port 6443. |
alert-rules/allclusters/junos-exporter.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| JunOS_Down | 5m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped. |
| JunOS_Down_All_in_Group | 5m | critical | network | All targets in cluster "{{ $labels.cluster_name }}" in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped. |
| JunOS_Rebooted | 0m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" recently rebooted. |
| JunOS_Alarms_Major_Present | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Major alarms present. Value: {{ humanize $value }} alarms |
| JunOS_Alarms_Minor_Present | 5m | info | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Minor alarms present. Value: {{ humanize $value }} alarms |
| JunOS_BGP_Flapping | 5m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}"" has flapping BGP sessions. Value: {{ humanize $value }} |
| JunOS_BGP_not_Up | 5m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" BGP session with "{{ $labels.ip }}" is DOWN. |
| JunOS_Interface_Receive_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped incoming packets on {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 10m. Value: {{ humanize $value }} |
| JunOS_Interface_Receive_Errors | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for incoming packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}% |
| JunOS_Interface_Transmit_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in 10 minutes. Value: {{ humanize $value }} |
| JunOS_Interface_Transmit_Errors | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}% |
| JunOS_Interface_Physical_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has physical layer errors on "{{ $labels.name }}" ( {{ $labels.description }} ) interface. |
| JunOS_Interface_Oper_Status_Down | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Operational status on "{{ $labels.name }}" ( {{ $labels.description }} ) interface changed to Down. |
| JunOS_Power_Source_Down | 5m | info | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Power source "{{ $labels.item }}" status not OK. Value: {{ $labels.status }} status |
| JunOS_Routing_Engine_Status_NOK | 30m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" routing engine {{ $labels.slot }} status not OK for more than 30m. |
| JunOS_Routing_Engine_High_CPU_Utilization | 30m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" CPU usage for routing engine {{ $labels.slot }} is greater than 90% for more than 30m. Value: {{ humanize $value }}% |
| JunOS_Interface_Rate_Limit_Packets_Drop | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets due to rate limit on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }} |
| JunOS_Interface_Queues_Packets_Drop | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }} |
| JunOS_Interface_Flapping | 0m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" interface "{{ $labels.name }}" ( {{ $labels.description }} ) in the last hour. Value: {{ humanize $value }} resets |
| JunOS_IPSec_Tunnels_Down | 5m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has {{ $value }} IPSec tunnels DOWN out of {{ printf junos_ipsec_configured_tunnels{alias="%s"} $labels.alias |
| JunOS_High_Memory_Heap_Utilization | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" memory heap utilization has exceeded the threshold of 60% for more than 5 minutes. Value: {{ humanize $value }}% |
alert-rules/allclusters/snmp-cisco-ncs.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_Cisco_NCS_SNMP_get_failed | 30m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed. |
| SNMP_Cisco_NCS_High_CPU_Utilization | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU utilization is greater than 90% for more than 30m. Value: {{ $value }}% |
| SNMP_Cisco_NCS_Interface_Flapping | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_SNMP_get_returned_too_few_results | 1h | info | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_Interface_Receive_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_Interface_Receive_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_Interface_Transmit_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_Interface_Transmit_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_NCS_Interface_Oper_Status_Down | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down |
| SNMP_Cisco_NCS_FRU_Power_Oper_Status_not_OK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status |
| SNMP_Cisco_NCS_SensorStatus_Celsius_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_voltsDC_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_amperes_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_dBm_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_other_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_rpm_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_unknown_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_SensorStatus_dB_NOK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}". |
| SNMP_Cisco_NCS_mplsL3VpnVrf_Oper_And_Admin_Status_Mismatch | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational and Admin status mismatch on "{{ $labels.mplsL3VpnVrfName }}". |
alert-rules/allclusters/snmp-cisco-nexus.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_Cisco_Nexus_SNMP_get_failed | 30m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed. |
| SNMP_Cisco_Nexus_SNMP_get_returned_too_few_results | 1h | info | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Flapping | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Receive_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Receive_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Transmit_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Transmit_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Cisco_Nexus_Interface_Oper_Status_Down | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down |
| SNMP_Cisco_Nexus_FRU_Power_Oper_Status_not_OK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status |
| SNMP_Cisco_Nexus_High_CPU_Utilization | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU usage is greater than 90% for more than 30m. Value: {{ $value }} |
| SNMP_Cisco_Nexus_High_Memory_Utilization | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Memory usage is greater than 60% for more than 5m. Value: {{ $value }} |
| SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Rcvr_Status_not_OK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg receive status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgRcvrStatus }} status |
| SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Send_Status_not_OK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg send status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgSendStatus }} status |
| SNMP_Cisco_Nexus_Port_Storm_Control_Active | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Port Storm Control is active on interface "{{ $labels.ifDescr }}" in state "{{ $labels.cpscStatus }}" |
| SNMP_Cisco_Nexus_bgpPeerState_not_OK | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] bgpPeers connection to "{{ $labels.bgpPeerRemoteAddr }}" is in non established "{{ $labels.bgpPeerState }} state. |
alert-rules/allclusters/snmp-dell-idrac.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_DELL_iDRAC_globalSystemStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }} |
| SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC. |
| SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC. |
| SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC. |
| SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC. |
| SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC. |
| SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC. |
| SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC. |
| SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC. |
| SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC. |
| SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC. |
| SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC. |
| SNMP_DELL_iDRAC_controllerRollUpStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}. |
| SNMP_DELL_iDRAC_controllerComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}. |
| SNMP_DELL_iDRAC_physicalDiskState_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}. |
| SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}. |
| SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). |
| SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }} |
| SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }} |
| SNMP_DELL_iDRAC_virtualDiskState_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}. |
| SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}. |
| SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). |
alert-rules/allclusters/snmp-fortinet-fortigate.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_Fortinet_FortiGate_SNMP_get_failed | 30m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed. |
| SNMP_Fortinet_FortiGate_SNMP_get_returned_too_few_results | 1h | info | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Flapping | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Receive_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Receive_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Transmit_Drops | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Transmit_Errors | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_Interface_Oper_Status_Down | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down |
| SNMP_Fortinet_FortiGate_HA_State_Not_Synchronized | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has "{{ $labels.fgHaStatsSyncStatus }}" for HA Index "{{ $labels.fgHaStatsIndex }}". |
| SNMP_Fortinet_FortiGate_HA_State_Changed | 0m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA State has changed in the last 10 minutes. |
| SNMP_Fortinet_FortiGate_PowerSupply_Status_Alarm_Active | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power Supply "{{ $labels.fgHwSensorEntDescr }}" status alarm is active in the last 5 minutes. |
| SNMP_Fortinet_FortiGate_Hardware_Sensor_Alarm_Active | 5m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Hardware Sensor "{{ $labels.fgHwSensorEntDescr }} status alarm is active in the last 5 minutes. |
| SNMP_Fortinet_FortiGate_High_CPU_Usage | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High CPU Usage in the last 30 minutes. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_High_Memory_Usage | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Memory Usage in the last 30 minutes. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_High_Disk_Usage | 30m | warning | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Disk Usage in the last 30 minutes. Value: {{ humanize $value }} |
| SNMP_Fortinet_FortiGate_HaStatsSyncStatus_NOK | 5m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Sync Status on unit "{{ $labels.fgHaStatsHostname }}" for more then 5m. |
| SNMP_Fortinet_FortiGate_HaStatus_Changed | 0m | error | network | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Status changed to "{{ $labels.fgVdEntHaState }}" in the last 30m. |
alert-rules/allclusters/snmp-hp-cpq.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_HP_CPQ_Overall_Health_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}" |
| SNMP_HP_CPQ_Event_Log_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}" |
| SNMP_HP_CPQ_CPU_Health_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}" |
| SNMP_HP_CPQ_Thermal_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}" |
| SNMP_HP_CPQ_Power_Supply_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}" |
| SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}" |
| SNMP_HP_CPQ_Controller_Overall_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}" status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}". This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator. |
| SNMP_HP_CPQ_iLO_LicenseKey_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation. |
alert-rules/allclusters/snmp-lenovo-xcc.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_Lenovo_XCC_systemHealthStat_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }} |
| SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }} |
| SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }} |
| SNMP_Lenovo_XCC_memoryHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }} |
| SNMP_Lenovo_XCC_fanHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }} |
| SNMP_Lenovo_XCC_voltHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }} |
alert-rules/management-cluster/flux.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Flux_Kustomization_Failing | 15m | warning | deployment | Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile. |
| Flux_Kustomization_Failing_Cluster | 60m | warning | deployment | Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile. |
| Flux_HelmRelease_Failing | 15m | warning | deployment | Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile. |
| Flux_Source_Failing | 15m | warning | deployment | Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile. |
| Flux_Resource_Suspended | 2h | warning | deployment | Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended. |
alert-rules/management-cluster/harbor.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Harbor_Component_Status_NOK | 5m | warning | tools | Harbor component "{{ $labels.component }}" status is DOWN. |
alert-rules/management-cluster/keycloak.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Keycloak-CNPG_WAL_Disk_Usage_High | 5m | warning | tools | WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB |
| Keycloak-CNPG_Replica_Lagging_Behind_Primary | 5m | warning | tools | CNPG replica slot {{ $labels.slot_name }} on {{ $labels.pod }} is lagging {{ $value |
alert-rules/management-cluster/loki.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Loki-Write_PVC_Disk_Usage_Warning | 5m | warning | logging | Loki write PVC "{{ $labels.persistentvolumeclaim }}" in namespace "{{ $labels.namespace }}" disk usage is above 70% (current value: {{ printf "%.1f" $value }}%). |
| Loki-Write_PVC_Disk_Usage_Critical | 5m | critical | logging | Loki write PVC "{{ $labels.persistentvolumeclaim }}" in namespace "{{ $labels.namespace }}" disk usage is above 85% (current value: {{ printf "%.1f" $value }}%). |
alert-rules/management-cluster/minio.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| MinIO_Cluster_Health_Status_NOK | 5m | critical | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK. |
| MinIO_Cluster_Health_Status_Unknown | 5m | critical | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages. |
| MinIO_Cluster_Disk_Offline | 5m | critical | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline. |
| MinIO_Cluster_Disk_Space_Usage | 5m | warning | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 20%. |
| MinIO_Cluster_Disk_Space_Usage | 5m | critical | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%. |
| MinIO_Cluster_Tolerance | 5m | critical | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes. |
| MinIO_Nodes_Offline | 5m | warning | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes. |
alert-rules/management-cluster/rancher.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Rancher_Token_Expires_Soon | 1h | warning | rancher | Token "{{ $labels.token_internal_name }}" for user "{{ $labels.user_email }}" expires in less than 10 days. |
| Rancher_Fleet_Agent_Detected | 5m | critical | rancher | Fleet Agent pod should be disabled starting with version 2.15.x, please check chart value fleet.bootstrap.localAgentDisabled |
alert-rules/management-cluster/thanos.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Thanos-Compact_Multiple_Running | 5m | warning | monitoring | More than one Thanos Compact instance is running. Current number of instances: {{ $value }}. |
| Thanos-Compact_Halted | 5m | critical | monitoring | Thanos Compact has failed to run and now is halted. |
| Thanos-Compact_Compaction_Failures_Rate_High | 15m | warning | monitoring | Thanos Compact is failing to execute {{ $value |
| Thanos-Compact_Bucket_Operation_Failures_Rate_High | 15m | warning | monitoring | Thanos Compact Bucket is failing to execute {{ $value |
| Thanos-Compact_Has_Not_Run | 5m | warning | monitoring | Thanos Compact has not uploaded anything for 24 hours. |
| Thanos-Query_Store_Endpoints_Missing | 5m | critical | monitoring | Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly. |
| Thanos-Query_HTTP_Request_Query_Error_Rate_High | 5m | critical | monitoring | Thanos Query is failing to handle {{ $value |
| Thanos-Query_HTTP_Request_QueryRange_Error_Rate_High | 5m | critical | monitoring | Thanos Query is failing to handle {{ $value |
| Thanos-Query_GRPC_Server_Error_Rate_High | 5m | warning | monitoring | Thanos Query is failing to handle {{ $value |
| Thanos-Query_GRPC_Client_Error_Rate_High | 5m | warning | monitoring | Thanos Query is failing to send {{ $value |
| Thanos-Query_Endpoint_DNS_Lookup_Failure_Rate_High | 15m | warning | monitoring | Thanos Query has {{ $value |
| Thanos-Query_Endpoint_Groups_DNS_Lookup_Failure_Rate_High | 15m | warning | monitoring | Thanos Query has {{ $value |
| Thanos-Query_Instant_Latency_High | 10m | critical | monitoring | Thanos Query has a 99th percentile latency of {{ $value }} seconds for instant queries. |
| Thanos-Query_Range_Latency_High | 10m | critical | monitoring | Thanos Query has a 99th percentile latency of {{ $value }} seconds for range queries. |
| Thanos-Query_Overload | 15m | warning | monitoring | Thanos Query has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support. |
| Thanos-Receive_HTTP_Request_Error_Rate_High | 5m | critical | monitoring | Thanos Receive is failing to handle {{ $value |
| Thanos-Receive_HTTP_Request_Latency_High | 10m | critical | monitoring | Thanos Receive has a 99th percentile latency of {{ $value }} seconds for requests. |
| Thanos-Receive_Replication_Failures_Rate_High | 5m | warning | monitoring | Thanos Receive is failing to replicate {{ $value |
| Thanos-Receive_Forward_Request_Failures_Rate_High | 5m | info | monitoring | Thanos Receive is failing to forward {{ $value |
| Thanos-Receive_Hashring_File_Refresh_Failures_Rate_High | 15m | warning | monitoring | Thanos Receive is failing to refresh hashring file, {{ $value |
| Thanos-Receive_Config_Reload_Failure | 5m | warning | monitoring | Thanos Receive has not been able to reload hashring configurations. |
| Thanos-Receive_No_Upload | 3h | critical | monitoring | Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage. |
| Thanos-Receive_Limits_Config_Reload_Failure | 5m | warning | monitoring | Thanos Receive has not been able to reload the limits configuration. |
| Thanos-Receive_Bucket_Operation_Failures_Rate_High | 15m | warning | monitoring | Thanos Receive Bucket is failing to execute {{ $value |
| Thanos-Store_GRPC_Error_Rate_High | 5m | warning | monitoring | Thanos Store is failing to handle {{ $value |
| Thanos-Store_Series_Gate_Latency_High | 10m | warning | monitoring | Thanos Store has a 99th percentile latency of {{ $value }} seconds for store series gate requests. |
| Thanos-Store_Bucket_Operation_Failures_Rate_High | 15m | warning | monitoring | Thanos Store Bucket is failing to execute {{ $value |
| Thanos-Store_Objstore_Operation_Latency_High | 10m | warning | monitoring | Thanos Store Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations. |
| Thanos-Store_Block_Drop_Rate_high | 10m | warning | monitoring | Thanos Store is evicting blocks from its in-memory cache at a high rate. This may increase query latency and indicate that the index cache size is insufficient for your workload. |
| Thanos-Ruler_Queue_is_Dropping_Alerts | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to queue alerts. |
| Thanos-Ruler_Sender_is_Failing_Alerts | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager. |
| Thanos-Ruler_Rule_Evaluation_Failures_Rate_High | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to evaluate rules. |
| Thanos-Ruler_Rule_Evaluation_Warnings_Rate_High | 15m | info | monitoring | Thanos Rule {{ $labels.pod }} has high number of evaluation warnings. |
| Thanos-Ruler_Rule_Evaluation_Latency_High | 5m | warning | monitoring | Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}. |
| Thanos-Ruler_GRPC_Error_Rate_High | 5m | warning | monitoring | Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value |
| Thanos-Ruler_Config_Reload_Failure | 5m | info | monitoring | Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration. |
| Thanos-Ruler_Query_DNS_Lookup_Failure_Rate_High | 15m | warning | monitoring | Thanos Ruler {{ $labels.pod }} has {{ $value |
| Thanos-Ruler_Alertmanager_DNS_Failure_Lookup_Rate_High | 15m | warning | monitoring | Thanos Rule {{ $labels.pod }} has {{ $value |
| Thanos-Ruler_No_Evaluation_For_10_Intervals | 5m | info | monitoring | Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval. |
| Thanos-Ruler_No_Rule_Evaluations | 5m | critical | monitoring | Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes. |
| Thanos-Rule_Bucket_Operation_Failures_Rate_High | 15m | warning | monitoring | Thanos Rule Bucket is failing to execute {{ $value |
| Thanos-Component_Compact_is_Down | 5m | critical | monitoring | Thanos Compact has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_Query_is_Down | 5m | critical | monitoring | Thanos Query has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_QueryFrontend_is_Down | 5m | critical | monitoring | Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_Receive_is_Down | 5m | critical | monitoring | Thanos Receive has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_Rule_is_Down | 5m | critical | monitoring | Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_Store_is_Down | 5m | critical | monitoring | Thanos Store has disappeared. Prometheus target for the component cannot be discovered. |
| Thanos-Component_Block_Meta_Sync_Failures | 10m | critical | monitoring | Thanos "{{ $labels.container }}" has failed to fetch or parse some block metadata from object storage in the last 10 minutes. This may cause missing data in queries or failed compactions. |
sylva-projects/sylva-elements/helm-charts/sylva-snmp-resources (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-snmp-resources.git)
v0.5.0: sylva-snmp-resources: 0.5.0
Merge Requests integrated in this release
4 merge requests were integrated in this repo between 0.3.3 and 0.5.0. These notes don't account for the MRs merged in secondary repos.
Other
- Add CPU metrics to Cisco NCS module !117
CI
- Update dependency to-be-continuous/gitleaks to v2.10.0 renovate !119 !120
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.0 !121 renovate
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules.git)
v0.6.0: sylva-thanos-rules: 0.6.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between 0.5.3 and 0.6.0. These notes don't account for the MRs merged in secondary repos.
Other
- Update the Kubernetes rules targeting only Sylva workloads !166
Contributors
1 person contributed.
sylva-thanos-rules
Objective
Generate ConfigMap object for consumption by Thanos Ruler
Rules should be individually addressable so they can be modified, deleted and new rules be added.
Rule format
The values.yaml format allows for individual rule addressing (dictionary entries under groups.<group_name>.rules)
Rules are configured in values.yaml; the syntax is close to the Prometheus rule format
Each entry under rules: is a rule:
- for recording rules (_type: record) the key represents the alertname+rule index separated by "|". The rule index is used for rule ordering so recording rules that use other recording rules' results are evaluated in the correct order. Recording rules are numerically sorted in ascending order.
- for alerting rules there is no rule index. The key will be the final alert name. Alerting rules are alphabetically sorted. Alerting rules come after recording rules so they use the latest results.
fileName.yml: # file name that will be visible inside thanos ruler
groups:
groupName: # group name. groups are processed in parallel
interval: 1m
rules:
ruleName|ruleIndex: # mandatory syntax for recording rules
_type: record
expr: vector(1)
...
ruleName: # alerting rules only need a rule name
_type: alert
expr: vector(1)
...
anothergroupName: # repeat for other group in the same file
...
anotherfileName.yml: # another file containing groups/rules
...Rule configuration
Example section from values.yaml
# ...
node_exporter.yml: # file name
groups:
Node CPU rules: # group name
interval: 1m # group interval
rules:
instance:node_cpus:count|1: # recording rule. name: instance:node_cpus:count, index 1 so it's the first rule
_type: record
expr: ...
instance_mode:node_cpu_seconds:rate5m|2: # recording rule. name: instance_mode:node_cpu_seconds:rate5m, index 2 so it comes after instance:node_cpus:count
_type: record
expr: ...
Node-Recently_Rebooted: # alerting rule
_type: alert # rule type
# all the other fields follow the standard Prometheus rule format
expr: (time() - node_boot_time_seconds) < 1800
for: 0m
labels:
severity: 'warning'
sylva_alert: 'true'
type: 'system'
annotations:
description: 'Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" has been rebooted in the last 30 minutes. Value: {{ humanizeDuration $value }} uptime.'
runbook_url: ''
summary: 'Node recently rebooted'
# ...Recording rules (_type: record) require one extra step - determining which index to encode in the key name to ensure that the rule will be placed in the right order for evaluation.
Rule list
Details about rules
_helper_kubernetes_metadata.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Metamonitoring_configuration_error_kube_namespace_labels | 45m | error | deployment | Metric "kube_namespace_labels" from cluster "{{ $labels.capi_cluster_name }}" is not exposed by "kube-state-metrics". |
| k8s-Metamonitoring_configuration_error_rancher_project_info | 45m | error | deployment | Metric "rancher_project_info" from the management cluster is not exposed by "kube-state-metrics". |
clusters_state_rules.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Sylva_cluster_Prometheus_not_Sending_Data_management | 45m | critical | deployment | Prometheus server from the management cluster has not sent data in the last 45m. |
| Sylva_cluster_Prometheus_not_Sending_Data | 5m | critical | deployment | Prometheus server from cluster "{{ $labels.capi_cluster_name }}" in namespace "{{ $labels.capi_cluster_namespace }}" has not sent data in the last 45m. |
| Sylva_clusters_different_number | 45m | critical | deployment | Some cluster is not properly provisioned in Rancher, check all clusters to see if cattle-agent is properly deployed |
| Sylva_clusters_metric_absent | 45m | error | deployment | Metric "capi_cluster_info" from the management cluster is not exposed by "kube-state-metrics". |
etcd.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| etcd-Members_Down | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" members are down. |
| etcd-Members_Insufficient | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" has insufficient members. Value: {{ $value }} |
| etcd-Members_No_Leader | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" member {{ $labels.instance }} has no leader. |
| etcd-High_Number_of_Leader_Changes | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated. |
| etcd-gRPC_High_Number_of_Failed_Requests | low-bar | 10m | warning | etcd |
| etcd-gRPC_High_Number_of_Failed_Requests | high-bar | 5m | critical | etcd |
| etcd-Members_Communication_Slow | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" member communication is taking too long. Value: {{ $value }}s. |
| etcd-High_Number_of_Failed_Proposals | 15m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has proposal failures within the last 30 minutes. Value: {{ $value }} |
| etcd-High_Fsync_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile fync duration. Value: {{ $value }}s |
| etcd-High_Commit_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile commit duration. Value: {{ $value }}s |
| etcd-High_Apply_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile apply duration. Value: {{ $value }}s |
| etcd-High_Range_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile range requests duration. Value: {{ $value }}s |
| etcd-HTTP_High_Error_Rate | low-bar | 10m | warning | etcd |
| etcd-HTTP_High_Error_Rate | high-bar | 10m | warning | etcd |
kubernetes_capacity.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Cluster_CPU_Overcommitted | 5m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable CPUs. Node failures may cause Pods to be unschedulable due to lack of resources |
| k8s-Cluster_Memory_Overcommitted | 5m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable Memory. Node failures may cause Pods to be unschedulable due to lack of resources |
| k8s-Cluster_Too_Many_Pods | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" number of pods over 90% of Pod number limit. Value: {{ humanize $value }}% |
| k8s-Node_Too_Many_Pods | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" node {{ $labels.node }} number of pods over 90% of Pod number limit. Value: {{ humanize $value }}% |
| k8s-Kube_Quota_Almost_Full | 15m | warning | k8s | Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value |
| k8s-Kube_Quota_Exceeded | 15m | error | k8s | Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value |
kubernetes_cluster_components.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Version_Mismatch | 4h | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has different versions of Kubernetes components running. Value: {{ $value }} |
| k8s-Client_Errors | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server client "{{ $labels.instance }}" job "{{ $labels.job }}" is experiencing errors. Value: {{ printf "%0.0f" $value }}% |
| k8s-Client_Certificate_Expiration | low-bar | 5m | warning | k8s |
| k8s-Client_Certificate_Expiration | high-bar | 5m | critical | k8s |
| k8s-API_Global_Error_Rate_High | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for over 3% of requests. Value: {{ humanize $value }}% |
| k8s-API_Error_Rate_High | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for 10% of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}. Value: {{ humanize $value }}% |
| k8s-Aggregated_API_Errors | 15m | warning | k8s | Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has reported errors. It has appeared unavailable {{ $value |
| k8s-Aggregated_API_Down | 15m | warning | k8s | Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has been only {{ $value |
| k8s-API_Endpoint_Down | 15m | error | k8s | Kubernetes API endpoint {{ $labels.instance }} in cluster {{ $labels.cluster }} is unreachable. |
| k8s-API_Down | 15m | critical | k8s | Kubernetes API in cluster {{ $labels.cluster }} is unreachable. |
| k8s-API_Terminated_Requests | 15m | warning | k8s | Kubernetes API in cluster {{ $labels.cluster }} has terminated {{ $value |
kubernetes_jobs.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-CronJob_Status_Failed | 5m | warning | k8s | CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" failed. Last job has failed multiple times. Value: {{ $value }} |
| k8s-CronJob_Taking_Too_Long | 0m | warning | k8s | CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" is taking too long to completes - is over deadline. Value: {{ humanizeDuration $value }} |
| k8s-Job_not_Completed | 15m | warning | k8s | Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" is taking more than 12h to complete. |
| k8s-Job_Failed | 15m | warning | k8s | Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" failed to complete. Removing failed job after investigation should clear this alert. |
kubernetes_nodes.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Node_Kubelet_Down | 5m | critical | k8s | Kubelet on {{ $labels.node }} in cluster "{{ $labels.cluster }}" is not reachable |
| k8s-Node_Status_OutOfDisk | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is almost out of disk space |
| k8s-Node_Status_MemoryPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under memory pressure. |
| k8s-Node_Status_DiskPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under disk pressure |
| k8s-Node_Status_PIDPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under PID pressure |
| k8s-Node_Status_NotReady | 5m | error | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has been not been Ready for more than an hour |
| k8s-Node_Status_NetworkUnavailable | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has NetworkUnavailable condition. |
| k8s-Node_Status_Ready_flapping | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" readiness status changed {{ $value }} times in the last 15 minutes. |
kubernetes_pods.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Pod_Status_not_Ready | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in a non-ready state for longer than 15 minutes. |
| k8s-Pod_Status_OOMKilled | 0m | error | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" has been restarted due to "OOMKilled" reason in the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Status_Crashlooping | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Init_Container_Status_Crashlooping | 15m | warning | k8s | Init Container from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Container_Status_Waiting | 1h | warning | k8s | Container "{{ $labels.container }}" from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in waiting state for longer than 1 hour. |
| k8s-Statefulset_Replicas_not_Ready | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }} |
| k8s-Statefulset_Generation_Mismatch | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" generation for does not match. This indicates that the StatefulSet has failed but has not been rolled back |
| k8s-Statefulset_Update_not_Rolled_Out | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" update has not been rolled out |
| k8s-Statefulset_Replicas_Mismatch | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_statefulset_replicas{statefulset="%s", cluster="%s"} $labels.statefulset $labels.cluster |
| k8s-Statefulset_Replicas_not_Updated | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment |
| k8s-ReplicaSet_Replicas_Mismatch | 15m | warning | k8s | ReplicaSet {{ $labels.namespace }}/{{ $labels.replicaset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas |
| k8s-Deployment_Replicas_not_Ready | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }} |
| k8s-Deployment_Replicas_Mismatch | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_deployment_spec_replicas{deployment="%s", cluster="%s"} $labels.deployment $labels.cluster |
| k8s-Deployment_Generation_Mismatch | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" generation does not match expected one |
| k8s-Deployment_Replicas_not_Updated | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment |
| k8s-Deployment_Rollout_Stuck | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" is not progressing for longer than 15 minutes. |
| k8s-Daemonset_Rollout_Stuck | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has less than 100% of desired pods scheduled and ready. Value: {{ humanize $value }}% |
| k8s-Daemonset_not_Scheduled | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has unscheduled pods. Value: {{ humanize $value }} |
| k8s-Daemonset_Misscheduled | 15m | warning | k8s | Daemonset pods {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" are running where they are not supposed to. Value: {{ humanize $value }} |
| k8s-Daemonset_Generation_Mismatch | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" generation does not match expected one |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_High_CPU | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_High_CPU | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_High_CPU | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_High_CPU | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_High_Memory | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Requests_Too_High_Memory | high-bar | 15m | warning | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_High_Memory | low-bar | 15m | info | k8s-info |
| k8s-Pod_Resource_Allocation_Limits_Too_High_Memory | high-bar | 15m | warning | k8s-info |
kubernetes_storage.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Persistent_Volume_Disk_Space_Usage_High | low-bar | 5m | info | k8s |
| k8s-Persistent_Volume_Disk_Space_Usage_High | high-bar | 5m | warning | k8s |
| k8s-Persistent_Volume_Full_in_4_days | 5m | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" will fill up in 4 days at the current rate of utilization. Value: {{ printf "%0.2f" $value }}% available |
| k8s-Persistent_Volume_Inodes_Usage_High | low-bar | 5m | info | k8s |
| k8s-Persistent_Volume_Inodes_Usage_High | high-bar | 5m | warning | k8s |
| k8s-Persistent_Volume_Errors | 5m | warning | k8s | PersistentVolume "{{ $labels.persistentvolume }}" in cluster "{{ $labels.cluster }}" has status "{{ $labels.phase }}" |
| k8s-Persistent_Volume_Claim_Orphan | 3h | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" is not used by any pod |
kubernetes_storage_ephemeral.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Ephemeral_Storage_Container_Usage_at_Limit | 5m | warning | k8s | Ephemeral storage usage of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is at {{ $value }}% of the limit. |
| k8s-Ephemeral_Storage_Container_Usage_Reaching_Limit | 15m | warning | k8s | Ephemeral storage limit of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is expected to be reached within 12 hours. Currently, {{ $value }}% is used. |
| k8s-Ephemeral_Storage_Volume_Filled_Up | 5m | warning | k8s | Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" high usage. Value: {{ $value }}% |
| k8s-Ephemeral_Storage_Volume_Filling_Up | 5m | warning | k8s | Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" is expected to be filled up within 12 hours. Currently, {{ $value }}% is used |
| k8s-Ephemeral_Storage_on_Node_Filling_Up | low-bar | 5m | info | k8s |
| k8s-Ephemeral_Storage_on_Node_Filling_Up | high-bar | 5m | warning | k8s |
longhorn.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Longhorn-Volume_Status_Critical | 5m | error | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Faulted" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Volume_Status_Warning | 5m | warning | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Degraded" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Volume_Status_Unknown | 5m | warning | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Unknown" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Node_Storage_Warning | 5m | warning | storage | The used storage of node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value |
| Longhorn-Disk_Storage_Warning | 5m | warning | storage | The used storage of disk "{{ $labels.disk }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value |
| Longhorn-Node_Down | 5m | error | storage | There are "{{ $value |
| Longhorn-Instance_Manager_CPU_Usage_Warning | 5m | info | storage | Longhorn instance manager "{{ $labels.instance_manager }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU request at "{{ $value |
| Longhorn-Node_CPU_Usage_Warning | 5m | info | storage | Longhorn node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU capacity at "{{ $value |
metallb.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| MetalLB-BGP_Session_Down | 5m | error | network | MetalLB speaker {{ $labels.instance }} in cluster "{{ $labels.cluster}}" has BGP session {{ $labels.peer }} down for more than 5 minutes. |
| MetalLB-BGP_All_Sessions_Down | 5m | critical | network | MetalLB in "{{ $labels.cluster}}" all {{ $value }} BGP sessions are down for more than 5 minutes. |
| MetalLB-Address_Pool_High_Usage | 5m | info | network | MetalLB pool "{{ $labels.pool }}" in cluster "{{ $labels.cluster}}" has more than 75% of the total addresses used. |
| MetalLB-Config_Stale | 5m | warning | network | MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" has a stale config. |
| MetalLB-Config_not_Loaded | 5m | warning | network | MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" config not loaded. |
minio.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| MinIO_Cluster_Disk_Space_Will_Fill_Up_Soon | 6h | warning | monitoring | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" at the current rate of utilization the available disk space will run out in the next 4 days. |
monitoring_stack_components.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Monitoring-Prometheus_Bad_Config | 10m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration. |
| Monitoring-Prometheus_SD_Refresh_Failure | 20m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to refresh SD with mechanism "{{ $labels.mechanism }}". |
| Monitoring-Prometheus_Kubernetes_List_Watch_Failures | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" service discovery is experiencing failures with LIST/WATCH requests to the Kubernetes API in the last 5 minutes. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Notification_Queue_Running_Full | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" alert notification queue predicted to run full in less than 30 minutes. |
| Monitoring-Prometheus_Error_Sending_Alerts_to_Some_Alertmanagers | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has {{ printf "%.1f" $value }}% of alerts sent to Alertmanager "{{ $labels.alertmanager }}" affected by errors. |
| Monitoring-Prometheus_not_Connected_to_Alertmanagers | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not connected to any Alertmanagers. |
| Monitoring-Prometheus_TSDB_Reloads_Failing | 4h | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced TSDB reload failures in the last 3h. Value: {{ $value }} |
| Monitoring-Prometheus_TSDB_Compactions_Failing | 4h | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced compaction failures in the last 3h. Value: {{ $value }} |
| Monitoring-Prometheus_not_Ingesting_Samples | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not ingesting samples. |
| Monitoring-Prometheus_Duplicate_Timestamps | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with different values but duplicated timestamps. Value: {{ printf "%.4g" $value }} |
| Monitoring-Prometheus_Out_of_Order_Timestamps | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with timestamps arriving out of order. Value: {{ printf "%.4g" $value }} |
| Monitoring-Prometheus_Remote_Storage_Failures | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send a high number of samples to "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}% |
| Monitoring-Prometheus_Remote_Write_Behind | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write is behind for "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}s |
| Monitoring-Prometheus_Remote_Write_Desired_Shards | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write desired shards calculation wants to run {{ $value }} shards for queue "{{ $labels.remote_name}}:{{ $labels.url }}", which is more than the max of "{{ printf prometheus_remote_storage_shards_max{instance="%s"} $labels.instance |
| Monitoring-Prometheus_Rule_Failures | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to evaluate rules in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Missing_Rule_Evaluations | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" missed rule group evaluations in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Target_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because the number of targets exceeds the configured target_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Label_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Scrape_Body_Size_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured body_size_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Scrape_Sample_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured sample_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Target_Sync_Failure | 5m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" targets failed to sync because invalid configuration was supplied. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_High_Query_Load | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" query API has less than 20% available capacity in its query engine for the last 15 minutes. |
| Monitoring-PrometheusOperator_List_Errors | 15m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "List" operations. |
| Monitoring-PrometheusOperator_Watch_Errors | 15m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "Watch" operations. |
| Monitoring-PrometheusOperator_Sync_Failed | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed sync operations for {{ $value }} objects. |
| Monitoring-PrometheusOperator_Reconcile_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed reconciling operations. Value: {{ $value |
| Monitoring-PrometheusOperator_Status_Update_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed status update operations. Value: {{ $value |
| Monitoring-PrometheusOperator_Node_Lookup_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" errors while reconciling Prometheus. |
| Monitoring-PrometheusOperator_Not_Ready | 5m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" is not ready to reconcile resources. |
| Monitoring-PrometheusOperator_Rejected_Resources | 5m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" rejected "{{ $labels.resource }}" resources. Value: {{ printf "%0.0f" $value }} |
| Monitoring-Alertmanager_Failed_Reload | 10m | critical | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration. |
| Monitoring-Alertmanager_Members_Inconsistent | 15m | critical | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has only found {{ $value }} members of the Alertmanager cluster. |
| Monitoring-Alertmanager_Failed_to_Send_Alerts | 5m | warning | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send {{ $value |
| Monitoring-Alertmanager_Cluster_Failed_to_Send_Alerts | 5m | critical | monitoring | Alertmanager in cluster "{{ $labels.cluster }}" has high notification failure rate to "{{ $labels.integration }}". Value: {{ $value |
| Monitoring-Alertmanager_Config_Inconsistent | 20m | critical | monitoring | Alertmanager instances in cluster "{{ $labels.cluster }}" have different configurations. |
node_exporter.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Node-Recently_Rebooted | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" has been rebooted in the last 30 minutes. Value: {{ humanizeDuration $value }} uptime. |
| Node-CPU_High_Usage | 30m | info | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - CPU usage has exceeded the threshold of 90% for more than 30 minutes. Value: {{ humanize $value }} |
| Node-CPU_High_steal | low-bar | 15m | info | system |
| Node-CPU_High_steal | high-bar | 15m | warning | system |
| Node-CPU_High_iowait | low-bar | 15m | info | system |
| Node-CPU_High_iowait | high-bar | 15m | error | system |
| Node-Memory_Major_Pages_Faults | 15m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - memory major page faults are occurring at very high rate. Value: {{ humanize $value }} |
| Node-Memory_High_Usage | low-bar | 10m | info | system |
| Node-Memory_High_Usage | high-bar | 10m | warning | system |
| Node-Disk_Space_High_Usage | low-bar | 15m | warning | system |
| Node-Disk_Space_High_Usage | high-bar | 15m | error | system |
| Node-Disk_Will_Fill_Up_In_4h | 5m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" will fill up in 4 hours at the current rate of utilization. Value: {{ printf `node_filesystem_avail_bytes{fstype=~"ext.* |
| Node-High_Disk_Inodes_High_Usage | low-bar | 15m | info | system |
| Node-High_Disk_Inodes_High_Usage | high-bar | 15m | warning | system |
| Node-Load_High | 15m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - system load per core is above 2 for the last 15 minutes. This might indicate this instance resources saturation and can cause it becoming unresponsive. Value: {{ humanize $value }} |
| Node-fds_Near_Limit_Process | 5m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - "{{ $labels.job }}" has more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }} |
| Node-fds_Near_Limit | 5m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }} |
| Node-Network_High_Receive_Drop | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has high drop in network reception. Value: {{ humanize $value }} |
| Node-Network_High_Receive_Errors | 30m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered receive errors. Value: {{ humanize $value }} |
| Node-Network_High_Transmit_Errors | 30m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered transmit errors. Value: {{ humanize $value }} |
| Node-Network_Interface_Flapping | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" changing its up status often. Value: {{ humanize $value }} |
| Node-Network_Bond_Interface_Misconfigured | 5m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" is misconfigured. Check bonding slaves configuration. |
| Node-Network_Bond_Interface_Down | 5m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" has interface(s) down. Value: {{ $value }} |
| Node-Too_Many_OOM_Kills | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - several OOM kills detected in the past 1h. Value: {{ humanize $value }}. Find out which process by running `dmesg |
| Node-Clock_Not_Synchronising | 10m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - clock is not synchronising. Ensure NTP is configured on this host. |
| Node-Clock_Skew_Detected | 10m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - clock is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host. |
| Node-Host_Conntrack_Limit | 10m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - more than 75% of conntrack entries are used. |
| Node-EDAC_Correctable_Errors_Detected | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - EDAC Correctable Errors detected. |
| Node-EDAC_Uncorrectable_Errors_Detected | 0m | error | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - EDAC Uncorrectable Errors detected. |
| Node-Filesystem_Device_Error | 0m | warning | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" filesystem error. |
| Node-Disk_Queue_Length_High | 10m | info | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has disk queue length greater than 1 for more than 10 minutes. Value: {{ humanize $value }} |
| Node-Disk_IO_Time_Weighted_Seconds | 10m | info | system | Node "{{ $labels.alias }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has high disk io queue (aqu-sq) for more than 10 minutes. Value: {{ humanize $value }} |
sylva-projects/sylva-elements/helm-charts/sync-openstack-images (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sync-openstack-images.git)
v0.10.0: sync-openstack-images: 0.10.0
Merge Requests integrated in this release
2 merge requests were integrated in this repo between 0.9.3 and 0.10.0. These notes don't account for the MRs merged in secondary repos.
Other dependency upgrades
- Update Helm chart sylva-library to v0.10.1 !91
CI
Contributors
0 person contributed.
sylva-projects/sylva-elements/helm-charts/workload-team-defs (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/workload-team-defs.git)
v0.9.0: workload-team-defs: 0.9.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between 0.8.6 and 0.9.0. These notes don't account for the MRs merged in secondary repos.
Other
- Condition JWTOIDCAuthEngineRole deployment on Vault being configured for OIDC !184
Contributors
1 person contributed.
sylva-projects/sylva-elements/container-images/ci-image (registry.gitlab.com/sylva-projects/sylva-elements/container-images/ci-image)
v1.11.0: ci-image: v1.11.0
Merge Requests integrated in this release
1 merge request was integrated in this repo between v1.10.3 and v1.11.0. These notes don't account for the MRs merged in secondary repos.
CI
- Update dependency sylva-projects/sylva-elements/container-images/sylva-toolbox to v1.10.1 !452 renovate
Contributors
0 person contributed.
sylva-projects/sylva-elements/container-images/kube-job (registry.gitlab.com/sylva-projects/sylva-elements/container-images/kube-job)
v1.8.0: kube-job: v1.8.0
Merge Requests integrated in this release
2 merge requests were integrated in this repo between v1.7.2 and v1.8.0. These notes don't account for the MRs merged in secondary repos.
Other
- Revert clusterctl version to v1.12.x !274
CI
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/sylva-library (sylva-library)
v0.11.0: sylva-library: 0.11.0
Merge Requests integrated in this release
3 merge requests were integrated in this repo between 0.10.1 and 0.11.0. These notes don't account for the MRs merged in secondary repos.
Networking
- Add ipv6 vip validation !168 typefeature (issues: #10)
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.5 renovate !169 !170
Contributors
1 person contributed.
Configuration
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
- If you want to rebase/retry this MR, check this box
This MR has been generated by Renovate Bot Sylva instance.
CI configuration couldn't be handle by MR description. A dedicated comment has been posted to control it.
If no checkbox is checked, a default pipeline will be enabled (capm3, or capo if capo label is set)