Update dependency https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git to v0.3.10 (main) (!7825) · Merge requests · Sylva-projects / sylva-core

This MR contains the following updates:

Package	Update	Change
https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git	patch	`0.3.7` -> `0.3.10`

⚠️ Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git)

`v0.3.10`: sylva-prometheus-rules: 0.3.10

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between 0.3.9 and 0.3.10. These notes don't account for the MRs merged in secondary repos.

Observability

Enable Goldpinger_Node_Unhealthy alert on both (mgmt & wkld) clusters !146 typebug (issues: #15)

Other

Add Goldpinger cluster_health_total alert rule !152 (issues: #17)

Contributors

1 person contributed.

Akshay Yadav

sylva-prometheus-rules

Generate PrometheusRule objects for consumption by Prometheus

Overview

There are two mechanisms that control which rules are deployed

createRules selects which directories are considered
optional_rules selects which files in those directories are added to the Configmap

Rules overrides

.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/

If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other clusters are specified

This allows for rule overriding. Example:

createRules:
  allclusters: true
  management-cluster: true

alert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml

alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml

First the PrometheusRule with the flux, minio and health-alerts name from management-cluster are created.
Then health-alerts and dummy from allcluster are parsed. Since health-alerts is already applied from mananagement-cluster it will not be applied again. dummy will be applied since it doesn't override anything

This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster

Rules activation

.Values.optional_rules controls which rules are enabled for optional components

Details about rules

alert-rules/allclusters/goldpinger.yaml

Alert Name	For	Severity	Type	Description
Goldpinger_Node_Unhealthy	5m	critical	network	Goldpinger reports unhealthy nodes: "{{ $labels.goldpinger_instance }}"
Goldpinger_Cluster_Health_Unhealthy	5m	critical	network	Goldpinger reports overall cluster health failure on node "{{ $labels.goldpinger_instance }}". This indicates a pod-to-pod connectivity failure or failure to reach goldpinger service or cluster VIP on port 6443.

alert-rules/allclusters/junos-exporter.yaml

Alert Name	For	Severity	Type	Description
JunOS_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Down_All_in_Group	5m	critical	network	All targets in cluster "{{ $labels.cluster_name }}" in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Rebooted	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" recently rebooted.
JunOS_Alarms_Major_Present	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Major alarms present. Value: {{ humanize $value }} alarms
JunOS_Alarms_Minor_Present	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Minor alarms present. Value: {{ humanize $value }} alarms
JunOS_BGP_Flapping	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}"" has flapping BGP sessions. Value: {{ humanize $value }}
JunOS_BGP_not_Up	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" BGP session with "{{ $labels.ip }}" is DOWN.
JunOS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped incoming packets on {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 10m. Value: {{ humanize $value }}
JunOS_Interface_Receive_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for incoming packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in 10 minutes. Value: {{ humanize $value }}
JunOS_Interface_Transmit_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Physical_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has physical layer errors on "{{ $labels.name }}" ( {{ $labels.description }} ) interface.
JunOS_Interface_Oper_Status_Down	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Operational status on "{{ $labels.name }}" ( {{ $labels.description }} ) interface changed to Down.
JunOS_Power_Source_Down	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Power source "{{ $labels.item }}" status not OK. Value: {{ $labels.status }} status
JunOS_Routing_Engine_Status_NOK	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" routing engine {{ $labels.slot }} status not OK for more than 30m.
JunOS_Routing_Engine_High_CPU_Utilization	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" CPU usage for routing engine {{ $labels.slot }} is greater than 90% for more than 30m. Value: {{ humanize $value }}%
JunOS_Interface_Rate_Limit_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets due to rate limit on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Queues_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Flapping	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" interface "{{ $labels.name }}" ( {{ $labels.description }} ) in the last hour. Value: {{ humanize $value }} resets
JunOS_IPSec_Tunnels_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has {{ $value }} IPSec tunnels DOWN out of {{ printf `junos_ipsec_configured_tunnels{alias="%s"}` $labels.alias
JunOS_High_Memory_Heap_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" memory heap utilization has exceeded the threshold of 60% for more than 5 minutes. Value: {{ humanize $value }}%

alert-rules/allclusters/snmp-cisco-ncs.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_NCS_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_NCS_High_CPU_Utilization	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU utilization is greater than 90% for more than 30m. Value: {{ $value }}%
SNMP_Cisco_NCS_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_NCS_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_NCS_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_NCS_SensorStatus_Celsius_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_voltsDC_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_amperes_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dBm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_other_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_rpm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_unknown_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dB_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_mplsL3VpnVrf_Oper_And_Admin_Status_Mismatch	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational and Admin status mismatch on "{{ $labels.mplsL3VpnVrfName }}".

alert-rules/allclusters/snmp-cisco-nexus.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_Nexus_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_Nexus_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_Nexus_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_Nexus_High_CPU_Utilization	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU usage is greater than 90% for more than 30m. Value: {{ $value }}
SNMP_Cisco_Nexus_High_Memory_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Memory usage is greater than 60% for more than 5m. Value: {{ $value }}
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Rcvr_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg receive status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgRcvrStatus }} status
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Send_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg send status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgSendStatus }} status
SNMP_Cisco_Nexus_Port_Storm_Control_Active	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Port Storm Control is active on interface "{{ $labels.ifDescr }}" in state "{{ $labels.cpscStatus }}"
SNMP_Cisco_Nexus_bgpPeerState_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] bgpPeers connection to "{{ $labels.bgpPeerRemoteAddr }}" is in non established "{{ $labels.bgpPeerState }} state.

alert-rules/allclusters/snmp-dell-idrac.yaml

Alert Name	For	Severity	Type	Description
SNMP_DELL_iDRAC_globalSystemStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }}
SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC.
SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_controllerRollUpStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}.
SNMP_DELL_iDRAC_controllerComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}.
SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}).
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_virtualDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}.
SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}.
SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}).

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

Alert Name	For	Severity	Type	Description
SNMP_Fortinet_FortiGate_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Fortinet_FortiGate_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Fortinet_FortiGate_HA_State_Not_Synchronized	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has "{{ $labels.fgHaStatsSyncStatus }}" for HA Index "{{ $labels.fgHaStatsIndex }}".
SNMP_Fortinet_FortiGate_HA_State_Changed	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA State has changed in the last 10 minutes.
SNMP_Fortinet_FortiGate_PowerSupply_Status_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power Supply "{{ $labels.fgHwSensorEntDescr }}" status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_Hardware_Sensor_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Hardware Sensor "{{ $labels.fgHwSensorEntDescr }} status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_High_CPU_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High CPU Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Memory_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Memory Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Disk_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Disk Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_HaStatsSyncStatus_NOK	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Sync Status on unit "{{ $labels.fgHaStatsHostname }}" for more then 5m.
SNMP_Fortinet_FortiGate_HaStatus_Changed	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Status changed to "{{ $labels.fgVdEntHaState }}" in the last 30m.

alert-rules/allclusters/snmp-hp-cpq.yaml

Alert Name	For	Severity	Type	Description
SNMP_HP_CPQ_Overall_Health_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}"
SNMP_HP_CPQ_Event_Log_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}"
SNMP_HP_CPQ_CPU_Health_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}"
SNMP_HP_CPQ_Thermal_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}"
SNMP_HP_CPQ_Power_Supply_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}"
SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}"
SNMP_HP_CPQ_Controller_Overall_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}" status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}". This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator.
SNMP_HP_CPQ_iLO_LicenseKey_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation.

alert-rules/allclusters/snmp-lenovo-xcc.yaml

Alert Name	For	Severity	Type	Description
SNMP_Lenovo_XCC_systemHealthStat_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }}
SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }}
SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }}
SNMP_Lenovo_XCC_memoryHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }}
SNMP_Lenovo_XCC_fanHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }}
SNMP_Lenovo_XCC_voltHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }}

alert-rules/management-cluster/flux.yaml

Alert Name	For	Severity	Type	Description
Flux_Kustomization_Failing	15m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_Kustomization_Failing_Cluster	60m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_HelmRelease_Failing	15m	warning	deployment	Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Source_Failing	15m	warning	deployment	Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Resource_Suspended	2h	warning	deployment	Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended.

alert-rules/management-cluster/harbor.yaml

Alert Name	For	Severity	Type	Description
Harbor_Component_Status_NOK	5m	warning	tools	Harbor component "{{ $labels.component }}" status is DOWN.

alert-rules/management-cluster/keycloak.yaml

Alert Name	For	Severity	Type	Description
Keycloak-CNPG_WAL_Disk_Usage_High	5m	warning	tools	WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB
Keycloak-CNPG_Replica_Lagging_Behind_Primary	5m	warning	tools	CNPG replica slot {{ $labels.slot_name }} on {{ $labels.pod }} is lagging {{ $value

alert-rules/management-cluster/minio.yaml

Alert Name	For	Severity	Type	Description
MinIO_Cluster_Health_Status_NOK	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK.
MinIO_Cluster_Health_Status_Unknown	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages.
MinIO_Cluster_Disk_Offline	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline.
MinIO_Cluster_Disk_Space_Usage	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 20%.
MinIO_Cluster_Disk_Space_Usage	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%.
MinIO_Cluster_Tolerance	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes.
MinIO_Nodes_Offline	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes.

alert-rules/management-cluster/rancher.yaml

Alert Name	For	Severity	Type	Description
Rancher_Token_Expires_Soon	1h	warning	rancher	Token "{{ $labels.token_internal_name }}" for user "{{ $labels.user_email }}" expires in less than 10 days.

alert-rules/management-cluster/thanos.yaml

Alert Name	For	Severity	Type	Description
Thanos-Compact_Multiple_Running	5m	warning	monitoring	More than one Thanos Compact instance is running. Current number of instances: {{ $value }}.
Thanos-Compact_Halted	5m	critical	monitoring	Thanos Compact has failed to run and now is halted.
Thanos-Compact_Compaction_Failures_Rate_High	15m	warning	monitoring	Thanos Compact is failing to execute {{ $value
Thanos-Compact_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Compact Bucket is failing to execute {{ $value
Thanos-Compact_Has_Not_Run	5m	warning	monitoring	Thanos Compact has not uploaded anything for 24 hours.
Thanos-Query_Store_Endpoints_Missing	5m	critical	monitoring	Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly.
Thanos-Query_HTTP_Request_Query_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_HTTP_Request_QueryRange_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Server_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Client_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to send {{ $value
Thanos-Query_Endpoint_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Endpoint_Groups_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Instant_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for instant queries.
Thanos-Query_Range_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for range queries.
Thanos-Query_Overload	15m	warning	monitoring	Thanos Query has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support.
Thanos-Receive_HTTP_Request_Error_Rate_High	5m	critical	monitoring	Thanos Receive is failing to handle {{ $value
Thanos-Receive_HTTP_Request_Latency_High	10m	critical	monitoring	Thanos Receive has a 99th percentile latency of {{ $value }} seconds for requests.
Thanos-Receive_Replication_Failures_Rate_High	5m	warning	monitoring	Thanos Receive is failing to replicate {{ $value
Thanos-Receive_Forward_Request_Failures_Rate_High	5m	info	monitoring	Thanos Receive is failing to forward {{ $value
Thanos-Receive_Hashring_File_Refresh_Failures_Rate_High	15m	warning	monitoring	Thanos Receive is failing to refresh hashring file, {{ $value
Thanos-Receive_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload hashring configurations.
Thanos-Receive_No_Upload	3h	critical	monitoring	Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage.
Thanos-Receive_Limits_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload the limits configuration.
Thanos-Receive_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Receive Bucket is failing to execute {{ $value
Thanos-Store_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Store is failing to handle {{ $value
Thanos-Store_Series_Gate_Latency_High	10m	warning	monitoring	Thanos Store has a 99th percentile latency of {{ $value }} seconds for store series gate requests.
Thanos-Store_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Store Bucket is failing to execute {{ $value
Thanos-Store_Objstore_Operation_Latency_High	10m	warning	monitoring	Thanos Store Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations.
Thanos-Store_Block_Drop_Rate_high	10m	warning	monitoring	Thanos Store is evicting blocks from its in-memory cache at a high rate. This may increase query latency and indicate that the index cache size is insufficient for your workload.
Thanos-Ruler_Queue_is_Dropping_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to queue alerts.
Thanos-Ruler_Sender_is_Failing_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager.
Thanos-Ruler_Rule_Evaluation_Failures_Rate_High	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to evaluate rules.
Thanos-Ruler_Rule_Evaluation_Warnings_Rate_High	15m	info	monitoring	Thanos Rule {{ $labels.pod }} has high number of evaluation warnings.
Thanos-Ruler_Rule_Evaluation_Latency_High	5m	warning	monitoring	Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}.
Thanos-Ruler_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value
Thanos-Ruler_Config_Reload_Failure	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration.
Thanos-Ruler_Query_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Ruler {{ $labels.pod }} has {{ $value
Thanos-Ruler_Alertmanager_DNS_Failure_Lookup_Rate_High	15m	warning	monitoring	Thanos Rule {{ $labels.pod }} has {{ $value
Thanos-Ruler_No_Evaluation_For_10_Intervals	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval.
Thanos-Ruler_No_Rule_Evaluations	5m	critical	monitoring	Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes.
Thanos-Rule_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Rule Bucket is failing to execute {{ $value
Thanos-Component_Compact_is_Down	5m	critical	monitoring	Thanos Compact has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Query_is_Down	5m	critical	monitoring	Thanos Query has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_QueryFrontend_is_Down	5m	critical	monitoring	Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Receive_is_Down	5m	critical	monitoring	Thanos Receive has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Rule_is_Down	5m	critical	monitoring	Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Store_is_Down	5m	critical	monitoring	Thanos Store has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Block_Meta_Sync_Failures	10m	critical	monitoring	Thanos "{{ $labels.container }}" has failed to fetch or parse some block metadata from object storage in the last 10 minutes. This may cause missing data in queries or failed compactions.

`v0.3.9`: sylva-prometheus-rules: 0.3.9

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between 0.3.8 and 0.3.9. These notes don't account for the MRs merged in secondary repos.

Observability

Create alert to detect cnpg lagging replicas. !133 (issues: #10)

Other

Add CPU and memory rules for Cisco NCS !136

Contributors

2 people contributed.

Akshay Yadav, Alin H

sylva-prometheus-rules

Generate PrometheusRule objects for consumption by Prometheus

Overview

There are two mechanisms that control which rules are deployed

createRules selects which directories are considered
optional_rules selects which files in those directories are added to the Configmap

Rules overrides

.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/

If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other clusters are specified

This allows for rule overriding. Example:

createRules:
  allclusters: true
  management-cluster: true

alert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml

alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml

First the PrometheusRule with the flux, minio and health-alerts name from management-cluster are created.
Then health-alerts and dummy from allcluster are parsed. Since health-alerts is already applied from mananagement-cluster it will not be applied again. dummy will be applied since it doesn't override anything

This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster

Rules activation

.Values.optional_rules controls which rules are enabled for optional components

Details about rules

alert-rules/allclusters/junos-exporter.yaml

Alert Name	For	Severity	Type	Description
JunOS_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Down_All_in_Group	5m	critical	network	All targets in cluster "{{ $labels.cluster_name }}" in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Rebooted	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" recently rebooted.
JunOS_Alarms_Major_Present	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Major alarms present. Value: {{ humanize $value }} alarms
JunOS_Alarms_Minor_Present	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Minor alarms present. Value: {{ humanize $value }} alarms
JunOS_BGP_Flapping	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}"" has flapping BGP sessions. Value: {{ humanize $value }}
JunOS_BGP_not_Up	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" BGP session with "{{ $labels.ip }}" is DOWN.
JunOS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped incoming packets on {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 10m. Value: {{ humanize $value }}
JunOS_Interface_Receive_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for incoming packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in 10 minutes. Value: {{ humanize $value }}
JunOS_Interface_Transmit_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Physical_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has physical layer errors on "{{ $labels.name }}" ( {{ $labels.description }} ) interface.
JunOS_Interface_Oper_Status_Down	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Operational status on "{{ $labels.name }}" ( {{ $labels.description }} ) interface changed to Down.
JunOS_Power_Source_Down	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Power source "{{ $labels.item }}" status not OK. Value: {{ $labels.status }} status
JunOS_Routing_Engine_Status_NOK	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" routing engine {{ $labels.slot }} status not OK for more than 30m.
JunOS_Routing_Engine_High_CPU_Utilization	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" CPU usage for routing engine {{ $labels.slot }} is greater than 90% for more than 30m. Value: {{ humanize $value }}%
JunOS_Interface_Rate_Limit_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets due to rate limit on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Queues_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Flapping	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" interface "{{ $labels.name }}" ( {{ $labels.description }} ) in the last hour. Value: {{ humanize $value }} resets
JunOS_IPSec_Tunnels_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has {{ $value }} IPSec tunnels DOWN out of {{ printf `junos_ipsec_configured_tunnels{alias="%s"}` $labels.alias
JunOS_High_Memory_Heap_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" memory heap utilization has exceeded the threshold of 60% for more than 5 minutes. Value: {{ humanize $value }}%

alert-rules/allclusters/snmp-cisco-ncs.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_NCS_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_NCS_High_CPU_Utilization	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU utilization is greater than 90% for more than 30m. Value: {{ $value }}%
SNMP_Cisco_NCS_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_NCS_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_NCS_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_NCS_SensorStatus_Celsius_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_voltsDC_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_amperes_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dBm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_other_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_rpm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_unknown_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dB_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_mplsL3VpnVrf_Oper_And_Admin_Status_Mismatch	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational and Admin status mismatch on "{{ $labels.mplsL3VpnVrfName }}".

alert-rules/allclusters/snmp-cisco-nexus.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_Nexus_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_Nexus_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_Nexus_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_Nexus_High_CPU_Utilization	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU usage is greater than 90% for more than 30m. Value: {{ $value }}
SNMP_Cisco_Nexus_High_Memory_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Memory usage is greater than 60% for more than 5m. Value: {{ $value }}
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Rcvr_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg receive status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgRcvrStatus }} status
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Send_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg send status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgSendStatus }} status
SNMP_Cisco_Nexus_Port_Storm_Control_Active	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Port Storm Control is active on interface "{{ $labels.ifDescr }}" in state "{{ $labels.cpscStatus }}"
SNMP_Cisco_Nexus_bgpPeerState_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] bgpPeers connection to "{{ $labels.bgpPeerRemoteAddr }}" is in non established "{{ $labels.bgpPeerState }} state.

alert-rules/allclusters/snmp-dell-idrac.yaml

Alert Name	For	Severity	Type	Description
SNMP_DELL_iDRAC_globalSystemStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }}
SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC.
SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_controllerRollUpStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}.
SNMP_DELL_iDRAC_controllerComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}.
SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}).
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_virtualDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}.
SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}.
SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}).

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

Alert Name	For	Severity	Type	Description
SNMP_Fortinet_FortiGate_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Fortinet_FortiGate_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Fortinet_FortiGate_HA_State_Not_Synchronized	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has "{{ $labels.fgHaStatsSyncStatus }}" for HA Index "{{ $labels.fgHaStatsIndex }}".
SNMP_Fortinet_FortiGate_HA_State_Changed	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA State has changed in the last 10 minutes.
SNMP_Fortinet_FortiGate_PowerSupply_Status_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power Supply "{{ $labels.fgHwSensorEntDescr }}" status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_Hardware_Sensor_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Hardware Sensor "{{ $labels.fgHwSensorEntDescr }} status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_High_CPU_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High CPU Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Memory_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Memory Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Disk_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Disk Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_HaStatsSyncStatus_NOK	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Sync Status on unit "{{ $labels.fgHaStatsHostname }}" for more then 5m.
SNMP_Fortinet_FortiGate_HaStatus_Changed	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Status changed to "{{ $labels.fgVdEntHaState }}" in the last 30m.

alert-rules/allclusters/snmp-hp-cpq.yaml

Alert Name	For	Severity	Type	Description
SNMP_HP_CPQ_Overall_Health_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}"
SNMP_HP_CPQ_Event_Log_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}"
SNMP_HP_CPQ_CPU_Health_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}"
SNMP_HP_CPQ_Thermal_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}"
SNMP_HP_CPQ_Power_Supply_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}"
SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}"
SNMP_HP_CPQ_Controller_Overall_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}" status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}". This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator.
SNMP_HP_CPQ_iLO_LicenseKey_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation.

alert-rules/allclusters/snmp-lenovo-xcc.yaml

Alert Name	For	Severity	Type	Description
SNMP_Lenovo_XCC_systemHealthStat_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }}
SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }}
SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }}
SNMP_Lenovo_XCC_memoryHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }}
SNMP_Lenovo_XCC_fanHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }}
SNMP_Lenovo_XCC_voltHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }}

alert-rules/management-cluster/flux.yaml

Alert Name	For	Severity	Type	Description
Flux_Kustomization_Failing	15m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_Kustomization_Failing_Cluster	60m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_HelmRelease_Failing	15m	warning	deployment	Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Source_Failing	15m	warning	deployment	Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Resource_Suspended	2h	warning	deployment	Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended.

alert-rules/management-cluster/goldpinger.yaml

Alert Name	For	Severity	Type	Description
Goldpinger_Node_Unhealthy	5m	critical	network	Goldpinger reports unhealthy nodes: "{{ $labels.node }}"

alert-rules/management-cluster/harbor.yaml

Alert Name	For	Severity	Type	Description
Harbor_Component_Status_NOK	5m	warning	tools	Harbor component "{{ $labels.component }}" status is DOWN.

alert-rules/management-cluster/keycloak.yaml

Alert Name	For	Severity	Type	Description
Keycloak-CNPG_WAL_Disk_Usage_High	5m	warning	tools	WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB
Keycloak-CNPG_Replica_Lagging_Behind_Primary	5m	warning	tools	CNPG replica slot {{ $labels.slot_name }} on {{ $labels.pod }} is lagging {{ $value

alert-rules/management-cluster/minio.yaml

Alert Name	For	Severity	Type	Description
MinIO_Cluster_Health_Status_NOK	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK.
MinIO_Cluster_Health_Status_Unknown	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages.
MinIO_Cluster_Disk_Offline	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline.
MinIO_Cluster_Disk_Space_Usage	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 20%.
MinIO_Cluster_Disk_Space_Usage	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%.
MinIO_Cluster_Tolerance	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes.
MinIO_Nodes_Offline	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes.

alert-rules/management-cluster/rancher.yaml

Alert Name	For	Severity	Type	Description
Rancher_Token_Expires_Soon	1h	warning	rancher	Token "{{ $labels.token_internal_name }}" for user "{{ $labels.user_email }}" expires in less than 10 days.

alert-rules/management-cluster/thanos.yaml

Alert Name	For	Severity	Type	Description
Thanos-Compact_Multiple_Running	5m	warning	monitoring	More than one Thanos Compact instance is running. Current number of instances: {{ $value }}.
Thanos-Compact_Halted	5m	critical	monitoring	Thanos Compact has failed to run and now is halted.
Thanos-Compact_Compaction_Failures_Rate_High	15m	warning	monitoring	Thanos Compact is failing to execute {{ $value
Thanos-Compact_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Compact Bucket is failing to execute {{ $value
Thanos-Compact_Has_Not_Run	5m	warning	monitoring	Thanos Compact has not uploaded anything for 24 hours.
Thanos-Query_Store_Endpoints_Missing	5m	critical	monitoring	Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly.
Thanos-Query_HTTP_Request_Query_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_HTTP_Request_QueryRange_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Server_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Client_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to send {{ $value
Thanos-Query_Endpoint_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Endpoint_Groups_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Instant_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for instant queries.
Thanos-Query_Range_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for range queries.
Thanos-Query_Overload	15m	warning	monitoring	Thanos Query has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support.
Thanos-Receive_HTTP_Request_Error_Rate_High	5m	critical	monitoring	Thanos Receive is failing to handle {{ $value
Thanos-Receive_HTTP_Request_Latency_High	10m	critical	monitoring	Thanos Receive has a 99th percentile latency of {{ $value }} seconds for requests.
Thanos-Receive_Replication_Failures_Rate_High	5m	warning	monitoring	Thanos Receive is failing to replicate {{ $value
Thanos-Receive_Forward_Request_Failures_Rate_High	5m	info	monitoring	Thanos Receive is failing to forward {{ $value
Thanos-Receive_Hashring_File_Refresh_Failures_Rate_High	15m	warning	monitoring	Thanos Receive is failing to refresh hashring file, {{ $value
Thanos-Receive_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload hashring configurations.
Thanos-Receive_No_Upload	3h	critical	monitoring	Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage.
Thanos-Receive_Limits_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload the limits configuration.
Thanos-Receive_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Receive Bucket is failing to execute {{ $value
Thanos-Store_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Store is failing to handle {{ $value
Thanos-Store_Series_Gate_Latency_High	10m	warning	monitoring	Thanos Store has a 99th percentile latency of {{ $value }} seconds for store series gate requests.
Thanos-Store_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Store Bucket is failing to execute {{ $value
Thanos-Store_Objstore_Operation_Latency_High	10m	warning	monitoring	Thanos Store Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations.
Thanos-Store_Block_Drop_Rate_high	10m	warning	monitoring	Thanos Store is evicting blocks from its in-memory cache at a high rate. This may increase query latency and indicate that the index cache size is insufficient for your workload.
Thanos-Ruler_Queue_is_Dropping_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to queue alerts.
Thanos-Ruler_Sender_is_Failing_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager.
Thanos-Ruler_Rule_Evaluation_Failures_Rate_High	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to evaluate rules.
Thanos-Ruler_Rule_Evaluation_Warnings_Rate_High	15m	info	monitoring	Thanos Rule {{ $labels.pod }} has high number of evaluation warnings.
Thanos-Ruler_Rule_Evaluation_Latency_High	5m	warning	monitoring	Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}.
Thanos-Ruler_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value
Thanos-Ruler_Config_Reload_Failure	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration.
Thanos-Ruler_Query_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Ruler {{ $labels.pod }} has {{ $value
Thanos-Ruler_Alertmanager_DNS_Failure_Lookup_Rate_High	15m	warning	monitoring	Thanos Rule {{ $labels.pod }} has {{ $value
Thanos-Ruler_No_Evaluation_For_10_Intervals	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval.
Thanos-Ruler_No_Rule_Evaluations	5m	critical	monitoring	Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes.
Thanos-Rule_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Rule Bucket is failing to execute {{ $value
Thanos-Component_Compact_is_Down	5m	critical	monitoring	Thanos Compact has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Query_is_Down	5m	critical	monitoring	Thanos Query has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_QueryFrontend_is_Down	5m	critical	monitoring	Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Receive_is_Down	5m	critical	monitoring	Thanos Receive has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Rule_is_Down	5m	critical	monitoring	Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Store_is_Down	5m	critical	monitoring	Thanos Store has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Block_Meta_Sync_Failures	10m	critical	monitoring	Thanos "{{ $labels.container }}" has failed to fetch or parse some block metadata from object storage in the last 10 minutes. This may cause missing data in queries or failed compactions.

`v0.3.8`: sylva-prometheus-rules: 0.3.8

Compare Source

Merge Requests integrated in this release

2 merge requests were integrated in this repo between 0.3.7 and 0.3.8. These notes don't account for the MRs merged in secondary repos.

Other

Remove MinIO_Cluster_Disk_Space_Will_Fill_Up_Soon rule !128

CI

Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.0 !142 renovate

Contributors

1 person contributed.

Alin H

sylva-prometheus-rules

Generate PrometheusRule objects for consumption by Prometheus

Overview

There are two mechanisms that control which rules are deployed

createRules selects which directories are considered
optional_rules selects which files in those directories are added to the Configmap

Rules overrides

.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/

If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other clusters are specified

This allows for rule overriding. Example:

createRules:
  allclusters: true
  management-cluster: true

alert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml

alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml

First the PrometheusRule with the flux, minio and health-alerts name from management-cluster are created.
Then health-alerts and dummy from allcluster are parsed. Since health-alerts is already applied from mananagement-cluster it will not be applied again. dummy will be applied since it doesn't override anything

This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster

Rules activation

.Values.optional_rules controls which rules are enabled for optional components

Details about rules

alert-rules/allclusters/junos-exporter.yaml

Alert Name	For	Severity	Type	Description
JunOS_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Down_All_in_Group	5m	critical	network	All targets in cluster "{{ $labels.cluster_name }}" in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" could not be scraped.
JunOS_Rebooted	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" recently rebooted.
JunOS_Alarms_Major_Present	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Major alarms present. Value: {{ humanize $value }} alarms
JunOS_Alarms_Minor_Present	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has Minor alarms present. Value: {{ humanize $value }} alarms
JunOS_BGP_Flapping	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}"" has flapping BGP sessions. Value: {{ humanize $value }}
JunOS_BGP_not_Up	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" BGP session with "{{ $labels.ip }}" is DOWN.
JunOS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped incoming packets on {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 10m. Value: {{ humanize $value }}
JunOS_Interface_Receive_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for incoming packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has dropped outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in 10 minutes. Value: {{ humanize $value }}
JunOS_Interface_Transmit_Errors	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has errors for outgoing packets on "{{ $labels.name }}" ( {{ $labels.description }} ) interface in the last hour for 20% of the time. Value: {{ humanize $value }}%
JunOS_Interface_Physical_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has physical layer errors on "{{ $labels.name }}" ( {{ $labels.description }} ) interface.
JunOS_Interface_Oper_Status_Down	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Operational status on "{{ $labels.name }}" ( {{ $labels.description }} ) interface changed to Down.
JunOS_Power_Source_Down	5m	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" Power source "{{ $labels.item }}" status not OK. Value: {{ $labels.status }} status
JunOS_Routing_Engine_Status_NOK	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" routing engine {{ $labels.slot }} status not OK for more than 30m.
JunOS_Routing_Engine_High_CPU_Utilization	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" CPU usage for routing engine {{ $labels.slot }} is greater than 90% for more than 30m. Value: {{ humanize $value }}%
JunOS_Interface_Rate_Limit_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets due to rate limit on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Queues_Packets_Drop	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" is dropping packets on queue {{ $labels.queue_number }} - {{ $labels.name }} ( {{ $labels.description }} ) interface in the last 5m. Value: {{ humanize $value }}
JunOS_Interface_Flapping	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" interface "{{ $labels.name }}" ( {{ $labels.description }} ) in the last hour. Value: {{ humanize $value }} resets
JunOS_IPSec_Tunnels_Down	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" has {{ $value }} IPSec tunnels DOWN out of {{ printf `junos_ipsec_configured_tunnels{alias="%s"}` $labels.alias
JunOS_High_Memory_Heap_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] in group "{{ $labels.device_group }} / {{ $labels.redundancy_group }}" memory heap utilization has exceeded the threshold of 60% for more than 5 minutes. Value: {{ humanize $value }}%

alert-rules/allclusters/snmp-cisco-ncs.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_NCS_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_NCS_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_NCS_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_NCS_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_NCS_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_NCS_SensorStatus_Celsius_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_voltsDC_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_amperes_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dBm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_other_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_rpm_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_unknown_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_SensorStatus_dB_NOK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] sensor "{{ $labels.entPhysicalName }}" of type "{{ $labels.entSensorType }}" status is "{{ $labels.entSensorStatus }}".
SNMP_Cisco_NCS_mplsL3VpnVrf_Oper_And_Admin_Status_Mismatch	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational and Admin status mismatch on "{{ $labels.mplsL3VpnVrfName }}".

alert-rules/allclusters/snmp-cisco-nexus.yaml

Alert Name	For	Severity	Type	Description
SNMP_Cisco_Nexus_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Cisco_Nexus_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Cisco_Nexus_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Cisco_Nexus_FRU_Power_Oper_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power supply "{{ $labels.entPhysicalName }}" oper status not OK. Value: {{ $labels.cefcFRUPowerOperStatus }} status
SNMP_Cisco_Nexus_High_CPU_Utilization	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] CPU usage is greater than 90% for more than 30m. Value: {{ $value }}
SNMP_Cisco_Nexus_High_Memory_Utilization	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Memory usage is greater than 60% for more than 5m. Value: {{ $value }}
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Rcvr_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg receive status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgRcvrStatus }} status
SNMP_Cisco_Nexus_Vpc_Peer_Keep_Alive_Msg_Send_Status_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Vpc Peer "{{ $labels.cVpcPeerKeepAliveVrfName }}" KeepAlive msg send status not OK. Value: {{ $labels.cVpcPeerKeepAliveMsgSendStatus }} status
SNMP_Cisco_Nexus_Port_Storm_Control_Active	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Port Storm Control is active on interface "{{ $labels.ifDescr }}" in state "{{ $labels.cpscStatus }}"
SNMP_Cisco_Nexus_bgpPeerState_not_OK	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] bgpPeers connection to "{{ $labels.bgpPeerRemoteAddr }}" is in non established "{{ $labels.bgpPeerState }} state.

alert-rules/allclusters/snmp-dell-idrac.yaml

Alert Name	For	Severity	Type	Description
SNMP_DELL_iDRAC_globalSystemStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }}
SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC.
SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC.
SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC.
SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC.
SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC.
SNMP_DELL_iDRAC_controllerRollUpStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}.
SNMP_DELL_iDRAC_controllerComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}.
SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}.
SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}).
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }}
SNMP_DELL_iDRAC_virtualDiskState_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}.
SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}.
SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}).

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

Alert Name	For	Severity	Type	Description
SNMP_Fortinet_FortiGate_SNMP_get_failed	30m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get failed.
SNMP_Fortinet_FortiGate_SNMP_get_returned_too_few_results	1h	info	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] SNMP get returned too few results. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Flapping	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Interface "{{ $labels.ifDescr }}" flapped in the last 60 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Receive_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for incoming packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Drops	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has dropped outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Transmit_Errors	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has errors for outgoing packets on "{{ $labels.ifDescr }}" interface in the last 5m. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_Interface_Oper_Status_Down	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Operational status on "{{ $labels.ifDescr }}" interface changed to Down
SNMP_Fortinet_FortiGate_HA_State_Not_Synchronized	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] has "{{ $labels.fgHaStatsSyncStatus }}" for HA Index "{{ $labels.fgHaStatsIndex }}".
SNMP_Fortinet_FortiGate_HA_State_Changed	0m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA State has changed in the last 10 minutes.
SNMP_Fortinet_FortiGate_PowerSupply_Status_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Power Supply "{{ $labels.fgHwSensorEntDescr }}" status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_Hardware_Sensor_Alarm_Active	5m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] Hardware Sensor "{{ $labels.fgHwSensorEntDescr }} status alarm is active in the last 5 minutes.
SNMP_Fortinet_FortiGate_High_CPU_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High CPU Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Memory_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Memory Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_High_Disk_Usage	30m	warning	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] High Disk Usage in the last 30 minutes. Value: {{ humanize $value }}
SNMP_Fortinet_FortiGate_HaStatsSyncStatus_NOK	5m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Sync Status on unit "{{ $labels.fgHaStatsHostname }}" for more then 5m.
SNMP_Fortinet_FortiGate_HaStatus_Changed	0m	error	network	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] HA Status changed to "{{ $labels.fgVdEntHaState }}" in the last 30m.

alert-rules/allclusters/snmp-hp-cpq.yaml

Alert Name	For	Severity	Type	Description
SNMP_HP_CPQ_Overall_Health_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}"
SNMP_HP_CPQ_Event_Log_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}"
SNMP_HP_CPQ_CPU_Health_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}"
SNMP_HP_CPQ_Thermal_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}"
SNMP_HP_CPQ_Power_Supply_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}"
SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}"
SNMP_HP_CPQ_Controller_Overall_Condition_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}" status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}". This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator.
SNMP_HP_CPQ_iLO_LicenseKey_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation.

alert-rules/allclusters/snmp-lenovo-xcc.yaml

Alert Name	For	Severity	Type	Description
SNMP_Lenovo_XCC_systemHealthStat_NOK	5m	critical	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }}
SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }}
SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }}
SNMP_Lenovo_XCC_memoryHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }}
SNMP_Lenovo_XCC_fanHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }}
SNMP_Lenovo_XCC_voltHealthStatus_NOK	5m	warning	hardware	Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }}

alert-rules/management-cluster/flux.yaml

Alert Name	For	Severity	Type	Description
Flux_Kustomization_Failing	15m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_Kustomization_Failing_Cluster	60m	warning	deployment	Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile.
Flux_HelmRelease_Failing	15m	warning	deployment	Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Source_Failing	15m	warning	deployment	Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile.
Flux_Resource_Suspended	2h	warning	deployment	Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended.

alert-rules/management-cluster/goldpinger.yaml

Alert Name	For	Severity	Type	Description
Goldpinger_Node_Unhealthy	5m	critical	network	Goldpinger reports unhealthy nodes: "{{ $labels.node }}"

alert-rules/management-cluster/harbor.yaml

Alert Name	For	Severity	Type	Description
Harbor_Component_Status_NOK	5m	warning	tools	Harbor component "{{ $labels.component }}" status is DOWN.

alert-rules/management-cluster/keycloak.yaml

Alert Name	For	Severity	Type	Description
Keycloak-CNPG_WAL_Disk_Usage_High	5m	warning	tools	WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB

alert-rules/management-cluster/minio.yaml

Alert Name	For	Severity	Type	Description
MinIO_Cluster_Health_Status_NOK	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK.
MinIO_Cluster_Health_Status_Unknown	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages.
MinIO_Cluster_Disk_Offline	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline.
MinIO_Cluster_Disk_Space_Usage	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 20%.
MinIO_Cluster_Disk_Space_Usage	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%.
MinIO_Cluster_Tolerance	5m	critical	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes.
MinIO_Nodes_Offline	5m	warning	monitoring	MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes.

alert-rules/management-cluster/rancher.yaml

Alert Name	For	Severity	Type	Description
Rancher_Token_Expires_Soon	1h	warning	rancher	Token "{{ $labels.token_internal_name }}" for user "{{ $labels.user_email }}" expires in less than 10 days.

alert-rules/management-cluster/thanos.yaml

Alert Name	For	Severity	Type	Description
Thanos-Compact_Multiple_Running	5m	warning	monitoring	More than one Thanos Compact instance is running. Current number of instances: {{ $value }}.
Thanos-Compact_Halted	5m	critical	monitoring	Thanos Compact has failed to run and now is halted.
Thanos-Compact_Compaction_Failures_Rate_High	15m	warning	monitoring	Thanos Compact is failing to execute {{ $value
Thanos-Compact_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Compact Bucket is failing to execute {{ $value
Thanos-Compact_Has_Not_Run	5m	warning	monitoring	Thanos Compact has not uploaded anything for 24 hours.
Thanos-Query_Store_Endpoints_Missing	5m	critical	monitoring	Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly.
Thanos-Query_HTTP_Request_Query_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_HTTP_Request_QueryRange_Error_Rate_High	5m	critical	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Server_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to handle {{ $value
Thanos-Query_GRPC_Client_Error_Rate_High	5m	warning	monitoring	Thanos Query is failing to send {{ $value
Thanos-Query_Endpoint_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Endpoint_Groups_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Query has {{ $value
Thanos-Query_Instant_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for instant queries.
Thanos-Query_Range_Latency_High	10m	critical	monitoring	Thanos Query has a 99th percentile latency of {{ $value }} seconds for range queries.
Thanos-Query_Overload	15m	warning	monitoring	Thanos Query has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support.
Thanos-Receive_HTTP_Request_Error_Rate_High	5m	critical	monitoring	Thanos Receive is failing to handle {{ $value
Thanos-Receive_HTTP_Request_Latency_High	10m	critical	monitoring	Thanos Receive has a 99th percentile latency of {{ $value }} seconds for requests.
Thanos-Receive_Replication_Failures_Rate_High	5m	warning	monitoring	Thanos Receive is failing to replicate {{ $value
Thanos-Receive_Forward_Request_Failures_Rate_High	5m	info	monitoring	Thanos Receive is failing to forward {{ $value
Thanos-Receive_Hashring_File_Refresh_Failures_Rate_High	15m	warning	monitoring	Thanos Receive is failing to refresh hashring file, {{ $value
Thanos-Receive_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload hashring configurations.
Thanos-Receive_No_Upload	3h	critical	monitoring	Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage.
Thanos-Receive_Limits_Config_Reload_Failure	5m	warning	monitoring	Thanos Receive has not been able to reload the limits configuration.
Thanos-Receive_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Receive Bucket is failing to execute {{ $value
Thanos-Store_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Store is failing to handle {{ $value
Thanos-Store_Series_Gate_Latency_High	10m	warning	monitoring	Thanos Store has a 99th percentile latency of {{ $value }} seconds for store series gate requests.
Thanos-Store_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Store Bucket is failing to execute {{ $value
Thanos-Store_Objstore_Operation_Latency_High	10m	warning	monitoring	Thanos Store Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations.
Thanos-Store_Block_Drop_Rate_high	10m	warning	monitoring	Thanos Store is evicting blocks from its in-memory cache at a high rate. This may increase query latency and indicate that the index cache size is insufficient for your workload.
Thanos-Ruler_Queue_is_Dropping_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to queue alerts.
Thanos-Ruler_Sender_is_Failing_Alerts	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager.
Thanos-Ruler_Rule_Evaluation_Failures_Rate_High	5m	critical	monitoring	Thanos Rule {{ $labels.pod }} is failing to evaluate rules.
Thanos-Ruler_Rule_Evaluation_Warnings_Rate_High	15m	info	monitoring	Thanos Rule {{ $labels.pod }} has high number of evaluation warnings.
Thanos-Ruler_Rule_Evaluation_Latency_High	5m	warning	monitoring	Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}.
Thanos-Ruler_GRPC_Error_Rate_High	5m	warning	monitoring	Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value
Thanos-Ruler_Config_Reload_Failure	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration.
Thanos-Ruler_Query_DNS_Lookup_Failure_Rate_High	15m	warning	monitoring	Thanos Ruler {{ $labels.pod }} has {{ $value
Thanos-Ruler_Alertmanager_DNS_Failure_Lookup_Rate_High	15m	warning	monitoring	Thanos Rule {{ $labels.pod }} has {{ $value
Thanos-Ruler_No_Evaluation_For_10_Intervals	5m	info	monitoring	Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval.
Thanos-Ruler_No_Rule_Evaluations	5m	critical	monitoring	Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes.
Thanos-Rule_Bucket_Operation_Failures_Rate_High	15m	warning	monitoring	Thanos Rule Bucket is failing to execute {{ $value
Thanos-Component_Compact_is_Down	5m	critical	monitoring	Thanos Compact has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Query_is_Down	5m	critical	monitoring	Thanos Query has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_QueryFrontend_is_Down	5m	critical	monitoring	Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Receive_is_Down	5m	critical	monitoring	Thanos Receive has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Rule_is_Down	5m	critical	monitoring	Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Store_is_Down	5m	critical	monitoring	Thanos Store has disappeared. Prometheus target for the component cannot be discovered.
Thanos-Component_Block_Meta_Sync_Failures	10m	critical	monitoring	Thanos "{{ $labels.container }}" has failed to fetch or parse some block metadata from object storage in the last 10 minutes. This may cause missing data in queries or failed compactions.

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this MR and you won't be reminded about this update again.

If you want to rebase/retry this MR, check this box

This MR has been generated by Renovate Bot Sylva instance.

CI configuration couldn't be handle by MR description. A dedicated comment has been posted to control it.

If no checkbox is checked, a default pipeline will be enabled (capm3, or capo if capo label is set)

Edited May 22, 2026 by Sylva Renovate bot

Update dependency https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git to v0.3.10 (main)

Release Notes

v0.3.10: sylva-prometheus-rules: 0.3.10

Merge Requests integrated in this release

Observability

Other

Contributors

sylva-prometheus-rules

Overview

Rules overrides

Rules activation

alert-rules/allclusters/goldpinger.yaml

alert-rules/allclusters/junos-exporter.yaml

alert-rules/allclusters/snmp-cisco-ncs.yaml

alert-rules/allclusters/snmp-cisco-nexus.yaml

alert-rules/allclusters/snmp-dell-idrac.yaml

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

alert-rules/allclusters/snmp-hp-cpq.yaml

alert-rules/allclusters/snmp-lenovo-xcc.yaml

alert-rules/management-cluster/flux.yaml

alert-rules/management-cluster/harbor.yaml

alert-rules/management-cluster/keycloak.yaml

alert-rules/management-cluster/minio.yaml

alert-rules/management-cluster/rancher.yaml

alert-rules/management-cluster/thanos.yaml

v0.3.9: sylva-prometheus-rules: 0.3.9

Merge Requests integrated in this release

Observability

Other

Contributors

sylva-prometheus-rules

Overview

Rules overrides

Rules activation

alert-rules/allclusters/junos-exporter.yaml

alert-rules/allclusters/snmp-cisco-ncs.yaml

alert-rules/allclusters/snmp-cisco-nexus.yaml

alert-rules/allclusters/snmp-dell-idrac.yaml

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

alert-rules/allclusters/snmp-hp-cpq.yaml

alert-rules/allclusters/snmp-lenovo-xcc.yaml

alert-rules/management-cluster/flux.yaml

alert-rules/management-cluster/goldpinger.yaml

alert-rules/management-cluster/harbor.yaml

alert-rules/management-cluster/keycloak.yaml

alert-rules/management-cluster/minio.yaml

alert-rules/management-cluster/rancher.yaml

alert-rules/management-cluster/thanos.yaml

v0.3.8: sylva-prometheus-rules: 0.3.8

Merge Requests integrated in this release

Other

CI

Contributors

sylva-prometheus-rules

Overview

Rules overrides

Rules activation

alert-rules/allclusters/junos-exporter.yaml

alert-rules/allclusters/snmp-cisco-ncs.yaml

alert-rules/allclusters/snmp-cisco-nexus.yaml

alert-rules/allclusters/snmp-dell-idrac.yaml

alert-rules/allclusters/snmp-fortinet-fortigate.yaml

alert-rules/allclusters/snmp-hp-cpq.yaml

alert-rules/allclusters/snmp-lenovo-xcc.yaml

alert-rules/management-cluster/flux.yaml

alert-rules/management-cluster/goldpinger.yaml

alert-rules/management-cluster/harbor.yaml

alert-rules/management-cluster/keycloak.yaml

alert-rules/management-cluster/minio.yaml

alert-rules/management-cluster/rancher.yaml

alert-rules/management-cluster/thanos.yaml

Configuration

Merge request reports

`v0.3.10`: sylva-prometheus-rules: 0.3.10

`v0.3.9`: sylva-prometheus-rules: 0.3.9

`v0.3.8`: sylva-prometheus-rules: 0.3.8