Loading
Update Sylva-elements (release-1.5) (patch)
This MR contains the following updates:
| Package | Update | Change |
|---|---|---|
| https://gitlab.com/sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas.git | patch | 1.2.3 -> 1.2.4 |
| https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git | patch | 0.1.3 -> 0.1.4 |
| https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules.git | patch | 0.2.5 -> 0.2.6 |
| registry.gitlab.com/sylva-projects/sylva-elements/container-images/keytool-image | patch | 0.2.1 -> 0.2.2 |
| registry.gitlab.com/sylva-projects/sylva-elements/container-images/kube-job | patch | v1.1.0 -> v1.1.1 |
⚠️ WarningSome dependencies could not be looked up. Check the Dependency Dashboard for more information.
Release Notes
sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/capo-contrail-bgpaas.git)
v1.2.4: capo-contrail-bgpaas: 1.2.4
Merge Requests integrated in this release
1 merge request was integrated in this repo between 1.2.3 and 1.2.4. These notes don't account for the MRs merged in secondary repos.
Other
- [backport-1.2 sylva-1.5] fix missing heatstack fields !147
Contributors
1 person contributed.
sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-prometheus-rules.git)
v0.1.4: sylva-prometheus-rules: 0.1.4
Merge Requests integrated in this release
4 merge requests were integrated in this repo between 0.0.17 and 0.1.4. These notes don't account for the MRs merged in secondary repos.
Other
- [backport-1.5] Configure CNPG high disk usage alerts before it gets full and keycloak fails !125 backport (issues: #9)
- [backport 1.5] Remove MinIO_Cluster_Disk_Space_Will_Fill_Up_Soon rule !145
- [backport-1.5] Create alert to detect cnpg lagging replicas !149
CI
Contributors
2 people contributed.
sylva-prometheus-rules
Generate PrometheusRule objects for consumption by Prometheus
Overview
There are two mechanisms that control which rules are deployed
createRulesselects which directories are consideredoptional_rulesselects which files in those directories are added to the Configmap
Rules overrides
.Values.createRules controls which cluster rules are checked and the keys represent the directories under alert-rules/
If .Values.createRules.allclusters is true (default) then the alert-rules/allclusters/*yaml rules are parsed last, regardless of what other
clusters are specified
This allows for rule overriding. Example:
createRules:
allclusters: true
management-cluster: truealert-rules/allclusters/health-alerts.yaml
alert-rules/allclusters/dummy.yaml
alert-rules/management-cluster/flux.yaml
alert-rules/management-cluster/health-alerts.yaml
alert-rules/management-cluster/minio.yaml- First the
PrometheusRulewith the flux, minio and health-alerts name frommanagement-clusterare created. - Then health-alerts and dummy from
allclusterare parsed. Since health-alerts is already applied frommananagement-clusterit will not be applied again. dummy will be applied since it doesn't override anything
This in effect allows the user to override the health-alerts from allclusters with health-alerts form management-cluster
Rules activation
.Values.optional_rules controls which rules are enabled for optional components
Details about rules
alert-rules/allclusters/health-alerts.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| KubeJobFailedAllClusters | 15m | warning | k8s | Job "{{ $labels.namespace }}"/ "{{ $labels.job_name }}" failed to complete. Removing failed job after investigation should clear this alert. |
alert-rules/allclusters/snmp-dell-idrac.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_DELL_iDRAC_globalSystemStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - globalSystemStatus is NOK. Current state is: {{ $labels.globalSystemStatus }} |
| SNMP_DELL_iDRAC_systemStateBatteryStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateBatteryStatus is NOK. Current state is: {{ $labels.systemStateBatteryStatusCombined }}. Check RAID Controller BBU or CMOS battery in iDRAC. |
| SNMP_DELL_iDRAC_systemStateCoolingDeviceStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingDeviceStatusCombined }}. Check system fans in iDRAC. |
| SNMP_DELL_iDRAC_systemStateCoolingUnitStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateCoolingDeviceStatus is NOK. Current state is: {{ $labels.systemStateCoolingUnitStatusCombined }}. Check system fans in iDRAC. |
| SNMP_DELL_iDRAC_systemStateMemoryDeviceStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateMemoryDeviceStatus is NOK. Current state is: {{ $labels.systemStateMemoryDeviceStatusCombined }}. Check system volatile memory in iDRAC. |
| SNMP_DELL_iDRAC_systemStatePowerSupplyStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerSupplyStatus is NOK. Current state is: {{ $labels.systemStatePowerSupplyStatusCombined }}. Check system power supply in iDRAC. |
| SNMP_DELL_iDRAC_systemStatePowerUnitStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStatePowerUnitStatus is NOK. Current state is: {{ $labels.systemStatePowerUnitStatusCombined }}. Check system power supply or external power delivery in iDRAC. |
| SNMP_DELL_iDRAC_systemStateProcessorDeviceStatusCombined_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateProcessorDeviceStatus is NOK. Current state is: {{ $labels.systemStateProcessorDeviceStatusCombined }}. Check system processor in iDRAC. |
| SNMP_DELL_iDRAC_systemStateTemperatureStatisticsStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatisticsStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatisticsStatusCombined }}. Check system temperatures in iDRAC. |
| SNMP_DELL_iDRAC_systemStateTemperatureStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateTemperatureStatus is NOK. Current state is: {{ $labels.systemStateTemperatureStatusCombined }}. Check system temperatures in iDRAC. |
| SNMP_DELL_iDRAC_systemStateVoltageStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateVoltageStatus is NOK. Current state is: {{ $labels.systemStateVoltageStatusCombined }}. Check system voltage in iDRAC. |
| SNMP_DELL_iDRAC_systemStateAmperageStatusCombined_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemStateAmperageStatus is NOK. Current state is: {{ $labels.systemStateAmperageStatusCombined }}. Check system voltage in iDRAC. |
| SNMP_DELL_iDRAC_controllerRollUpStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerRollUpStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerRollUpStatus }}. |
| SNMP_DELL_iDRAC_controllerComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - controllerComponentStatus is NOK for controllerNumber {{ $labels.controllerNumber }} ( {{ $labels.controllerName }}). Current state is: {{ $labels.controllerComponentStatus }}. |
| SNMP_DELL_iDRAC_physicalDiskState_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskState is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskState }}. |
| SNMP_DELL_iDRAC_physicalDiskComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskComponentStatus is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Current state is: {{ $labels.physicalDiskComponentStatus }}. |
| SNMP_DELL_iDRAC_physicalDiskSmartAlertIndication_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskSmartAlertIndication is NOK for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). |
| SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_WARNING | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 40 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }} |
| SNMP_DELL_iDRAC_physicalDiskRemainingRatedWriteEndurance_CRITICAL | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - physicalDiskRemainingRatedWriteEndurance is less than 20 for physicalDiskNumber {{ $labels.physicalDiskNumber }} ( {{ $labels.physicalDiskDisplayName }}). Value: {{ humanize $value }} |
| SNMP_DELL_iDRAC_virtualDiskState_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskState is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskState }}. |
| SNMP_DELL_iDRAC_virtualDiskComponentStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskComponentStatus is NOK for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). Current state is: {{ $labels.virtualDiskComponentStatus }}. |
| SNMP_DELL_iDRAC_virtualDiskBadBlocksDetected | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - virtualDiskBadBlocksDetected for virtualDiskNumber {{ $labels.virtualDiskNumber }} ( {{ $labels.virtualDiskDisplayName }}). |
alert-rules/allclusters/snmp-hp-cpq.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_HP_CPQ_Overall_Health_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Overall health status is NOK. Value: "{{ $labels.cpqHeMibCondition }}" |
| SNMP_HP_CPQ_Event_Log_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Event Log Condition is NOK. Value: "{{ $labels.cpqHeEventLogCondition }}"}} |
| SNMP_HP_CPQ_CPU_Health_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - CPU status is NOK. Value: "{{ $labels.cpqSeCpuCondition }}"}} |
| SNMP_HP_CPQ_Thermal_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Thermal condition status is NOK. Value: "{{ $labels.cpqHeThermalCondition }}"}} |
| SNMP_HP_CPQ_Power_Supply_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ]- Power supply condition status is NOK. Value: "{{ $labels.cpqHeFltTolPwrSupplyCondition }}"}} |
| SNMP_HP_CPQ_Storage_Subsystem_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Storage subsystem condition status is NOK. Value: "{{ $labels.cpqSsMibCondition }}"}} |
| SNMP_HP_CPQ_Controller_Overall_Condition_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - Controller "{{ $labels.cpqDaCntlrIndex }}"}} status is NOK. Value: "{{ $labels.cpqDaCntlrCondition }}"}}. This value represents the overall condition of this controller, and any associated logical drives, physical drives, and array accelerator. |
| SNMP_HP_CPQ_iLO_LicenseKey_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - HP iLO interface is missing its License activation. |
alert-rules/allclusters/snmp-lenovo-xcc.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| SNMP_Lenovo_XCC_systemHealthStat_NOK | 5m | critical | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - systemHealthStat is not "normal". Current state is: {{ $labels.systemHealthStat }} |
| SNMP_Lenovo_XCC_cpuVpdHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - cpuVpdHealthStatus for CPU "{{ $labels.cpuVpdDescription }}" is not "normal". Current state is: {{ $labels.cpuVpdHealthStatus }} |
| SNMP_Lenovo_XCC_raidDriveHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - raidDriveHealthStatus for "{{ $labels.raidDriveName }}" is not "Normal". Current state is: {{ $labels.raidDriveHealthStatus }} |
| SNMP_Lenovo_XCC_memoryHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - memoryHealthStatus for DIMM "{{ $labels.memoryVpdDescription }}" is not "Normal". Current state is: {{ $labels.memoryHealthStatus }} |
| SNMP_Lenovo_XCC_fanHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - fanHealthStatus for Fan "{{ $labels.fanDescr }}" is not "Normal". Current state is: {{ $labels.fanHealthStatus }} |
| SNMP_Lenovo_XCC_voltHealthStatus_NOK | 5m | warning | hardware | Target "{{ $labels.alias }}" [ cluster: "{{ $labels.cluster_name }}" / address: "{{ $labels.instance }}" ] - voltHealthStatus for System Component "{{ $labels.voltDescr }}" is not "Normal". Current state is: {{ $labels.voltHealthStatus }} |
alert-rules/management-cluster/flux.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Flux_Kustomization_Failing | 15m | warning | deployment | Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile. |
| Flux_Kustomization_Failing_Cluster | 60m | warning | deployment | Flux Kustomization "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" fails to reconcile. |
| Flux_HelmRelease_Failing | 15m | warning | deployment | Flux HelmRelease "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile. |
| Flux_Source_Failing | 15m | warning | deployment | Flux Source "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace}}" fails to reconcile. |
| Flux_Resource_Suspended | 2h | warning | deployment | Flux Resource "{{ $labels.name }}" in namespace "{{ $labels.exported_namespace }}" suspended. |
alert-rules/management-cluster/harbor.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Harbor_Component_Status_NOK | 5m | warning | tools | Harbor component "{{ $labels.component }}" status is DOWN. |
alert-rules/management-cluster/health-alerts.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| KubeContainerWaitingManagement | 1h | critical | k8s | Pod "{{ $labels.namespace }}" / "{{ $labels.pod }}" has been in waiting state for more than 1 hour. |
alert-rules/management-cluster/keycloak.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Keycloak-CNPG_WAL_Disk_Usage_High | 5m | warning | tools | WAL directory usage on "{{ $labels.pod }}" has exceeded 2GiB |
| Keycloak-CNPG_Replica_Lagging_Behind_Primary | 5m | warning | tools | CNPG replica slot {{ $labels.slot_name }} on {{ $labels.pod }} is lagging {{ $value |
alert-rules/management-cluster/minio.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| MinIO_Cluster_Health_Status_NOK | 5m | critical | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status not OK. |
| MinIO_Cluster_Health_Status_Unknown | 5m | critical | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" health status is Unknown. The cluster does not return cluster metrics. Check pods logs for error messages. |
| MinIO_Cluster_Disk_Offline | 5m | critical | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" disk offline. |
| MinIO_Cluster_Disk_Space_Usage | 5m | warning | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 30%. |
| MinIO_Cluster_Disk_Space_Usage | 5m | critical | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" available disk space is less than 10%. |
| MinIO_Cluster_Tolerance | 5m | critical | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has lost quorum on pool "{{ $labels.pool }}" / set "{{ $labels.set }}" for more than 5 minutes. |
| MinIO_Nodes_Offline | 5m | warning | storage | MinIO cluster "{{ $labels.minio_tenant }}" in namespace "{{ $labels.namespace }}" has offline nodes. |
alert-rules/management-cluster/thanos.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| ThanosQueryStoreEndpointsMissing | 5m | critical | monitoring | Thanos Query is missing "{{ $labels.store_type }}" store type. Metrics served by this store type will not be available which can lead to alerting rules not evaluating properly. |
| ThanosCompactMultipleRunning | 5m | warning | monitoring | More than one Thanos Compact instance is running. Current number of instances: {{ $value }}. |
| ThanosCompactHalted | 5m | warning | monitoring | Thanos Compact {{ $labels.job }} has failed to run and now is halted. |
| ThanosCompactHighCompactionFailures | 15m | warning | monitoring | Thanos Compact {{ $labels.job }} is failing to execute {{ $value |
| ThanosCompactBucketHighOperationFailures | 15m | warning | monitoring | Thanos Compact {{ $labels.job }} Bucket is failing to execute {{ $value |
| ThanosCompactHasNotRun | 5m | warning | monitoring | Thanos Compact {{ $labels.job }} has not uploaded anything for 24 hours. |
| ThanosQueryHttpRequestQueryErrorRateHigh | 5m | critical | monitoring | Thanos Query {{ $labels.job }} is failing to handle {{ $value |
| ThanosQueryHttpRequestQueryRangeErrorRateHigh | 5m | critical | monitoring | Thanos Query {{ $labels.job }} is failing to handle {{ $value |
| ThanosQueryGrpcServerErrorRate | 5m | warning | monitoring | Thanos Query {{ $labels.job }} is failing to handle {{ $value |
| ThanosQueryGrpcClientErrorRate | 5m | warning | monitoring | Thanos Query {{ $labels.job }} is failing to send {{ $value |
| ThanosQueryHighDNSFailures | 15m | warning | monitoring | Thanos Query {{ $labels.job }} have {{ $value |
| ThanosQueryInstantLatencyHigh | 10m | critical | monitoring | Thanos Query {{ $labels.job }} has a 99th percentile latency of {{ $value }} seconds for instant queries. |
| ThanosQueryRangeLatencyHigh | 10m | critical | monitoring | Thanos Query {{ $labels.job }} has a 99th percentile latency of {{ $value }} seconds for range queries. |
| ThanosQueryOverload | 15m | warning | monitoring | Thanos Query {{ $labels.job }} has been overloaded for more than 15 minutes. This may be a symptom of excessive simultanous complex requests, low performance of the Prometheus API, or failures within these components. Assess the health of the Thanos query instances, the connnected Prometheus instances, look for potential senders of these requests and then contact support. |
| ThanosReceiveHttpRequestErrorRateHigh | 5m | critical | monitoring | Thanos Receive {{ $labels.job }} is failing to handle {{ $value |
| ThanosReceiveHttpRequestLatencyHigh | 10m | critical | monitoring | Thanos Receive {{ $labels.job }} has a 99th percentile latency of {{ $value }} seconds for requests. |
| ThanosReceiveHighReplicationFailures | 5m | warning | monitoring | Thanos Receive {{ $labels.job }} is failing to replicate {{ $value |
| ThanosReceiveHighForwardRequestFailures | 5m | info | monitoring | Thanos Receive {{ $labels.job }} is failing to forward {{ $value |
| ThanosReceiveHighHashringFileRefreshFailures | 15m | warning | monitoring | Thanos Receive {{ $labels.job }} is failing to refresh hashring file, {{ $value |
| ThanosReceiveConfigReloadFailure | 5m | warning | monitoring | Thanos Receive {{ $labels.job }} has not been able to reload hashring configurations. |
| ThanosReceiveNoUpload | 3h | critical | monitoring | Thanos Receive {{ $labels.pod }} has not uploaded latest data to object storage. |
| ThanosReceiveLimitsConfigReloadFailure | 5m | warning | monitoring | Thanos Receive {{ $labels.job }} has not been able to reload the limits configuration. |
| ThanosReceiveLimitsHighMetaMonitoringQueriesFailureRate | 5m | warning | monitoring | Thanos Receive {{ $labels.job }} is failing for {{ $value |
| ThanosReceiveTenantLimitedByHeadSeries | 5m | warning | monitoring | Thanos Receive tenant {{ $labels.tenant }} is limited by head series. |
| ThanosStoreGrpcErrorRate | 5m | warning | monitoring | Thanos Store {{ $labels.job }} is failing to handle {{ $value |
| ThanosStoreSeriesGateLatencyHigh | 10m | warning | monitoring | Thanos Store {{ $labels.job }} has a 99th percentile latency of {{ $value }} seconds for store series gate requests. |
| ThanosStoreBucketHighOperationFailures | 15m | warning | monitoring | Thanos Store {{ $labels.job }} Bucket is failing to execute {{ $value |
| ThanosStoreObjstoreOperationLatencyHigh | 10m | warning | monitoring | Thanos Store {{ $labels.job }} Bucket has a 99th percentile latency of {{ $value }} seconds for the bucket operations. |
| ThanosRuleQueueIsDroppingAlerts | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to queue alerts. |
| ThanosRuleSenderIsFailingAlerts | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to send alerts to alertmanager. |
| ThanosRuleHighRuleEvaluationFailures | 5m | critical | monitoring | Thanos Rule {{ $labels.pod }} is failing to evaluate rules. |
| ThanosRuleHighRuleEvaluationWarnings | 15m | info | monitoring | Thanos Rule {{ $labels.pod }} has high number of evaluation warnings. |
| ThanosRuleRuleEvaluationLatencyHigh | 5m | warning | monitoring | Thanos Rule {{ labels.pod }} has higher evaluation latency than interval for {{labels.rule_group}}. |
| ThanosRuleGrpcErrorRate | 5m | warning | monitoring | Thanos Ruler {{ $labels.pod }} is failing to handle {{ $value |
| ThanosRuleConfigReloadFailure | 5m | info | monitoring | Thanos Ruler {{ $labels.pod }} has not been able to reload its configuration. |
| ThanosRuleQueryHighDNSFailures | 15m | warning | monitoring | Thanos Ruler {{ $labels.pod }} has {{ $value |
| ThanosRuleAlertmanagerHighDNSFailures | 15m | warning | monitoring | Thanos Rule {{ $labels.pod }} has {{ $value |
| ThanosRuleNoEvaluationFor10Intervals | 5m | info | monitoring | Thanos Ruler {{ $labels.pod }} has rule groups that did not evaluate for at least 10x of their expected interval. |
| ThanosNoRuleEvaluations | 5m | critical | monitoring | Thanos Ruler {{ $labels.pod }} did not perform any rule evaluations in the past 10 minutes. |
| ThanosBucketReplicateErrorRate | 5m | critical | monitoring | Thanos Replicate is failing to run, {{ $value |
| ThanosBucketReplicateRunLatency | 5m | critical | monitoring | Thanos Replicate {{ $labels.job }} has a 99th percentile latency of {{ $value }} seconds for the replicate operations. |
| ThanosCompactIsDown | 5m | critical | monitoring | Thanos Compact has disappeared. Prometheus target for the component cannot be discovered. |
| ThanosQueryIsDown | 5m | critical | monitoring | Thanos Query has disappeared. Prometheus target for the component cannot be discovered. |
| ThanosQueryFrontendIsDown | 5m | critical | monitoring | Thanos Query Frontend has disappeared. Prometheus target for the component cannot be discovered. |
| ThanosReceiveIsDown | 5m | critical | monitoring | Thanos Receive has disappeared. Prometheus target for the component cannot be discovered. |
| ThanosRuleIsDown | 5m | critical | monitoring | Thanos Ruler has disappeared. Prometheus target for the component cannot be discovered. |
| ThanosStoreIsDown | 5m | critical | monitoring | Thanos Store has disappeared. Prometheus target for the component cannot be discovered. |
alert-rules/my-workload-cluster/health-alerts.yaml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| KubeJobFailedWorkload | 15m | warning | k8s | Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert. |
sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules (https://gitlab.com/sylva-projects/sylva-elements/helm-charts/sylva-thanos-rules.git)
v0.2.6: sylva-thanos-rules: 0.2.6
Merge Requests integrated in this release
1 merge request was integrated in this repo between 0.2.5 and 0.2.6. These notes don't account for the MRs merged in secondary repos.
Other
Contributors
1 person contributed.
sylva-thanos-rules
Generate ConfigMap object for consumption by Thanos Ruler
Details about rules
rules/_helper_kubernetes_metadata.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Metamonitoring_configuration_error_kube_namespace_labels | 45m | error | deployment | Metric "kube_namespace_labels" from cluster "{{ $labels.capi_cluster_name }}" is not exposed by "kube-state-metrics". |
| k8s-Metamonitoring_configuration_error_rancher_project_info | 45m | error | deployment | Metric "rancher_project_info" from the management cluster is not exposed by "kube-state-metrics". |
rules/clusters_state_rules.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Sylva_cluster_Prometheus_not_Sending_Data_management | 45m | critical | deployment | Prometheus server from the management cluster has not sent data in the last 45m. |
| Sylva_cluster_Prometheus_not_Sending_Data | 45m | critical | deployment | Prometheus server from cluster "{{ $labels.capi_cluster_name }}" in namespace "{{ $labels.capi_cluster_namespace }}" has not sent data in the last 45m. |
| Sylva_clusters_different_number | 45m | critical | deployment | Some cluster is not properly provisioned in Rancher, check all clusters to see if cattle-agent is properly deployed |
| Sylva_clusters_metric_absent | 45m | error | deployment | Metric "capi_cluster_info" from the management cluster is not exposed by "kube-state-metrics". |
rules/etcd.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| etcd-Members_Down | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" members are down. |
| etcd-Members_Insufficient | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" has insufficient members. Value: {{ $value }} |
| etcd-Members_No_Leader | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" member {{ $labels.instance }} has no leader. |
| etcd-High_Number_of_Leader_Changes | 5m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" {{ $value }} leader changes within the last 15 minutes. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated. |
| etcd-gRPC_High_Number_of_Failed_Requests | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" gRPC requests failed for "{{ $labels.grpc_method }}". Value: {{ $value }} |
| etcd-gRPC_High_Number_of_Failed_Requests | 5m | critical | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" gRPC requests failed for "{{ $labels.grpc_method }}". Value: {{ $value }} |
| etcd-Members_Communication_Slow | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" "{{ $labels.instance }}" to "{{ $labels.To }}" member communication is taking too long. Value: {{ $value }}s. |
| etcd-High_Number_of_Failed_Proposals | 15m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has proposal failures within the last 30 minutes. Value: {{ $value }} |
| etcd-High_Fsync_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile fync durations. Value: {{ $value }}s |
| etcd-High_Commit_Duration | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" has high 99th percentile commit durations. Value: {{ $value }}s |
| etcd-HTTP_High_Number_of_Failed_Requests | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" HTTP requests failed for "{{ $labels.method }}". Value: {{ $value }} |
| etcd-HTTP_High_Number_of_Failed_Requests | 10m | critical | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" HTTP requests failed for "{{ $labels.method }}". Value: {{ $value }} |
| etcd-HTTP_Requests_Slow | 10m | warning | etcd | etcd in cluster "{{ $labels.cluster }}" instance "{{ $labels.instance }}" HTTP requests for "{{ $labels.method }}" are slow. Value: {{ $value }}s |
rules/kubernetes_capacity.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Cluster_CPU_Overcommitted | 5m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable CPUs. Node failures may cause Pods to be unschadulable due to lack of resources |
| k8s-Cluster_Memory_Overcommitted | 5m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has allocated over 75% of the allocatable Memory. Node failures may cause Pods to be unschadulable due to lack of resources |
| k8s-Cluster_Too_Many_Pods | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" number of pods over 90% of Pod number limit. Value: {{ humanize $value }}% |
| k8s-Node_Too_Many_Pods | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" node {{ $labels.node }} number of pods over 90% of Pod number limit. Value: {{ humanize $value }}% |
| k8s-Kube_Quota_Almost_Full | 15m | warning | k8s | Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value |
| k8s-Kube_Quota_Exceeded | 15m | error | k8s | Namespace {{ $labels.namespace }} in cluster {{ $labels.cluster }} is using {{ $value |
rules/kubernetes_cluster_components.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Version_Mismatch | 4h | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" has different versions of Kubernetes components running. Value: {{ $value }} |
| k8s-Client_Errors | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server client "{{ $labels.instance }}" job "{{ $labels.job }}" is experiencing errors. Value: {{ printf "%0.0f" $value }}% |
| k8s-Client_Certificate_Expiration | 5m | warning | k8s | A client certificate used to authenticate to Kubernetes apiserver is expiring in less than 7 days on cluster {{ $labels.cluster }}. |
| k8s-Client_Certificate_Expiration | 5m | critical | k8s | A client certificate used to authenticate to Kubernetes apiserver is expiring in less than 24h on cluster {{ $labels.cluster }}. |
| k8s-API_Global_Error_Rate_High | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for over 3% of requests. Value: {{ humanize $value }}% |
| k8s-API_Error_Rate_High | 15m | warning | k8s | Kubernetes cluster "{{ $labels.cluster }}" API server is returning errors for 10% of requests for {{ $labels.verb }} {{ $labels.resource }} {{ $labels.subresource }}. Value: {{ humanize $value }}% |
| k8s-Aggregated_API_Errors | 15m | warning | k8s | Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has reported errors. It has appeared unavailable {{ $value |
| k8s-Aggregated_API_Down | 15m | warning | k8s | Kubernetes aggregated API i{{ $labels.name }}/{{ $labels.namespace }} in cluster "{{ $labels.cluster }}" has been only {{ $value |
| k8s-API_Endpoint_Down | 15m | error | k8s | Kubernetes API endpoint {{ $labels.instance }} in cluster {{ $labels.cluster }} is unreachable. |
| k8s-API_Down | 15m | critical | k8s | Kubernetes API in cluster {{ $labels.cluster }} is unreachable. |
| k8s-API_Terminated_Requests | 15m | warning | k8s | Kubernetes API in cluster {{ $labels.cluster }} has terminated {{ $value |
rules/kubernetes_jobs.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-CronJob_Status_Failed | 5m | warning | k8s | CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" failed. Last job has failed multiple times. Value: {{ $value }} |
| k8s-CronJob_Taking_Too_Long | 0m | warning | k8s | CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} in cluster "{{ $labels.cluster }}" is taking too long to completes - is over deadline. Value: {{ humanizeDuration $value }} |
| k8s-Job_not_Completed | 15m | warning | k8s | Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" is taking more than 12h to complete. |
| k8s-Job_Failed | 15m | warning | k8s | Job {{ $labels.namespace }}/{{ $labels.job_name }} in cluster "{{ $labels.cluster }}" failed to complete. Removing failed job after investigation should clear this alert. |
rules/kubernetes_nodes.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Node_Kubelet_Down | 5m | critical | k8s | Kubelet on {{ $labels.node }} in cluster "{{ $labels.cluster }}" is not reachable |
| k8s-Node_Status_OutOfDisk | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is almost out of disk space |
| k8s-Node_Status_MemoryPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under memory pressure. |
| k8s-Node_Status_DiskPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under disk pressure |
| k8s-Node_Status_PIDPressure | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" is under PID pressure |
| k8s-Node_Status_NotReady | 5m | error | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has been not been Ready for more than an hour |
| k8s-Node_Status_NetworkUnavailable | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" has NetworkUnavailable condition. |
| k8s-Node_Status_Ready_flapping | 5m | warning | k8s | Node {{ $labels.node }} in cluster "{{ $labels.cluster }}" readiness status changed {{ $value }} times in the last 15 minutes. |
rules/kubernetes_pods.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Pod_Status_not_Ready | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in a non-ready state for longer than 15 minutes. |
| k8s-Pod_Status_OOMKilled | 0m | error | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" has been restarted due to "OOMKilled" reason in the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Status_Crashlooping | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster}}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Init_Container_Status_Crashlooping | 15m | warning | k8s | Init Container from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" was restarted more than 5 times within the last hour. Value: {{ humanize $value }} |
| k8s-Pod_Container_Status_Waiting | 1h | warning | k8s | Container "{{ $labels.container }}" from Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" on node "{{ $labels.node }}" has been in waiting state for longer than 1 hour. |
| k8s-Statefulset_Replicas_not_Ready | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }} |
| k8s-Statefulset_Generation_Mismatch | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" generation for does not match. This indicates that the StatefulSet has failed but has not been rolled back |
| k8s-Statefulset_Update_not_Rolled_Out | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" update has not been rolled out |
| k8s-Statefulset_Replicas_Mismatch | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_statefulset_spec_replicas{statefulset="%s", cluster="%s"} $labels.statefulset $labels.cluster |
| k8s-Statefulset_Replicas_not_Updated | 15m | warning | k8s | Statefulset {{ $labels.namespace }}/{{ $labels.statefulset }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment |
| k8s-ReplicaSet_Replicas_Mismatch | 15m | warning | k8s | ReplicaSet {{ $labels.namespace }}/{{ $labels.replicaset }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas |
| k8s-Deployment_Replicas_not_Ready | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has replicas in "notReady" state. Value: {{ humanize $value }} |
| k8s-Deployment_Replicas_Mismatch | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" has not matched the expected number of replicas. Value: {{ $value }} / {{ printf kube_deployment_spec_replicas{deployment="%s", cluster="%s"} $labels.deployment $labels.cluster |
| k8s-Deployment_Generation_Mismatch | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" generation does not match expected one |
| k8s-Deployment_Replicas_not_Updated | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" replicas are not updated and available for deployment |
| k8s-Deployment_Rollout_Stuck | 15m | warning | k8s | Deployment {{ $labels.namespace }}/{{ $labels.deployment }} in cluster "{{ $labels.cluster }}" is not progressing for longer than 15 minutes. |
| k8s-Daemonset_Rollout_Stuck | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has less than 100% of desired pods scheduled and ready. Value: {{ humanize $value }}% |
| k8s-Daemonset_not_Scheduled | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" has unscheduled pods. Value: {{ humanize $value }} |
| k8s-Daemonset_Misscheduled | 15m | warning | k8s | Daemonset pods {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" are running where they are not supposed to. Value: {{ humanize $value }} |
| k8s-Daemonset_Generation_Mismatch | 15m | warning | k8s | Daemonset {{ $labels.namespace }}/{{ $labels.daemonset }} in cluster "{{ $labels.cluster }}" generation does not match expected one |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is greater than 80% the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_CPU | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is greater than 90% the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is greater than 80% the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_CPU | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is greater than 90% the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is greater than 80% of the resource requets. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_Low_Memory | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is greater than 90% of the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is greater than 80% of the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_Low_Memory | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is greater than 90% of the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_High_CPU | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is smaller than 20% the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_High_CPU | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is smaller than 10% the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_High_CPU | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is smaller than 20% the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_High_CPU | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" CPU utilization is smaller than 10% the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_High_Memory | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is smaller than 20% of the resource requets. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Requests_Too_High_Memory | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is smaller than 10% of the resource requests. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_High_Memory | 15m | info | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is smaller than 20% of the resource limits. Consider updating the allocated value. |
| k8s-Pod_Resource_Allocation_Limits_Too_High_Memory | 15m | warning | k8s | Pod {{ $labels.namespace }}/{{ $labels.pod }} in cluster "{{ $labels.cluster }}" memory utilization is smaller than 10% of the resource limits. Consider updating the allocated value. |
rules/kubernetes_storage.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Persistent_Volume_Disk_Space_Usage_High | 5m | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" disk space usage is over 80% used. Value: {{ printf "%0.2f" $value }}% |
| k8s-Persistent_Volume_Disk_Space_Usage_High | 5m | warning | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" disk space usage is over 90% used. Value: {{ printf "%0.2f" $value }}% |
| k8s-Persistent_Volume_Full_in_4_days | 5m | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" will fill up in 4 days at the current rate of utilization. Value: {{ printf "%0.2f" $value }}% available |
| k8s-Persistent_Volume_Inodes_Usage_High | 5m | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" inodes usage is over 80% used. Value: {{ printf "%0.2f" $value }}% |
| k8s-Persistent_Volume_Inodes_Usage_High | 5m | warning | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" inodes usage is over 90% used. Value: {{ printf "%0.2f" $value }}% |
| k8s-Persistent_Volume_Errors | 5m | warning | k8s | PersistentVolume "{{ $labels.persistentvolume }}" in cluster "{{ $labels.cluster }}" has status "{{ $labels.phase }}" |
| k8s-Persistent_Volume_Claim_Orphan | 3h | info | k8s | PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} in cluster "{{ $labels.cluster }}" is not used by any pod |
rules/kubernetes_storage_ephemeral.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| k8s-Ephemeral_Storage_Container_Usage_at_Limit | 5m | warning | k8s | Ephemeral storage usage of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is at {{ $value }}% of the limit. |
| k8s-Ephemeral_Storage_Container_Usage_Reaching_Limit | 15m | warning | k8s | Ephemeral storage limit of pod/container "{{ $labels.pod_name }}"/"{{ $labels.exported_container }}" in namespace "{{ $labels.pod_namespace }}" on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is expected to be reached within 12 hours. Currently, {{ $value }}% is used. |
| k8s-Ephemeral_Storage_Volume_Filled_Up | 5m | warning | k8s | Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" high usage. Value: {{ $value }}% |
| k8s-Ephemeral_Storage_Volume_Filling_Up | 5m | warning | k8s | Ephemeral storage volume "{{ $labels.volume_name }}" of pod "{{ $labels.pod_name }}" in namespace "{{ $labels.pod_namespace }}" in cluster "{{ $labels.cluster }}" is expected to be filled up within 12 hours. Currently, {{ $value }}% is used |
| k8s-Ephemeral_Storage_on_Node_Filling_Up | 5m | warning | k8s | Ephemeral storage on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is is gearter than 75%. Value: {{ $value }}% |
| k8s-Ephemeral_Storage_on_Node_Filling_Up | 5m | warning | k8s | Ephemeral storage on node "{{ $labels.node_name }}" in cluster "{{ $labels.cluster }}" is is gearter than 90%. Value: {{ $value }}% |
rules/longhorn.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Longhorn-Volume_Status_Critical | 5m | error | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Faulted" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Volume_Status_Warning | 5m | warning | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Degraded" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Volume_Status_Unknown | 5m | warning | storage | Longhorn volume "{{ $labels.volume }} / {{ $labels.pvc }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is in state "Unknown" for more than 5 minutes. Possible values are: 0=unknown, 1=healthy, 2=degraded, 3=faulted |
| Longhorn-Node_Storage_Warning | 5m | warning | storage | The used storage of node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value |
| Longhorn-Disk_Storage_Warning | 5m | warning | storage | The used storage of disk "{{ $labels.disk }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" is at "{{ $value |
| Longhorn-Node_Down | 5m | error | storage | There are "{{ $value |
| Longhorn-Instance_Manager_CPU_Usage_Warning | 5m | info | storage | Longhorn instance manager "{{ $labels.instance_manager }}" on node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU request at "{{ $value |
| Longhorn-Node_CPU_Usage_Warning | 5m | info | storage | Longhorn node "{{ $labels.node }}" in cluster "{{ $labels.cluster }}" has CPU Usage / CPU capacity at "{{ $value |
rules/metallb.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| MetalLB-BGP_Session_Down | 5m | error | network | MetalLB speaker {{ $labels.instance }} in cluster "{{ $labels.cluster}}" has BGP session {{ $labels.peer }} down for more than 5 minutes. |
| MetalLB-BGP_All_Sessions_Down | 5m | critical | network | MetalLB in "{{ $labels.cluster}}" all {{ $value }} BGP sessions are down for more than 5 minutes. |
| MetalLB-Address_Pool_High_Usage | 5m | info | network | MetalLB pool "{{ $labels.pool }}" in cluster "{{ $labels.cluster}}" has more than 75% of the total addresses used. |
| MetalLB-Config_Stale | 5m | warning | network | MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" has a stale config. |
| MetalLB-Config_not_Loaded | 5m | warning | network | MetalLB {{ $labels.instance }} container "{{ $labels.container }}" in cluster "{{ $labels.cluster}}" config not loaded. |
rules/monitoring_stack_components.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Monitoring-Prometheus_Bad_Config | 10m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration. |
| Monitoring-Prometheus_SD_Refresh_Failure | 20m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to refresh SD with mechanism "{{ $labels.mechanism }}". |
| Monitoring-Prometheus_Kubernetes_List_Watch_Failures | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" service discovery is experiencing failures with LIST/WATCH requests to the Kubernetes API in the last 5 minutes. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Notification_Queue_Running_Full | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" alert notification queue predicted to run full in less than 30 minutes. |
| Monitoring-Prometheus_Error_Sending_Alerts_to_Some_Alertmanagers | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has {{ printf "%.1f" $value }}% of alerts sent to Alertmanager "{{ $labels.alertmanager }}" affected by errors. |
| Monitoring-Prometheus_not_Connected_to_Alertmanagers | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not connected to any Alertmanagers. |
| Monitoring-Prometheus_TSDB_Reloads_Failing | 4h | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced TSDB reload failures in the last 3h. Value: {{ $value }} |
| Monitoring-Prometheus_TSDB_Compactions_Failing | 4h | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" experienced compaction failures in the last 3h. Value: {{ $value }} |
| Monitoring-Prometheus_not_Ingesting_Samples | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is not ingesting samples. |
| Monitoring-Prometheus_Duplicate_Timestamps | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with different values but duplicated timestamps. Value: {{ printf "%.4g" $value }} |
| Monitoring-Prometheus_Out_of_Order_Timestamps | 10m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" is dropping samples/s with timestamps arriving out of order. Value: {{ printf "%.4g" $value }} |
| Monitoring-Prometheus_Remote_Storage_Failures | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send a high number of samples to "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}% |
| Monitoring-Prometheus_Remote_Write_Behind | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write is behind for "{{ $labels.remote_name }}:{{ $labels.url }}". Value: {{ printf "%.1f" $value }}s |
| Monitoring-Prometheus_Remote_Write_Desired_Shards | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" remote write desired shards calculation wants to run {{ $value }} shards for queue "{{ $labels.remote_name}}:{{ $labels.url }}", which is more than the max of "{{ printf prometheus_remote_storage_shards_max{instance="%s"} $labels.instance |
| Monitoring-Prometheus_Rule_Failures | 15m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to evaluate rules in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Missing_Rule_Evaluations | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" missed rule group evaluations in the last 5m in rule group "{{ $labels.rule_group }}". Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Target_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because the number of targets exceeds the configured target_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Label_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" dropped targets because some samples exceeded the configured label_limit, label_name_length_limit or label_value_length_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Scrape_Body_Size_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured body_size_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Scrape_Sample_Limit_Hit | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed scrapes in the last 5m because some targets exceeded the configured sample_limit. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_Target_Sync_Failure | 5m | critical | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" targets failed to sync because invalid configuration was supplied. Value: {{ printf "%.0f" $value }} |
| Monitoring-Prometheus_High_Query_Load | 15m | warning | monitoring | Prometheus "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" query API has less than 20% available capacity in its query engine for the last 15 minutes. |
| Monitoring-PrometheusOperator_List_Errors | 15m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "List" operations. |
| Monitoring-PrometheusOperator_Watch_Errors | 15m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has errors while performing "Watch" operations. |
| Monitoring-PrometheusOperator_Sync_Failed | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed sync operations for {{ $value }} objects. |
| Monitoring-PrometheusOperator_Reconcile_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed reconciling operations. Value: {{ $value |
| Monitoring-PrometheusOperator_Status_Update_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" has failed status update operations. Value: {{ $value |
| Monitoring-PrometheusOperator_Node_Lookup_Errors | 10m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" errors while reconciling Prometheus. |
| Monitoring-PrometheusOperator_Not_Ready | 5m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" is not ready to reconcile resources. |
| Monitoring-PrometheusOperator_Rejected_Resources | 5m | warning | monitoring | Controller "{{ $labels.controller }}" in cluster "{{ $labels.cluster }}" rejected "{{ $labels.resource }}" resources. Value: {{ printf "%0.0f" $value }} |
| Monitoring-Alertmanager_Failed_Reload | 10m | critical | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has failed to load its configuration. |
| Monitoring-Alertmanager_Members_Inconsistent | 15m | critical | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" has only found {{ $value }} members of the Alertmanager cluster. |
| Monitoring-Alertmanager_Failed_to_Send_Alerts | 5m | warning | monitoring | Alertmanager "{{ $labels.pod }}" in cluster "{{ $labels.cluster }}" failed to send {{ $value |
| Monitoring-Alertmanager_Cluster_Failed_to_Send_Alerts | 5m | critical | monitoring | Alertmanager in cluster "{{ $labels.cluster }}" has high notification failure rate to "{{ $labels.integration }}". Value: {{ $value |
| Monitoring-Alertmanager_Config_Inconsistent | 20m | critical | monitoring | Alertmanager instances in cluster "{{ $labels.cluster }}" have different configurations. |
rules/node_exporter.yml
| Alert Name | For | Severity | Type | Description |
|---|---|---|---|---|
| Node-Recently_Rebooted | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" has been rebooted in the last 30 minutes. Value: {{ humanizeDuration $value }} uptime. |
| Node-CPU_High_Usage | 30m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - CPU usage has exceeded the threshold of 90% for more than 30 minutes. Value: {{ humanize $value }} |
| Node-CPU_High_steal | 15m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - CPU %steal has exceeded the threshold of 10% for more than 15 minutes. Value: {{ humanize $value }} |
| Node-CPU_High_steal | 15m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - CPU %steal has exceeded the threshold of 30% for more than 15 minutes. Value: {{ humanize $value }} |
| Node-CPU_High_iowait | 15m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - CPU %iowait has exceeded the threshold of 10% for more than 15 minutes. Value: {{ humanize $value }} |
| Node-CPU_High_iowait | 15m | error | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - CPU %iowait has exceeded the threshold of 30% for more than 15 minutes. Value: {{ humanize $value }} |
| Node-Memory_Major_Pages_Faults | 15m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - memory major page faults are occurring at very high rate. Value: {{ humanize $value }} |
| Node-Memory_High_Usage | 10m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - more than 80% memory used for more than 10 minutes. Value: {{ humanize $value }} |
| Node-Memory_High_Usage | 10m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - more than 90% memory used for more than 10 minutes. Value: {{ humanize $value }} |
| Node-Disk_Space_High_Usage | 15m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" has more than 80% used disk space for more than 15m. Value: {{ humanize $value }} |
| Node-Disk_Space_High_Usage | 15m | error | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" has more than 90% used disk space for more than 15m. Value: {{ humanize $value }} |
| Node-Disk_Will_Fill_Up_In_4h | 5m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" will fill up in 4 hours at the current rate of utilization. Value: {{ printf `node_filesystem_avail_bytes{fstype=~"ext.* |
| Node-High_Disk_Inodes_High_Usage | 15m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" has more than 60% used inodes for more than 15m. Value: {{ humanize $value }} |
| Node-High_Disk_Inodes_High_Usage | 15m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - mountpoint "{{ $labels.mountpoint }}" has more than 70% used inodes for more than 15m. Value: {{ humanize $value }} |
| Node-Load_High | 15m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - system load per core is above 2 for the last 15 minutes. This might indicate this instance resources saturation and can cause it becoming unresponsive. Value: {{ humanize $value }} |
| Node-fds_Near_Limit_Process | 5m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - "{{ $labels.job }}" has more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }} |
| Node-fds_Near_Limit | 5m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - more than 80% used file descriptors for more than 5m. Value: {{ humanize $value }} |
| Node-Network_High_Receive_Drop | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has high drop in network reception. Value: {{ humanize $value }} |
| Node-Network_High_Transmit_Drop | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has high drop in network transmission. Value: {{ humanize $value }} |
| Node-Network_High_Receive_Errors | 30m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered receive errors. Value: {{ humanize $value }} |
| Node-Network_High_Transmit_Errors | 30m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" has encountered transmit errors. Value: {{ humanize $value }} |
| Node-Network_Interface_Flapping | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - interface "{{ $labels.device }}" changing its up status often. Value: {{ humanize $value }} |
| Node-Network_Bond_Interface_Misconfigured | 5m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" is misconfigured. Check bonding slaves configuration. |
| Node-Network_Bond_Interface_Down | 5m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - bond "{{ $labels.master }}" has interface(s) down. Value: {{ $value }} |
| Node-Too_Many_OOM_Kills | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - several OOM kills detected in the past 1h. Value: {{ humanize $value }}. Find out which process by running `dmesg |
| Node-Clock_Not_Synchronising | 10m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - clock is not synchronising. Ensure NTP is configured on this host. |
| Node-Clock_Skew_Detected | 10m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - clock is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host. |
| Node-Host_Conntrack_Limit | 10m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - more than 75% of conntrack entries are used. |
| Node-EDAC_Correctable_Errors_Detected | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - EDAC Correctable Errors detected. |
| Node-EDAC_Uncorrectable_Errors_Detected | 0m | error | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - EDAC Uncorrectable Errors detected. |
| Node-Filesystem_Device_Error | 0m | warning | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" filesystem error. |
| Node-Disk_Queue_Length_High | 10m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has disk queue length greater than 1 for more than 10 minutes. Value: {{ humanize $value }} |
| Node-Disk_IO_Time_Weighted_Seconds | 10m | info | system | Node "{{ $labels.instance }}" in cluster "{{ $labels.cluster }}" - device "{{ $labels.device }}" has high disk io queue (aqu-sq) for more than 10 minutes. Value: {{ humanize $value }} |
sylva-projects/sylva-elements/container-images/keytool-image (registry.gitlab.com/sylva-projects/sylva-elements/container-images/keytool-image)
v0.2.2: keytool-image: 0.2.2
Merge Requests integrated in this release
49 merge requests were integrated in this repo between 0.2.1 and 0.2.2. These notes don't account for the MRs merged in secondary repos.
Other dependency upgrades
- Update dependency mikefarah/yq to v4.45.1 !116
- Update alpine container to v3.23.4 !115 !128 !137 !149 !159 !167 !155
CI
- Update dependency sylva-projects/sylva-elements/ci-tooling/ci-templates to v1.1.0 renovate !113 !119 !120 !122 !127 !130 !131 !136 !143 !146 !147 !148 !150 !151 !152 !153 !156 !157 !162 !168
- Update dependency renovate-bot/renovate-runner to v24 renovate !117 !118 !121 !124 !125 !126 !129 !134 !135 !139 !141
- Update dependency to-be-continuous/gitleaks to v2.10.0 renovate !123 !132 !138 !140 !145 !154 !158 !163 !164
- Fix LegacyKeyValueFormat !166 (issues: #6)
Contributors
1 person contributed.
sylva-projects/sylva-elements/container-images/kube-job (registry.gitlab.com/sylva-projects/sylva-elements/container-images/kube-job)
v1.1.1: kube-job: v1.1.1
Merge Requests integrated in this release
CI
-
Update dependency renovate-bot/renovate-runner to v19.64.0
-
Update dependency renovate-bot/renovate-runner to v19.41.2 !148 renovate
-
Update dependency renovate-bot/renovate-runner to v19.49.2 !153 renovate
-
Update dependency renovate-bot/renovate-runner to v19.56.1 !155 renovate
-
Update dependency renovate-bot/renovate-runner to v19.60.0 !156 renovate
-
Update dependency renovate-bot/renovate-runner to v19.64.0 !157 renovate
-
Other dependency upgrades
- Update dependency kubernetes-sigs/cluster-api to v1.9.3 !160
Contributors
Configuration
- If you want to rebase/retry this MR, check this box
This MR has been generated by Renovate Bot Sylva instance.
CI configuration couldn't be handle by MR description. A dedicated comment has been posted to control it.
If no checkbox is checked, a default pipeline will be enabled (capm3, or capo if capo label is set)
Edited by Sylva Renovate bot