Currently, we are emitting certain metrics from cgroups for monitoring. In an effort to increase our understanding of the fault lines in the system in production, and so we can improve automation in preventing certain pathological situations from cascading, let's increase the number of metrics we emit to prometheus. Here is a full list of what's available through the [containerd library](https://github.com/containerd/cgroups/) [source](https://github.com/containerd/cgroups/blob/main/cgroup2/stats/metrics.proto) ## Top-Level Metrics Structure The `stats.Metrics` struct contains these categories: | Field | Type | Description | |-------|------|-------------| | Pids | `*PidsStat` | Process/task limits and counts | | CPU | `*CPUStat` | CPU usage and throttling | | Memory | `*MemoryStat` | Memory usage statistics | | MemoryEvents | `*MemoryEvents` | Memory pressure events | | Io | `*IOStat` | Block I/O statistics | | Rdma | `*RdmaStat` | RDMA device statistics | | Hugetlb | `[]*HugeTlbStat` | Huge page statistics (per page size) | | Network | `[]*NetworkStat` | Network interface statistics (v3.1.0+) | --- ### PidsStat | Field | Type | Description | |-------|------|-------------| | Current | `uint64` | Current number of processes | | Limit | `uint64` | Maximum allowed processes | --- ### CPUStat | Field | Type | Description | |-------|------|-------------| | UsageUsec | `uint64` | Total CPU usage in microseconds | | UserUsec | `uint64` | User space CPU time in microseconds | | SystemUsec | `uint64` | Kernel space CPU time in microseconds | | NrPeriods | `uint64` | Number of enforcement intervals | | NrThrottled | `uint64` | Number of times throttled | | ThrottledUsec | `uint64` | Total throttle time in microseconds | | NrBursts | `uint64` | Number of burst periods (v3.1.0+) | | BurstUsec | `uint64` | Burst time in microseconds (v3.1.0+) | | PSI | `*PSIStats` | Pressure stall information | --- ### MemoryStat | Field | Type | Description | |-------|------|-------------| | Anon | `uint64` | Anonymous memory | | File | `uint64` | File-backed memory (page cache) | | KernelStack | `uint64` | Kernel stack memory | | Slab | `uint64` | Slab allocator memory | | Sock | `uint64` | Socket buffer memory | | Shmem | `uint64` | Shared memory | | FileMapped | `uint64` | Memory-mapped files | | FileDirty | `uint64` | Dirty file pages | | FileWriteback | `uint64` | Pages being written back | | AnonThp | `uint64` | Anonymous transparent huge pages | | InactiveAnon | `uint64` | Inactive anonymous pages | | ActiveAnon | `uint64` | Active anonymous pages | | InactiveFile | `uint64` | Inactive file pages | | ActiveFile | `uint64` | Active file pages | | Unevictable | `uint64` | Unevictable pages | | SlabReclaimable | `uint64` | Reclaimable slab memory | | SlabUnreclaimable | `uint64` | Unreclaimable slab memory | | Pgfault | `uint64` | Page faults | | Pgmajfault | `uint64` | Major page faults | | WorkingsetRefault | `uint64` | Working set refaults | | WorkingsetActivate | `uint64` | Working set activations | | WorkingsetNodereclaim | `uint64` | Working set node reclaims | | Pgrefill | `uint64` | Pages refilled | | Pgscan | `uint64` | Pages scanned | | Pgsteal | `uint64` | Pages stolen | | Pgactivate | `uint64` | Pages activated | | Pgdeactivate | `uint64` | Pages deactivated | | Pglazyfree | `uint64` | Pages marked lazy free | | Pglazyfreed | `uint64` | Pages actually lazy freed | | ThpFaultAlloc | `uint64` | THP fault allocations | | ThpCollapseAlloc | `uint64` | THP collapse allocations | | Usage | `uint64` | Current memory usage | | UsageLimit | `uint64` | Memory limit | | SwapUsage | `uint64` | Current swap usage | | SwapLimit | `uint64` | Swap limit | | MaxUsage | `uint64` | Peak memory usage (v3.0.3+) | | SwapMaxUsage | `uint64` | Peak swap usage (v3.0.3+) | | PSI | `*PSIStats` | Pressure stall information (v3.0.3+) | --- ### MemoryEvents | Field | Type | Description | |-------|------|-------------| | Low | `uint64` | Times memory dropped below low threshold | | High | `uint64` | Times memory exceeded high threshold | | Max | `uint64` | Times memory hit max limit | | Oom | `uint64` | OOM events | | OomKill | `uint64` | OOM kills | --- ### IOStat | Field | Type | Description | |-------|------|-------------| | Usage | `[]*IOEntry` | Per-device I/O statistics | | PSI | `*PSIStats` | Pressure stall information (v3.0.3+) | ### IOEntry | Field | Type | Description | |-------|------|-------------| | Major | `uint64` | Device major number | | Minor | `uint64` | Device minor number | | Rbytes | `uint64` | Bytes read | | Wbytes | `uint64` | Bytes written | | Rios | `uint64` | Read operations | | Wios | `uint64` | Write operations | --- ### RdmaStat | Field | Type | Description | |-------|------|-------------| | Current | `[]*RdmaEntry` | Current RDMA usage per device | | Limit | `[]*RdmaEntry` | RDMA limits per device | ### RdmaEntry | Field | Type | Description | |-------|------|-------------| | Device | `string` | Device name | | HcaHandles | `uint32` | HCA handles in use | | HcaObjects | `uint32` | HCA objects in use | --- ### HugeTlbStat | Field | Type | Description | |-------|------|-------------| | Current | `uint64` | Current huge page usage | | Max | `uint64` | Maximum huge page usage | | Pagesize | `string` | Page size (e.g., "2MB", "1GB") | | Failcnt | `uint64` | Allocation failures (v3.1.0+) | --- ### NetworkStat (v3.1.0+) | Field | Type | Description | |-------|------|-------------| | Name | `string` | Interface name | | RxBytes | `uint64` | Bytes received | | RxPackets | `uint64` | Packets received | | RxErrors | `uint64` | Receive errors | | RxDropped | `uint64` | Receive drops | | TxBytes | `uint64` | Bytes transmitted | | TxPackets | `uint64` | Packets transmitted | | TxErrors | `uint64` | Transmit errors | | TxDropped | `uint64` | Transmit drops | --- ### PSIStats Available for CPU, Memory, and IO. | Field | Type | Description | |-------|------|-------------| | Some | `*PSIData` | Some tasks stalled | | Full | `*PSIData` | All tasks stalled | ### PSIData | Field | Type | Description | |-------|------|-------------| | Avg10 | `float64` | 10-second average (percentage) | | Avg60 | `float64` | 60-second average (percentage) | | Avg300 | `float64` | 300-second average (percentage) | | Total | `uint64` | Total stall time in microseconds |