Profiling and linux tracing on kubernetes
One of the capabilities we are currently sacrificing when moving workloads to kubernetes is profiling via perf and low-level tracing via bcc and bpftrace.
This is something we should figure out how to do in kubernetes (and GKE in particular).
- For bpftrace, we should evaluate kubectl-trace.
- For BCC, we can take a look at Linux BPF CPU Profiling with kubectl on Kubernetes
- For perf, we can take a look at Enabling
perf
in Kubernetes with Docker’s default seccomp profile
One overall question here is whether we need to allow privileged containers in GKE, if these approaches even work in GKE, and what the security implications of that are.
See also: Using PodSecurityPolicies.
Possible collaboration between @gitlab-org/delivery and @gitlab-com/gl-infra/sre-observability.