Detect & Alert on manual changes to Kubernetes configuration
We should build a mechanism to flag and optionally alert when there have been manual changes performed on a cluster. Weaveworks has a creative solution for that noted in this blog post, and built a tool to help monitor called
This does seem to require a SSOT configuration hosted outside the cluster, which may not be typical for most enterprises (especially if they are using tools like Helm, etc.)
One interesting option is that we could take a snapshot of the k8s configuration after each deployment, and then use this as the reference configuration to compare future changes that may occur in between a deploy.