Generate Kustomize variants prior to SAST scans
Release notes
Prior to running kics on a GitOps repository:
- detect the presence of Kustomize roots and determine which ones are "top-level" (not referenced by other Kustomize roots)
- run
kustomize build
on all top-level kustomize roots, caching the resulting variants for later scanning - exclude all detected Kustomize roots from kics scanning so it only looks at their resulting variants
Problem to solve
Running SAST-IaC scanning on a GitOps repository where Kustomize overlays are used frequently results in duplicate and phantom vulnerability reports.
Intended users
User experience goal
IaC scanning should not report vulnerabilities that will not actually exist in the cluster once overlays are applied. It would also be nice to group vulnerabilities in overlays that result from the same issue in a shared base.
Proposal
Prior to running kics on a GitOps repository:
- detect the presence of Kustomize roots and determine which ones are "top-level" (not referenced by other Kustomize roots). Kustomize roots can be easily identified by the presence of a "kustomization.yaml", "kustomization.yml", or "Kustomization" file. Then by reading the
resources
section of each Kustomization file and checking for references to other directories we can determine which are top-level and which are not. - run
kustomize build
on all top-level Kustomize roots, caching the resulting variants for later scanning - exclude all detected Kustomize roots (not just top-level) from kics scanning so it only looks at their resulting variants (and any non-Kustomize directories)
Further details
We use Kustomize overlays to keep are dev and prod environments as similar as possible, to ensure our testing is valid. Currently the scanning will report an error on a base file, such as "Memory Limits Not Defined" even if all overlays set memory limits, meaning there is no actual issue in the cluster. And when the overlays do not set a value, there is no indication in the vulnerability reporting that they're all stemming from the same issue in a base file.
My current workaround is a Go script that performs the scanning outlined above and writes lists of directories to files - one for all Kustomize roots, and one containing only the top-level targets. I then have a job that runs kustomize build
on all the top-level targets and stores the resulting variants as job artifacts, and also populates SAST_EXCLUDED_PATHS
with all Kustomize roots via a dotenv
report. I then configure the official kics-iac-sast
job to depend on my custom job. In the first two repositories I've used this setup on, we went from 125 to 96 vulnerabilities and from 696 to 320, making it much easier to focus on fixing real vulnerabilities.
Permissions and Security
Do not believe any permissions changes are required