OCS errors in Gitlab Self Managed
Summary
I'm updating the description since the thread discussions are getting quite lengthy. There are 2 main issues that have been identified:
-
403 errors are observed when the gitlab agent attempts to create vulnerabilities in the Gitlab rails app
- This comment contains details on how Gitlab Agent achieves this.
-
Unable to fetch image when containerd is used
- This is still pending more information by customer on whether it is an upstream(Trivy) issue.
Original description by customer
We enabled Operational cluster scanning using agentk. We have deployed using the Helm chart at version 1.16.0
which uses app version v16.1.3
. We have been experiencing operational container scanning job failures resulting from rate limits from docker.io
, and began investigating why.
We have noticed that the job configurations in kuberentes has the Trivy image set to docker.io/aquasec/trivy:0.25.2
. When looking here at the code it looks like Trivy should be set to aquasec/trivy:0.38.3
.
Also of note is that the image tags for agentk show a alert icon and mention issues with missing a digest manifest, see attachment.
Also, it is creating a job for every replicaset, job, statefulset, etc., is that normal?
The example below is a scan job for a replicaset not cleaned up by the helm chart, nor by us, so it shows an old previous version of gitlab. Regardless, the issue at hand is the version of trivy being used...
Example job created by agent pod:
apiVersion: batch/v1
kind: Job
metadata:
annotations:
batch.kubernetes.io/job-tracking: ""
starboard.container-images: '{"gitlab-workhorse":"registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v15.10.7","webservice":"registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v15.10.7"}'
creationTimestamp: "2023-08-28T00:33:32Z"
generation: 1
labels:
app.kubernetes.io/managed-by: starboard
resource-spec-hash: fd6d5f56f
starboard.resource.kind: ReplicaSet
starboard.resource.name: gitlab-webservice-default-7fb97b7cfb
starboard.resource.namespace: <redacted>
vulnerabilityReport.scanner: trivy
name: scan-vulnerabilityreport-5dfdf487f9
namespace: <redacted>
resourceVersion: "1094242842"
uid: 9d02edb6-1ab7-4831-bb3d-f3e51ffb2b45
spec:
activeDeadlineSeconds: 300
backoffLimit: 0
completionMode: NonIndexed
completions: 1
parallelism: 1
selector:
matchLabels:
controller-uid: 9d02edb6-1ab7-4831-bb3d-f3e51ffb2b45
suspend: false
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/managed-by: starboard
controller-uid: 9d02edb6-1ab7-4831-bb3d-f3e51ffb2b45
job-name: scan-vulnerabilityreport-5dfdf487f9
resource-spec-hash: fd6d5f56f
starboard.resource.kind: ReplicaSet
starboard.resource.name: gitlab-webservice-default-7fb97b7cfb
starboard.resource.namespace: <redacted>
vulnerabilityReport.scanner: trivy
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
automountServiceAccountToken: false
containers:
- args:
- --cache-dir
- /tmp/trivy/.cache
- --quiet
- image
- --skip-update
- --format
- json
- registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v15.10.7
command:
- trivy
env:
- name: TRIVY_SEVERITY
valueFrom:
configMapKeyRef:
key: trivy.severity
name: starboard-trivy-config
optional: true
- name: TRIVY_IGNORE_UNFIXED
valueFrom:
configMapKeyRef:
key: trivy.ignoreUnfixed
name: starboard-trivy-config
optional: true
- name: TRIVY_TIMEOUT
valueFrom:
configMapKeyRef:
key: trivy.timeout
name: starboard-trivy-config
optional: true
- name: TRIVY_SKIP_FILES
valueFrom:
configMapKeyRef:
key: trivy.skipFiles
name: starboard-trivy-config
optional: true
- name: TRIVY_SKIP_DIRS
valueFrom:
configMapKeyRef:
key: trivy.skipDirs
name: starboard-trivy-config
optional: true
- name: HTTP_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpProxy
name: starboard-trivy-config
optional: true
- name: HTTPS_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpsProxy
name: starboard-trivy-config
optional: true
- name: NO_PROXY
valueFrom:
configMapKeyRef:
key: trivy.noProxy
name: starboard-trivy-config
optional: true
image: docker.io/aquasec/trivy:0.25.2
imagePullPolicy: IfNotPresent
name: webservice
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 100M
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
privileged: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /tmp
name: tmp
- args:
- --cache-dir
- /tmp/trivy/.cache
- --quiet
- image
- --skip-update
- --format
- json
- registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v15.10.7
command:
- trivy
env:
- name: TRIVY_SEVERITY
valueFrom:
configMapKeyRef:
key: trivy.severity
name: starboard-trivy-config
optional: true
- name: TRIVY_IGNORE_UNFIXED
valueFrom:
configMapKeyRef:
key: trivy.ignoreUnfixed
name: starboard-trivy-config
optional: true
- name: TRIVY_TIMEOUT
valueFrom:
configMapKeyRef:
key: trivy.timeout
name: starboard-trivy-config
optional: true
- name: TRIVY_SKIP_FILES
valueFrom:
configMapKeyRef:
key: trivy.skipFiles
name: starboard-trivy-config
optional: true
- name: TRIVY_SKIP_DIRS
valueFrom:
configMapKeyRef:
key: trivy.skipDirs
name: starboard-trivy-config
optional: true
- name: HTTP_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpProxy
name: starboard-trivy-config
optional: true
- name: HTTPS_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpsProxy
name: starboard-trivy-config
optional: true
- name: NO_PROXY
valueFrom:
configMapKeyRef:
key: trivy.noProxy
name: starboard-trivy-config
optional: true
image: docker.io/aquasec/trivy:0.25.2
imagePullPolicy: IfNotPresent
name: gitlab-workhorse
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 100M
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
privileged: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- args:
- --cache-dir
- /tmp/trivy/.cache
- image
- --download-db-only
- --db-repository
- registry.gitlab.com/gitlab-org/security-products/dependencies/trivy-db-glad
command:
- trivy
env:
- name: HTTP_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpProxy
name: starboard-trivy-config
optional: true
- name: HTTPS_PROXY
valueFrom:
configMapKeyRef:
key: trivy.httpsProxy
name: starboard-trivy-config
optional: true
- name: NO_PROXY
valueFrom:
configMapKeyRef:
key: trivy.noProxy
name: starboard-trivy-config
optional: true
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
key: trivy.githubToken
name: starboard-trivy-config
optional: true
image: docker.io/aquasec/trivy:0.25.2
imagePullPolicy: IfNotPresent
name: 54bec7b3-0d0f-49e5-9c64-a9aec5e74db1
resources:
limits:
cpu: 500m
memory: 500M
requests:
cpu: 100m
memory: 100M
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /tmp
name: tmp
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: <redacted>
serviceAccountName: <redacted>
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: tmp
status:
conditions:
- lastProbeTime: "2023-08-28T00:34:31Z"
lastTransitionTime: "2023-08-28T00:34:31Z"
message: Job has reached the specified backoff limit
reason: BackoffLimitExceeded
status: "True"
type: Failed
failed: 1
ready: 0
startTime: "2023-08-28T00:33:33Z"
uncountedTerminatedPods: {}