secret_detection fails in merge request with SSL certificate problem but passes after merge

Summary

secret_detection fails in merge request but passes after merge.

The error is SSL certificate problem: unable to get local issuer certificate. But no other CI jobs have problems with the SSL certificate (an internal CA signs that), including secret_detection jobs running in the master branch.

Steps to reproduce

  1. Create a merge request
  2. secret_detection job failed
  3. Merge
  4. secret_detection job passed

Notes:

  • gitlab-ci.yml doesn't have different workflows
  • There is no customization on secret detection
  • Tried in another repo, and I have the same issue. The pipeline and code base are different, but it is in the same group.

Example Project

The error is with certificates. I assume it makes no sense to replicate in gitlab.com. If you disagree let me know.

What is the current bug behavior?

secret_detection fails in merge request but passes after merge

Additionally, artifacts are not available. gl-sast-report.json is available.

What is the expected correct behavior?

secret_detection should pass in both situations, and artifacts should be available

Relevant logs and/or screenshots

.gitlab-ci.yml (partial file, removing the actual test stage)

stages:
- test
sast:
  stage: test
  artifacts:
    paths:
    - gl-sast-report.json
    - gl-secret-detection-report.json

include:
- template: Security/SAST.gitlab-ci.yml
- template: Security/Secret-Detection.gitlab-ci.yml

CI/CD job in merge request

Running with gitlab-runner 11.5.0 (3afdaba6)
  on RUN01 12ccbb74
Using Docker executor with image registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Pulling docker image registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Using docker image sha256:a4df80764a4ba7522ce40d764b334a1e2c2334340bbd3d57ab871268a6bd5d43 for registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Running on runner-12ccbb74-project-102-concurrent-0 via <sanitized>...
00:01
Fetching changes...
00:02
Removing gl-sast-report.json
Removing semgrep.sarif
HEAD is now at 616d9a1 add security artifacts to ci
Checking out 616d9a19 as 1.2.0...
Skipping Git submodules setup
$ if [ -n "$CI_COMMIT_TAG" ]; then echo "Skipping Secret Detection for tags. No code changes have occurred."; exit 0; fi
00:01
$ if [ "$CI_COMMIT_BRANCH" = "$CI_DEFAULT_BRANCH" ]; then echo "Running Secret Detection on default branch."; /analyzer run; exit 0; fi
$ git fetch origin $CI_DEFAULT_BRANCH $CI_COMMIT_REF_NAME
fatal: unable to access 'https://gitlab.<sanitized>.git/': SSL certificate problem: unable to get local issuer certificate
Uploading artifacts...
00:01
WARNING: gl-secret-detection-report.json: no matching files 
ERROR: No files to upload                          
ERROR: Job failed: exit code 128

CI/CD job after merge to main (same code base)

Running with gitlab-runner 11.5.0 (3afdaba6)
  on RUN01 12ccbb74
Using Docker executor with image registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Pulling docker image registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Using docker image sha256:a4df80764a4ba7522ce40d764b334a1e2c2334340bbd3d57ab871268a6bd5d43 for registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3 ...
Running on runner-12ccbb74-project-102-concurrent-0 via <sanitized>...
00:02
Fetching changes...
00:01
Removing gl-sast-report.json
Removing semgrep.sarif
HEAD is now at 7336483 Merge branch '1.2.0' into 'master'
Checking out 7336483f as master...
Skipping Git submodules setup
$ if [ -n "$CI_COMMIT_TAG" ]; then echo "Skipping Secret Detection for tags. No code changes have occurred."; exit 0; fi
00:05
$ if [ "$CI_COMMIT_BRANCH" = "$CI_DEFAULT_BRANCH" ]; then echo "Running Secret Detection on default branch."; /analyzer run; exit 0; fi
Running Secret Detection on default branch.
[INFO] [secrets] [2021-12-27T15:23:16Z] ▶ GitLab secrets analyzer v3.23.1
[INFO] [secrets] [2021-12-27T15:23:16Z] ▶ Detecting project
[INFO] [secrets] [2021-12-27T15:23:16Z] ▶ Found project in /builds/<sanitized>
[INFO] [secrets] [2021-12-27T15:23:16Z] ▶ Running analyzer
[INFO] [secrets] [2021-12-27T15:23:18Z] ▶ Creating report
Uploading artifacts...
00:01
gl-secret-detection-report.json: found 1 matching files 
Trying to load /builds/<sanitized>.tmp/CI_SERVER_TLS_CA_FILE ... 
Dialing: tcp gitlab.<sanitized>:443 ...            
Uploading artifacts to coordinator... ok            id=3143 responseStatus=201 Created token=DLoyHY-C
Job succeeded

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 18.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.16
Git Version:    2.33.1.
Sidekiq Version:6.2.2
Go Version:     unknown

GitLab information
Version:        14.5.2-ee
Revision:       4511944420f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://gitlab.
HTTP Clone URL: https://gitlab./some-group/some-project.git
SSH Clone URL:  git@:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: gitlab

GitLab Shell
Version:        13.22.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
34/1 ... yes
13/5 ... yes
14/6 ... yes
6/8 ... yes
15/9 ... yes
18/10 ... yes
6/11 ... yes
6/12 ... yes
11/13 ... yes
14/14 ... yes
14/15 ... yes
11/16 ... yes
11/17 ... yes
6/18 ... yes
15/21 ... yes
26/22 ... yes
26/23 ... yes
60/24 ... yes
9/26 ... yes
9/27 ... yes
9/28 ... yes
9/29 ... yes
9/30 ... yes
9/31 ... yes
14/32 ... yes
14/33 ... yes
7/34 ... yes
34/35 ... yes
34/37 ... yes
34/38 ... yes
34/39 ... yes
7/40 ... yes
18/43 ... yes
51/44 ... yes
33/45 ... yes
7/46 ... yes
36/47 ... yes
6/48 ... yes
6/49 ... yes
60/51 ... yes
60/52 ... yes
60/53 ... yes
60/54 ... yes
60/55 ... yes
61/56 ... yes
11/57 ... yes
11/58 ... yes
65/60 ... yes
14/62 ... yes
61/63 ... yes
69/70 ... yes
89/71 ... yes
69/73 ... yes
69/74 ... yes
65/75 ... yes
18/76 ... yes
14/77 ... yes
75/78 ... yes
75/84 ... yes
34/85 ... yes
80/86 ... yes
80/87 ... yes
80/88 ... yes
80/89 ... yes
64/90 ... yes
14/91 ... yes
61/92 ... yes
6/94 ... yes
61/95 ... yes
61/96 ... yes
89/97 ... yes
61/99 ... yes
95/100 ... yes
9/101 ... yes
9/102 ... yes
9/103 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 63
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes