Repository mirroring fails due to 'server certificate verification failed' in OpenShift

Summary

Repository mirroring and project migration don't work in GitLab instance built with Operator in OpenShift.

From initial investigation in gitlab-org/quality/quality-engineering/team-tasks#935 (comment 1289905180), it's not clear whether it happens due to misconfiguration with certificates in OpenShift CI clusters or due to a bug in GitLab Operaor in OpenShift.

Note that this failure doesn't occur in GKE clusters.

Steps to reproduce

  1. Deploy GitLab using Operator in OpenShift or trigger a build using CI
  2. Connect to the instance and configure a project to mirror another project

Configuration used

GitLab configuration in CI for OpenShift - happens in both 4.8 and 4.9 setups.

Pipeline: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/pipelines/785638457

Current behavior

13:fetch remote: "fatal: unable to access 'https://gitlab-93bfc72c-qe-run-full-suite-jobs-manually.apps.ocp-ci-4917.k8s-ft.win/gitlab-qa-sandbox-group-4/qa-test-2023-02-22-16-20-58-aa5a9f7381b5b446/pull-mirror-source-project-323b678a4fa37183.git/': 
server certificate verification failed. CAfile: none CRLfile: none\n": exit status 128.

Screenshot_2023-02-23_at_13.29.08

Expected behavior

Mirroring works as expected

Versions

Relevant logs

Further details on this in gitlab-org/quality/quality-engineering/team-tasks#935 (comment 1289905180)

Failure logs:

Logs

Logs from gitlab-sidekiq-all-in-1-v2-578fb9bfb4-fnqg8 sidekiq

{
  "component": "gitlab",
  "subcomponent": "application_json",
  "level": "fatal",
  "severity": "ERROR",
  "time": "2023-02-23T12:34:46.753Z",
  "correlation_id": "01GSZ49P19TW8GE7YP7B45BXAD",
  "message": "Mirror update for gitlab-qa-sandbox-group-4/qa-test-2023-02-22-16-20-58-aa5a9f7381b5b446/pull-mirror-target-project-58ed05d2541ea889 failed with the following message: 13:fetch remote: \"fatal: unable to access 'https://gitlab-93bfc72c-qe-run-full-suite-jobs-manually.apps.ocp-ci-4917.k8s-ft.win/gitlab-qa-sandbox-group-4/qa-test-2023-02-22-16-20-58-aa5a9f7381b5b446/pull-mirror-source-project-323b678a4fa37183.git/': server certificate verification failed. CAfile: none CRLfile: none\\n\": exit status 128..",
  "jid": "58ac0aa981248c1052be1721"
}

Searching for 01GSZ49P19TW8GE7YP7B45BXAD in Gitaly pods:

{
  "component": "gitaly",
  "subcomponent": "gitaly",
  "level": "error",
  "command.count": 1,
  "command.cpu_time_ms": 1,
  "command.inblock": 0,
  "command.majflt": 0,
  "command.maxrss": 359196,
  "command.minflt": 166,
  "command.oublock": 0,
  "command.real_time_ms": 56,
  "command.system_time_ms": 0,
  "command.user_time_ms": 1,
  "correlation_id": "01GSZ49P19TW8GE7YP7B45BXAD",
  "error": "fetch remote: \"fatal: unable to access 'https://gitlab-93bfc72c-qe-run-full-suite-jobs-manually.apps.ocp-ci-4917.k8s-ft.win/gitlab-qa-sandbox-group-4/qa-test-2023-02-22-16-20-58-aa5a9f7381b5b446/pull-mirror-source-project-323b678a4fa37183.git/': server certificate verification failed. CAfile: none CRLfile: none\\n\": exit status 128",
  "grpc.code": "Internal",
  "grpc.meta.auth_version": "v2",
  "grpc.meta.client_name": "gitlab-sidekiq",
  "grpc.meta.deadline_type": "unknown",
  "grpc.meta.method_type": "unary",
  "grpc.method": "FetchRemote",
  "grpc.request.deadline": "2023-02-23T18:34:46.677",
  "grpc.request.fullMethod": "/gitaly.RepositoryService/FetchRemote",
  "grpc.request.glProjectPath": "gitlab-qa-sandbox-group-4/qa-test-2023-02-22-16-20-58-aa5a9f7381b5b446/pull-mirror-target-project-58ed05d2541ea889",
  "grpc.request.glRepository": "project-22",
  "grpc.request.payload_bytes": 548,
  "grpc.request.repoPath": "@hashed/78/5f/785f3ec7eb32f30b90cd0fcf3657d388b5ff4297f2f9716ff66e9b69c05ddd09.git",
  "grpc.request.repoStorage": "default",
  "grpc.response.payload_bytes": 0,
  "grpc.service": "gitaly.RepositoryService",
  "grpc.start_time": "2023-02-23T12:34:46.677",
  "grpc.time_ms": 61.153,
  "msg": "finished unary call with code Internal",
  "peer.address": "10.130.5.208:55404",
  "pid": 1,
  "remote_ip": "213.10.34.7",
  "span.kind": "server",
  "system": "grpc",
  "time": "2023-02-23T12:34:46.738Z"
}