Reorganize TestCredSetup

Guillaume Chauvelrequested to merge
gitlab-community/gitlab-org/gitlab-runner:gchauvel/reorg-testcredsetup into main

What does this MR do?

This MR is somewhat of a follow-up to !5430 (merged).

It updates TestCredSetup to:

  • prepare for !6609
  • might fix flakiness

To do so:

  • getGitCred now takes a username parameter, queries the git credential helper chain via git credential fill and prints marker username=<u> password=<p> on a single line so both values can be asserted together
  • setGitCred now takes a username parameter and can set up fake credentials for any username
  • GITLAB_TEST_USERNAME is removed. It looks like !5982 (merged) assumed it was used for git clone/fetch operations. The purpose of setGitCred/getGitCred is just to insert/check fake git credentials to assert what happens to stored credentials. What is required to run locally is to edit repoURLWithSubmodules and repoShaWithSubmodules if your token doesn't give you access to the private repository
  • extractGitConfig is removed, assertions are performed with explicit markers injected directly into the shell scripts (#pre_get_source#, #helper#, #post_get_source#, #build#)
  • The global setInvalidGitCreds helper is removed. Fake credentials are now injected per build through setGitCred in the pre_get_sources hook, so each build starts from a known credential state that matches it
  • 1 credential helper is removed from setupCachingCredHelpers, as test setup is performed at each pre_get_source phase, it is no longer required
  • Each build gets its own unique personal-token username (my-personal-token-<n>) in addition to gitlab-ci-token, so assertions show that per-build pre-existing credentials are not impacted by the build
  • When gitUrlsWithoutTokens is false, the credential helper list is not (re)set to empty at local level, so the global credential helpers are called, which the tests are set up to do. In order to properly manage/assert "git credential reject" on the "gitlab-ci-token" username (which is the purpose of !6609), those tests have to be run sequentially: t.Parallel() is only called on the gitUrlsWithoutTokens: true variant
  • A small onlySchemeAndHost helper is extracted so the test body and onlyHost share the same URL parsing

Each test now performs the following:

step what description
pre_get_source setGitCred setup credentials in git credential helpers for the test
pre_get_source getGitCred check credentials in git credential helpers just to make sure set up was ok
post_get_source listGitConfig check git config used during source retrieval
post_get_source getGitCred check credentials from git credential helpers after sources were retrieved
build listGitConfig check git config
after build read credentials stored in $CI_BUILDS_DIR/git-credentials check git credential helpers data after build (job is finished)

Why was this MR needed?

Prerequisite for !6608 (merged) and !6609

About git authentication and credential helpers

If you are not familiar with git credential helpers, take a look at https://git-scm.com/docs/gitcredentials, particularly: "If credential.helper is configured to the empty string, this resets the helper list to empty (so you may override a helper set by a lower-priority config file by configuring the empty-string helper, followed by whatever set of helpers you would like)." And of course https://git-scm.com/docs/git-config: "git config will read configuration options from multiple files":

  • (prefix)/etc/gitconfig
  • ~/.gitconfig
  • $GIT_DIR/config
  • $GIT_DIR/config.worktree


credential helpers precedence during each test :

system
  credential.helper = **********

global
  credential.helper = (EMPTY: clears system level)
  credential.helper = per-build $CI_BUILDS_DIR/git-credentials file

local (only when gitUrlsWithoutTokens = true)
  credential.helper = (EMPTY: clears global level)
  credential.helper = shell script which returns CI_JOB_TOKEN env var for any username


gitUrlsWithoutTokens TRUE resets credential helpers by setting an empty one first at local level, so it deactivates the global one (.gitconfig in user HOME directory) and the system one (gitconfig in git installation directory):

  • no previously stored global or system credentials can be accessed during : get_sources, post_get_sources, build
  • CI_JOB_TOKEN will not be dumped to every possible credential helper (local, global, system)
  • a password query for any username returns CI_JOB_TOKEN

gitUrlsWithoutTokens FALSE does not, so:

  • previously stored global and system credentials can be accessed during any step: pre_get_sources, get_sources, post_get_sources, build
  • CI_JOB_TOKEN will be dumped to every possible reacheable credential helper (local, global, system)

What's the best way to test this MR?

$env:GITLAB_TEST_TOKEN = 'an-access-token'
# update shell_integration_test.go repoURLWithSubmodules and repoShaWithSubmodules with data from a private project you have access to
go test -v -tags integration -run TestCredSetup ./executors/shell/...

Use either a Personal or Group access token with one of read_repository, read_api, api permission.

What are the relevant issue numbers?

#39338 (closed) (!6575 (closed))
#39108

!5982 (merged)

TODO

  • update MR description to add more info about #39108
Edited by Guillaume Chauvel

Merge request reports