feat(duo): enable FF_GIT_URLS_WITHOUT_TOKENS and remove sed-based token exposure

Andras Herczegrequested to merge
duo/fix/593467-dap-git-auth-credential-helper into master

What does this MR do and why?

Fixes DAP (Duo Agent Platform) git authentication by replacing sed-based token injection with a proper Git credential helper.

Problem

DAP workloads authenticated git via oauth_remote_commands, which rewrote the origin remote with git remote set-url using a sed command that embedded the OAuth token in the URL (https[REDACTED]gitlab.com/...). This had two security issues:

  1. The token was written into .git/config on disk.
  2. The sed command line, including the token, was visible in ps output.

Solution

The changes live in ee/app/services/ai/duo_workflows/start_workflow_service.rb and ee/app/services/ai/duo_workflows/resume_workflow_service.rb, and are gated behind a new feature flag. When dap_git_credential_helper is enabled, git_environment_variables does two things together:

  1. Sets FF_GIT_URLS_WITHOUT_TOKENS: 'true', a GitLab Runner feature flag that tells the Runner not to embed CI_JOB_TOKEN in clone URLs or insteadOf rules, leaving the origin URL credential-free.
  2. Adds two GIT_CONFIG_* entries at credential.<gitlab-url>.helper: an empty-value reset, then the DAP OAuth helper. The reset is required - see below.

oauth_remote_commands returns an empty array when the flag is on, so the sed-based rewrite no longer runs. The method itself is kept as the flag-off code path, to be deleted together with the feature flag once it's rolled out.

The helper is built by self.git_credential_helper (a predecessor constant was removed from master in d27b3e1719c0, "Let the runner do the full blobless clone for Duo Developer"). It reads GIT_PASSWORD at invocation time, so the token is never written to disk, and it guards itself:

  • acts only on the get operation, so store/erase are no-ops;
  • parses git's request from stdin — where the host actually arrives, not argv — a whole line at a time, since matching the raw request by glob would answer a host merely containing ours;
  • refuses unless protocol://host equals the instance origin exactly, so a blank origin (which would collapse the config key to the unscoped credential..helper) or an http:// downgrade gets nothing;
  • prints nothing when GIT_PASSWORD is empty. Git ignores a helper's exit status and treats a bare password= as an empty password, so declining to print is what stops it authenticating as oauth: with no secret and failing on every operation.

The config key uses Gitlab.config.gitlab.url (the full instance URL, including any relative-URL path). The helper's stdin check uses self.gitlab_origin (scheme+host only), because git sends the host — not the path — in the credential request. Path-scoped credential keys match without useHttpPath, so using the full URL narrows the scope: on a relative-URL install (https://host/gitlab), the OAuth token is only offered to the GitLab instance path, not to unrelated applications on the same host.

Two follow-ups from review

A resumed workflow follows its start-time path. ResumeWorkflowService#variables carries the previous workload's variables forward wholesale (refreshing only the token keys), but it inherits #commands, so oauth_remote_commands used to re-evaluate the flag fresh at resume time. A flip between start and resume then put the two on opposite paths: with the flag off at start and on at resume, the preserved variables had no FF_GIT_URLS_WITHOUT_TOKENS and no credential helper, yet oauth_remote_commands skipped its sed rewrite — leaving the resumed job with no git auth of its own and falling back to the Runner's CI_JOB_TOKEN. git_credential_helper? is now overridden in ResumeWorkflowService to derive its answer from the preserved variables, so commands and variables always agree. Raised in !243872 (comment 3725438782).

Old Runners no longer degrade silently. FF_GIT_URLS_WITHOUT_TOKENS is the first thing here that depends on the Runner's version: one too old to honour it embeds the job token in the origin URL anyway, so the clone still works but the security objective is quietly missed. git_auth_diagnostic_commands now writes a warning to the job trace when that happens (details in step 4 below). Raised in !243872 (comment 3708675520).

Two decisions worth calling out

The helper is URL-scoped. It is installed as credential.<gitlab-url>.helper, not the bare credential.helper key, so the DAP OAuth token is only offered when git is challenged by the GitLab instance — not by submodules, LFS, or bundle-URI remotes on other hosts. Path-scoped credential keys match without useHttpPath (gitcredentials(7)), so on a relative-URL install the path component further narrows the scope to the GitLab instance path. This mirrors the scoping already used by the neighbouring http.<gitlab-url>.proactiveAuth entry.

The empty-value reset is back, URL-scoped, immediately before the helper. An earlier revision dropped it, on the theory that git <= 2.39 rejects an empty GIT_CONFIG_VALUE_n outright (the failure gitlab-runner!6729 (merged) hit) and that it couldn't have cleared the Runner's helper anyway. Both turned out wrong: git errors on an absent GIT_CONFIG_VALUE_n, never an empty one (verified on git 2.32.7, 2.36.6, 2.39.5, 2.43.7, 2.53.0 - !6729 (merged)'s error came from PowerShell's ${env:X}="" unsetting the variable, not from git rejecting an empty value). More importantly, FF_GIT_URLS_WITHOUT_TOKENS makes the Runner write its own CI_JOB_TOKEN helper at this same credential.<scheme://host> scope; without the reset, that helper is tried first and wins every GitLab challenge, so pushes fail with the job token instead of the OAuth token. The reset is URL-scoped, so it only clears helpers at the GitLab host - other hosts' helpers (submodules, LFS, bundle URIs) are untouched. Reproduced with a representative Runner config in the wins over the Runner job-token helper that FF_GIT_URLS_WITHOUT_TOKENS installs spec example.

Feature flag

New flag dap_git_credential_helper (type gitlab_com_derisk, default_enabled: false, actor: project), defined in ee/config/feature_flags/gitlab_com_derisk/dap_git_credential_helper.yml. With the flag off, behaviour is byte-identical to current master.

Rollout is tracked in #612645, which lists the preconditions that must be confirmed before the flag is enabled anywhere.

Risks

  • With the flag on, the Runner's own get_sources clone authenticates with the service-account OAuth token instead of CI_JOB_TOKEN. If the OAuth token is ever blank or expired, the clone now fails where it previously succeeded.
  • Related to the above: the composite-identity token TTL is 1 hour while the workload job timeout is 2 hours, so a long-running workflow can lose git access mid-job. This does not affect the flag-off path. This is pre-existing and unconditional on the flag — the sed path authenticates the same ongoing git operations with the same token today — so it does not gate this rollout. Only get_sources dispatch-time staleness is a precondition; see the rollout issue.
  • Runners too old to honour FF_GIT_URLS_WITHOUT_TOKENS degrade silently: the URL keeps its embedded token and the clone still works, so nothing surfaces that the security objective wasn't met for that job. Confirm the fleet version as a rollout precondition.
  • The credential helper is only visible to user-authored git commands on Runner versions containing gitlab-runner!6729 (merged), which writes a per-job seed file and exports GIT_CONFIG_GLOBAL. The runners serving DAP jobs need to be on a version containing it; this is tracked as a rollout precondition.

References

Screenshots or screen recordings

N/A — backend-only change, no UI impact.

Before After
Token embedded in git remote set-url via sed, visible in process listings and written to .git/config Token supplied via Git credential helper at invocation time, never written to disk

How to set up and validate locally

  1. Enable the flag for a test project in a Rails console:

    Feature.enable(:dap_git_credential_helper, project)

  2. Start a DAP workflow against that project and inspect the generated job variables — confirm FF_GIT_URLS_WITHOUT_TOKENS is 'true' and the GIT_CONFIG_* variables include credential.<gitlab-url>.helper set to the DAP helper.

    • Expect GIT_CONFIG_COUNT=5 without session tracking (key_0 safe.directory, key_1 url.<gitlab>/.insteadOf, key_2 credential.<gitlab>.helper = '' (the reset), key_3 credential.<gitlab>.helper = the DAP helper, key_4 http.<gitlab>.proactiveAuth), or GIT_CONFIG_COUNT=6 with session tracking (core.hooksPath inserted at key_2, shifting the rest).
    • Expect exactly one empty GIT_CONFIG_VALUE_n - the reset. Every other value is non-empty.

  3. Confirm no git remote set-url command runs anywhere in the job (check the job log / script).

  4. Inside the running job, check git config --get-all credential.<gitlab-url>.helper — it should list two values — an empty line (the reset) then the DAP OAuth helper — and no token-embedding command.

    This is now part of the MR: git_auth_diagnostic_commands reads remote.origin.url after get_sources and the workspace fixup, and echoes a warning to the job trace if the URL still carries embedded credentials — which is what happens on a Runner too old to honour FF_GIT_URLS_WITHOUT_TOKENS. It matches on URL shape (scheme://user@), not on a literal token prefix, so it is not tied to gitlab-ci-token specifically, and it never echoes the URL or the token — only the static warning string, since job traces are readable by low-privilege and anonymous users (see https://gitlab.com/gitlab-org/gitlab/-/issues/602194).

    An earlier revision of this section suggested grep -qi oauth "${CI_PROJECT_DIR}/.git/config". That check was wrong: the old-Runner fallback embeds gitlab-ci-token:$CI_JOB_TOKEN@, never the string oauth, so it would have reported "no credentials" in exactly the case it was meant to catch.

  5. Inspect .git/config in the job workspace and confirm it contains no token or credentials.

  6. Disable the flag and re-run the same workflow — confirm the legacy behaviour returns: git remote set-url runs with the sed-based token injection, FF_GIT_URLS_WITHOUT_TOKENS is absent, and GIT_CONFIG_COUNT is back to 3 (or 4 with session tracking).

  7. Run both spec files: bundle exec rspec ee/spec/services/ai/duo_workflows/start_workflow_service_spec.rb ee/spec/services/ai/duo_workflows/resume_workflow_service_spec.rb, including the when dap_git_credential_helper is disabled context and its nested session-tracking sub-context, which lock in the legacy path.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Duo Developer

Merge request reports