Parse registry credentials from the repository config file

Stack, in merge order:

  1. updater !208
  2. orchestrator !67
  3. orchestrator !68
  4. orchestrator !62
  5. gitlab !256224 (merged) (this MR)
  6. gitlab !264336 (merged)
  7. gitlab !264337
  8. docs !256327
Merge notes
  • Different repositories can merge in parallel. Within a repository, keep the order above.
  • Tag an updater release after !208 and an orchestrator release after !62.
  • Keep dependency_management_registry_credentials off. Nothing is active in production until it is on and the proxy is enabled.
  • Merge the docs last, after !264336 (merged), the orchestrator release and enabling the proxy.

First of three stacked merge requests for #624431 and #624433, part of the configuration epic. Nothing calls the new class yet: !264336 (merged) wires it into the update service behind a feature flag, and !264337 adds the rubygems and go kinds. Docs are in !256327.

Auto-remediation cannot authenticate to a private package registry. Two gaps cause it: only_workload_variables? drops every instance, group and project CI/CD variable from the update job, and the job configuration sent to the orchestrator has no credentials field. This merge request adds the parser that both fixes need, and !264336 (merged) closes the gaps.

The file

An optional .gitlab/dependency_management.yml on the default branch:

registries:
  gitlab-npm:
    kind: npm
    url: "https://gitlab.example.com/api/v4/projects/1/packages/npm/"
    password: "${DS_AUTO_REMEDIATION_REGISTRY_NPM_TOKEN}"
    replaces-base: true

RepositoryConfig#credentials returns dependabot credential hashes with the placeholders left in. RepositoryConfig#variables returns the CI/CD variable names those placeholders reference, which are the ones the job needs exposed.

Rules the parser enforces

registries is the only top-level key, with up to 20 entries, in a file of at most 16 KB. The blob is fetched with that limit, so an oversized file is rejected without being loaded.

Key Required Rule
kind Yes npm, maven or python
url Yes https:// only, no userinfo and no ${VARIABLE}, 2048 characters
username For maven Literal, or a ${VARIABLE} placeholder
password Yes A ${VARIABLE} placeholder. A literal is rejected
replaces-base No true or false

A placeholder must name an upper-case variable starting with DS_AUTO_REMEDIATION_, at most 255 characters to match the p_ci_workload_variable_inclusions check constraint.

Mapping to dependabot: npm becomes npm_registry with registry set to the URL minus its scheme, maven becomes maven_repository with the username/password pair, python becomes python_index with index-url. npm and python take dependabot's single token, built as user:pass when a username is set, since dependabot splits it on the colon.

Design points

  • Secrets never resolve in Rails. The placeholder is emitted verbatim, so real values stay out of Rails logs, the pipeline YAML and the workload records. The orchestrator substitutes them inside the job. Rejecting a literal password also stops a token being committed to the repository.
  • One prefix for everything users provide. The file is Developer-editable while CI/CD variables are Maintainer-provisioned, so the prefix is what stops the file pointing at an unrelated secret such as a deploy token. DS_AUTO_REMEDIATION_ is also disjoint from the DEPENDENCY_MANAGEMENT_ variables GitLab sets on the job. A project variable takes precedence over a job variable, so a shared prefix would let a user variable override them, and the orchestrator used to copy that prefix into the job artifact (fixed in gitlab-org/security-products/dependency-management/orchestrator!67). Later settings read from CI/CD variables use the same prefix.
  • No separate allowlist. An earlier draft had an allowEnv list. The variables to expose are now the ones the placeholders reference, so a variable cannot be listed but unused, or used but unlisted.
  • One file for auto-remediation's repository settings, like dependabot.yml and renovate.json. registries is the only key today. Unknown keys are rejected, so adding one later is safe.
  • Parsed here, not in Go. The variables to expose must be known before the pipeline is created, so the monolith reads the file anyway. Parsing it again in the orchestrator would duplicate the schema.
  • One blob read per project. The scheduler builds an UpdateService per remediation, so the parsed file is cached in the request store.
  • Protected variables stay unsupported. They are dropped on an unprotected ref, and protecting dependency-management/* would break the Guest service account and expose every protected variable to install scripts. Agreed with Product on #624433.
  • NuGet, Cargo and Pub are absent. Cargo keys its credential off a registry name rather than a URL (#631273), and neither the NuGet nor the Pub dependabot gem has Ruby-side credential handling (#631285).
Edited by Albina Yusupova

Merge request reports

Loading
Loading