Flag, configuration, BatchRequest, OutputParser and JobBuilder for batched updates

Stack

Merge from the bottom up. Each MR targets the one below it.

Why

The gap

Why this approach

  • One reader, BatchUpdates.enabled?, checks the flag and the opt-in together, so the batch flow asks one place.
  • Opt-in is two fields on the existing auto_remediation configuration, as the validation spike recommended over a new profile type.
  • BatchRequest composes the existing single-dependency Request objects and checks that they share one project, package manager and lockfile. Both request types answer dependency_names, so JobBuilder takes either.
  • A batch branch is stable per lockfile, so a refresh reuses it. A digest of the raw path keeps it unique and short enough for a merge request source branch.
  • The occurrence and vulnerability pairs travel in one job variable, DEPENDENCY_MANAGEMENT_REMEDIATIONS, as the spike settled.
  • The updater lists a dependency as both updated and failed when its file update fails after its check, as the conflict spike showed. The parser lets the failure win only when the caller asks, so the batch flow can, and a single-dependency run keeps today's behaviour.

What

  • The dependency_management_batch_updates flag, of type wip and off by default. Its rollout issue is [Feature flag] Rollout of dependency_management... (#637446) • Philip Cunningham.
  • Configuration: batch_updates and max_updates_per_batch on auto_remediation.
    • The JSON schema, with a cap from 1 to 30.
    • The defaults, false and 10. The effective configuration merges every profile over these defaults, so it always carries both fields. The Conservative and Proactive presets are unchanged.
    • The AutoRemediationConfiguration type and the SecurityScanProfileAutoRemediationInput input, with the input arguments marked as experiments and descriptions that match the schema.
  • BatchUpdates.enabled?(project:, configuration:).
  • Requests:
    • BatchRequest, for the requests that share one lockfile, on dependency-management/batch/{package manager}/{lockfile path}-{digest}. The path is cut to 100 characters, so the longest branch is 148 characters, and the digest is 8 characters of the raw path's SHA-256.
    • Request.sanitize_ref_segment, shared by the single-dependency and batch branch names, and Request#dependency_names.
  • Parsing: OutputParser#failed_dependencies, with name, directory, stage and reason. With failed_overrides_updated: true, a failed name leaves dependencies, whatever its case. OutputParser.comparable_name is the one name comparison, used by that filter and by version_for.
  • Job input: JobBuilder sends the request's dependency_names and converts each Maven and Gradle name. UpdateService sends a BatchRequest's pairs in DEPENDENCY_MANAGEMENT_REMEDIATIONS, and a single request keeps its two id variables. The variable names the merge request worker and service read back are constants on UpdateService.
  • For Batched updates: admission-first scheduler, bat... (#635920) • Philip Cunningham • 19.7: parse batch output with failed_overrides_updated: true, read the cap from configuration[:max_updates_per_batch], and read the pairs through UpdateService::REMEDIATIONS_VARIABLE.
  • The flag is off, and nothing calls the new code until then. A single request builds the same job and the same workload variables as before.
  • No changelog entry. The new GraphQL arguments are experiments, and nothing acts on the fields until the flag is on.

References

Manual QA

In a GDK Rails console, on a project with a remediation profile and bundler findings in Gemfile.lock, for example gitlab-duo/bundler-multi-demo:

su = DependencyManagement::SecurityUpdate
project = Project.find_by_full_path('gitlab-duo/bundler-multi-demo')

Feature.enable(:dependency_management_batch_updates, project)
su::BatchUpdates.enabled?(project: project, configuration: { batch_updates: true }) # => true
su::BatchUpdates.enabled?(project: project, configuration: {})                      # => false
Feature.disable(:dependency_management_batch_updates, project)
su::BatchUpdates.enabled?(project: project, configuration: { batch_updates: true }) # => false

config = su::Eligibility.remediation_configuration(project).fetch(:auto_remediation, {})
config.slice(:batch_updates, :max_updates_per_batch) # => {batch_updates: false, max_updates_per_batch: 10}

requests = project.vulnerabilities.where(state: %i[detected confirmed]).filter_map do |vulnerability|
  next unless su::Eligibility.remediable?(vulnerability)

  occurrence = vulnerability.sbom_occurrences.find { |o| o.input_file_path == 'Gemfile.lock' }
  su::Request.new(sbom_occurrence: occurrence, vulnerability: vulnerability) if occurrence
end.uniq(&:dependency).first(3)
batch = su::BatchRequest.new(requests: requests)
batch.source_ref       # => "dependency-management/batch/bundler/gemfile-lock-89cade48"
batch.dependency_names # => three names, for example ["nokogiri", "rack", "puma"]
su::BatchRequest.source_ref_for('bundler', "#{'a' * 1011}/Gemfile.lock").length # => 145

su::JobBuilder.new(request: batch, project: project,
  auto_remediation_configuration: config).build['dependencies'] # => the same three names
su::JobBuilder.new(request: requests.first, project: project,
  auto_remediation_configuration: config).build['dependencies'] # => the first name alone

json = {
  dependency_updates: [{ name: 'rack', new_version: '2.2.24' }, { name: 'puma', new_version: '5.6.9' }],
  updated_files: [],
  failed_dependencies: [{ name: 'puma', directory: '/', stage: 'file_update', reason: 'conflict' }, { directory: '/' }]
}.to_json
output = su::OutputParser.parse(json, failed_overrides_updated: true)
output.dependencies.map(&:name)                     # => ["rack"]
output.failed_dependencies.size                     # => 1
su::OutputParser.parse(json).dependencies.map(&:name) # => ["rack", "puma"], as before

GDK result on 32280537, gitlab-duo/bundler-multi-demo:

Check Result
Flag off, batch_updates: true false
Flag on, batch_updates: true and unset true and false
Effective configuration batch_updates: false, max_updates_per_batch: 10
GraphQL fields on AutoRemediationConfiguration batchUpdates, maxUpdatesPerBatch
Batch branch dependency-management/batch/bundler/gemfile-lock-89cade48
Branch for a 1,024-character path 145 characters
services/api_v1 and services/api-v1 lockfiles Two branches, ...-6c1cf4ea and ...-d0260a70
Batch job dependencies nokogiri, rack, puma, with directory /
Single request job dependencies nokogiri
Batch output with puma in both arrays updated rack, failed puma at file_update
Single run output with puma in both arrays updated rack and puma, as before
Edited by Philip Cunningham

Merge request reports

Loading
Loading