Loading
Flag, configuration, BatchRequest, OutputParser and JobBuilder for batched updates
Stack
Merge from the bottom up. Each MR targets the one below it.
- !266781 (merged) (top)
- !266763 (merged) (bottom) <- you're here
Why
The gap
- Batching auto-remediation updates per lockfile needs a flag, two configuration fields, a request that covers one lockfile, a parser for the updater's failure list, and a job that names every dependency.
- None of them exists yet, and the batch flow in Batched updates: admission-first scheduler, bat... (#635920) • Philip Cunningham • 19.7 needs all of them.
- None of them needs anything else from Allow users to batch dependency updates (gitlab-org#24196) • Philip Cunningham, Albina Yusupova • 19.7, so they can land first.
Why this approach
- One reader,
BatchUpdates.enabled?, checks the flag and the opt-in together, so the batch flow asks one place. - Opt-in is two fields on the existing
auto_remediationconfiguration, as the validation spike recommended over a new profile type. BatchRequestcomposes the existing single-dependencyRequestobjects and checks that they share one project, package manager and lockfile. Both request types answerdependency_names, soJobBuildertakes either.- A batch branch is stable per lockfile, so a refresh reuses it. A digest of the raw path keeps it unique and short enough for a merge request source branch.
- The occurrence and vulnerability pairs travel in one job variable,
DEPENDENCY_MANAGEMENT_REMEDIATIONS, as the spike settled. - The updater lists a dependency as both updated and failed when its file update fails after its check, as the conflict spike showed. The parser lets the failure win only when the caller asks, so the batch flow can, and a single-dependency run keeps today's behaviour.
What
- The
dependency_management_batch_updatesflag, of typewipand off by default. Its rollout issue is [Feature flag] Rollout of dependency_management... (#637446) • Philip Cunningham. - Configuration:
batch_updatesandmax_updates_per_batchonauto_remediation.- The JSON schema, with a cap from 1 to 30.
- The defaults,
falseand 10. The effective configuration merges every profile over these defaults, so it always carries both fields. The Conservative and Proactive presets are unchanged. - The
AutoRemediationConfigurationtype and theSecurityScanProfileAutoRemediationInputinput, with the input arguments marked as experiments and descriptions that match the schema.
BatchUpdates.enabled?(project:, configuration:).- Requests:
BatchRequest, for the requests that share one lockfile, ondependency-management/batch/{package manager}/{lockfile path}-{digest}. The path is cut to 100 characters, so the longest branch is 148 characters, and the digest is 8 characters of the raw path's SHA-256.Request.sanitize_ref_segment, shared by the single-dependency and batch branch names, andRequest#dependency_names.
- Parsing:
OutputParser#failed_dependencies, with name, directory, stage and reason. Withfailed_overrides_updated: true, a failed name leavesdependencies, whatever its case.OutputParser.comparable_nameis the one name comparison, used by that filter and byversion_for. - Job input:
JobBuildersends the request'sdependency_namesand converts each Maven and Gradle name.UpdateServicesends aBatchRequest's pairs inDEPENDENCY_MANAGEMENT_REMEDIATIONS, and a single request keeps its two id variables. The variable names the merge request worker and service read back are constants onUpdateService. - For Batched updates: admission-first scheduler, bat... (#635920) • Philip Cunningham • 19.7: parse batch output with
failed_overrides_updated: true, read the cap fromconfiguration[:max_updates_per_batch], and read the pairs throughUpdateService::REMEDIATIONS_VARIABLE. - The flag is off, and nothing calls the new code until then. A single request builds the same job and the same workload variables as before.
- No changelog entry. The new GraphQL arguments are experiments, and nothing acts on the fields until the flag is on.
References
- Closes #635918 (closed)
- Epic: Allow users to batch dependency updates (gitlab-org#24196) • Philip Cunningham, Albina Yusupova • 19.7
- Plan: the implementation plan in the issue description
Manual QA
In a GDK Rails console, on a project with a remediation profile and bundler findings in Gemfile.lock, for example gitlab-duo/bundler-multi-demo:
su = DependencyManagement::SecurityUpdate
project = Project.find_by_full_path('gitlab-duo/bundler-multi-demo')
Feature.enable(:dependency_management_batch_updates, project)
su::BatchUpdates.enabled?(project: project, configuration: { batch_updates: true }) # => true
su::BatchUpdates.enabled?(project: project, configuration: {}) # => false
Feature.disable(:dependency_management_batch_updates, project)
su::BatchUpdates.enabled?(project: project, configuration: { batch_updates: true }) # => false
config = su::Eligibility.remediation_configuration(project).fetch(:auto_remediation, {})
config.slice(:batch_updates, :max_updates_per_batch) # => {batch_updates: false, max_updates_per_batch: 10}
requests = project.vulnerabilities.where(state: %i[detected confirmed]).filter_map do |vulnerability|
next unless su::Eligibility.remediable?(vulnerability)
occurrence = vulnerability.sbom_occurrences.find { |o| o.input_file_path == 'Gemfile.lock' }
su::Request.new(sbom_occurrence: occurrence, vulnerability: vulnerability) if occurrence
end.uniq(&:dependency).first(3)
batch = su::BatchRequest.new(requests: requests)
batch.source_ref # => "dependency-management/batch/bundler/gemfile-lock-89cade48"
batch.dependency_names # => three names, for example ["nokogiri", "rack", "puma"]
su::BatchRequest.source_ref_for('bundler', "#{'a' * 1011}/Gemfile.lock").length # => 145
su::JobBuilder.new(request: batch, project: project,
auto_remediation_configuration: config).build['dependencies'] # => the same three names
su::JobBuilder.new(request: requests.first, project: project,
auto_remediation_configuration: config).build['dependencies'] # => the first name alone
json = {
dependency_updates: [{ name: 'rack', new_version: '2.2.24' }, { name: 'puma', new_version: '5.6.9' }],
updated_files: [],
failed_dependencies: [{ name: 'puma', directory: '/', stage: 'file_update', reason: 'conflict' }, { directory: '/' }]
}.to_json
output = su::OutputParser.parse(json, failed_overrides_updated: true)
output.dependencies.map(&:name) # => ["rack"]
output.failed_dependencies.size # => 1
su::OutputParser.parse(json).dependencies.map(&:name) # => ["rack", "puma"], as beforeGDK result on 32280537, gitlab-duo/bundler-multi-demo:
| Check | Result |
|---|---|
Flag off, batch_updates: true |
false |
Flag on, batch_updates: true and unset |
true and false |
| Effective configuration | batch_updates: false, max_updates_per_batch: 10 |
GraphQL fields on AutoRemediationConfiguration |
batchUpdates, maxUpdatesPerBatch |
| Batch branch | dependency-management/batch/bundler/gemfile-lock-89cade48 |
| Branch for a 1,024-character path | 145 characters |
services/api_v1 and services/api-v1 lockfiles |
Two branches, ...-6c1cf4ea and ...-d0260a70 |
| Batch job dependencies | nokogiri, rack, puma, with directory / |
| Single request job dependencies | nokogiri |
Batch output with puma in both arrays |
updated rack, failed puma at file_update |
Single run output with puma in both arrays |
updated rack and puma, as before |
Edited by Philip Cunningham