Skip auto-remediation when the service account cannot open MRs

Why

The gap

  • The auto-remediation service account joins a project as a Guest, and an internal role grants it what it needs.
  • If a project limits merge requests to project members, only users with at least the Reporter role can create one. The internal role does not override that.
  • Auto-remediation only finds out at the end. The workload runs and the update is committed, then MergeRequests::CreateService raises Gitlab::Access::AccessDeniedError.
  • The catch-all reports it to Sentry as "Unexpected error creating merge request". The next SBOM ingestion then starts the same work again.
  • This is the bug in https://gitlab.com/gitlab-org/gitlab/-/work_items/636802+s.

Why this approach

  • The policy already decides. The check asks it what MergeRequests::CreateService#set_projects! asks, create_merge_request_from and create_merge_request_in. It reads the project's feature settings only to name the reason.
  • A check before any work means a project that cannot take a merge request does none.
  • The check runs every time, so a Reporter role granted to the account takes effect on the next run.
  • No feature flag. A denial the check acts on is one that merge request creation would hit anyway, so skipping only saves the workload. A revert is the rollback.
  • The account stays a Guest, with no policy exception. A configurable role is Make the dependency management service account ... (#632654) • Unassigned.

What

  • DependencyManagement::BaseService#merge_request_denial returns nil when both abilities are allowed.
    • Otherwise it logs DependencyManagement: service account cannot create merge requests at info, with denial_reason and gl_project_id, and returns the reason.
    • merge_request_denial_response turns the reason into the error response, so UpdateService, CreateMergeRequestService and the rescue share one.
  • The reasons, in the order they are checked:
    • repository_disabled and merge_requests_disabled, when the project turns either feature off.
    • merge_requests_private, only when the members-only setting alone blocks the account. Merge requests must be out of the account's reach, while push_code, from the same internal role, is still allowed.
    • access_denied for anything else: an archived project, a removed membership or a read-only namespace. The Reporter role would fix none of these. A read-only namespace keeps push_code, so it is checked by name.
  • Three checks, before any work:
    • SchedulerService#execute, after the profile check and before the refresh of open merge requests. It reads the existing account and never provisions one. With no account yet, UpdateService provisions it and checks, and the first refusal ends the scheduler's walk.
    • UpdateService#start_workload, before any branch write or workload. This covers the manual remediation from the vulnerability report, which shows Service account cannot create merge requests in this project. UpdateService also logs its existing failed to start an auto-remediation workload error line.
    • CreateMergeRequestService#execute, before it reads the output, for workloads already running when this deploys.
  • create_new_merge_request rescues Gitlab::Access::AccessDeniedError by name. It returns the same message and logs the denial line with access_denied, with no Sentry event, for access that changes between the check and the create.
  • A denial that reaches the refresh of open merge requests ends the scheduler's run there, as an invalid repository config does. Only a change in access during the run gets that far.
  • A denied project's open remediation merge requests stop being refreshed too, because the skip comes before the refresh. The same policy rule prevents update_merge_request, though MergeRequests::UpdateService does not check it.
  • The docs correct "Service account permissions". The Troubleshooting entry lists the symptoms and every cause, with its denial_reason and resolution.
  • The log field is denial_reason, not reason, because Labkit lists reason as a deprecated alias of message.
Case Result
Members-only merge requests, Guest account No workload, reason merge_requests_private
Archived project with members-only merge requests access_denied
Account no longer a member, members-only merge requests access_denied
No account yet, members-only merge requests UpdateService provisions the account and refuses, and the scheduler stops at the first refusal
No account yet, merge requests open to all Provisions the account and starts the workload
Account with the Reporter role Works as today
Merge requests or repository disabled merge_requests_disabled or repository_disabled
Archived project access_denied
Read-only namespace, members-only merge requests access_denied
Denial during the refresh of open merge requests The scheduler stops without refreshing the rest or scanning
Access removed after the check The message and the denial line, without Sentry

References

Manual QA

GDK on 86f7e0f8, on a project with a remediation profile and the GitLab Dependency Management service account as a Guest. The workloads ran on the GDK runner.

  1. Limit the project's merge requests to members. Remediate a vulnerability from the vulnerability report, then run the scheduler as an SBOM ingestion would.
  2. Archive the project, then run the manual remediation and the scheduler. Unarchive it.
  3. Give the service account the Reporter role, and remediate from the report again.
  4. Restore the defaults, with merge requests open to everyone with access and the service account a Guest. Remediate another vulnerability. Then let an open merge request's refresh fall due and run the scheduler.
Case Merge requests Service account Path Workloads Result
Denied Only Project Members Guest Manual remediation 0 The report shows Service account cannot create merge requests in this project, reason merge_requests_private
Denied Only Project Members Guest Scheduler 0 One info line, merge_requests_private, and no branch commit
Archived Only Project Members Guest Manual remediation and scheduler 0 Reason access_denied, not merge_requests_private
Fixed by the role Only Project Members Reporter Manual remediation 1, passed The rack merge request gets a new commit
Defaults Everyone With Access Guest Manual remediation 1, passed Reaches the merge request, where the output is unchanged and no commit is needed
Defaults Everyone With Access Guest Scheduler refresh 1, passed Reaches the merge request, where the output is unchanged and no commit is needed

The defaults rows end in the existing skip for unchanged output, from !261596 (merged), because earlier runs had already written these updates. No denial line appears once the defaults are restored. A spec covers the first refusal ending the scheduler's walk, which needs a project with no account yet.

The log lines below are from log/application_json.log, with empty fields dropped.

Log lines: denied
{"time":"2026-10-30T13:40:22.765Z","severity":"INFO","message":"DependencyManagement: service account cannot create merge requests","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"denial_reason":"merge_requests_private"}
{"time":"2026-10-30T13:40:22.765Z","severity":"ERROR","message":"DependencyManagement: failed to start an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"error_message":"Service account cannot create merge requests in this project"}
{"time":"2026-10-30T13:40:41.992Z","severity":"INFO","message":"DependencyManagement: service account cannot create merge requests","class_name":"DependencyManagement::SecurityUpdate::SchedulerService","gl_project_id":58,"denial_reason":"merge_requests_private"}
Log lines: archived
{"time":"2026-10-30T13:41:30.503Z","severity":"INFO","message":"DependencyManagement: service account cannot create merge requests","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"denial_reason":"access_denied"}
{"time":"2026-10-30T13:41:30.503Z","severity":"ERROR","message":"DependencyManagement: failed to start an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"error_message":"Service account cannot create merge requests in this project"}
{"time":"2026-10-30T13:41:30.564Z","severity":"INFO","message":"DependencyManagement: service account cannot create merge requests","class_name":"DependencyManagement::SecurityUpdate::SchedulerService","gl_project_id":58,"denial_reason":"access_denied"}
Log lines: fixed by the role
{"time":"2026-10-30T13:42:12.003Z","severity":"INFO","message":"DependencyManagement: started an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"workload_ref":"refs/workloads/bdcd8c3460c","gl_pipeline_id":1168}
{"time":"2026-10-30T13:42:39.762Z","severity":"INFO","message":"DependencyManagement::SecurityUpdate: merge request created or updated","merge_request_id":6272}
Log lines: defaults
{"time":"2026-10-30T13:44:26.318Z","severity":"INFO","message":"DependencyManagement: started an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"workload_ref":"refs/workloads/30b47464e20","gl_pipeline_id":1170}
{"time":"2026-10-30T13:44:50.037Z","severity":"INFO","message":"DependencyManagement: workload output matches the open merge request, skipping commit","gl_project_id":58,"skip_reason":"unchanged"}
{"time":"2026-10-30T13:46:02.629Z","severity":"INFO","message":"DependencyManagement: started an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":58,"workload_ref":"refs/workloads/73d9ba1cd5d","gl_pipeline_id":1171}
{"time":"2026-10-30T13:46:24.110Z","severity":"INFO","message":"DependencyManagement: workload output matches the open merge request, skipping commit","gl_project_id":58,"skip_reason":"unchanged"}
Screenshots: denied

Merge requests limited to project members:

01-mr-visibility-members-only

The service account is a Guest:

02-members-guest

Remediating from the vulnerability report:

03-denied-resolve-popover

The report shows the error, and no workload starts:

03-denied-resolve-after

Screenshots: archived

The project is archived, with merge requests still limited to members. The log lines above show access_denied for both paths:

04-archived-project

Screenshots: fixed by the role

The service account has the Reporter role:

05-members-reporter

The remediation starts:

06-reporter-resolve-after

The workload passes on a hidden workload ref:

07-reporter-workload-pipeline

The merge request gets the new commit:

08-reporter-mr-updated

Screenshots: defaults

Merge requests are open to everyone with access again:

09-mr-visibility-everyone

The remediation starts:

10-default-resolve-after

The scheduler's refresh workload passes:

11-default-scheduler-workload

The merge requests, each with a passing pipeline:

12-default-mr-list

A spec covers the race at creation by raising AccessDeniedError from MergeRequests::CreateService. It returns the message with no Sentry event.

Re-run after the rebase

GDK on f32923f9, after the rebase onto the scheduler rate limit, on a project with one open remediation merge request and the service account as a Guest. dependency_management_scheduler_rate_limit was on for the project, the other auto-remediation flags off. Each run published Sbom::SbomIngestedEvent for the project's last default-branch pipeline, which is how SBOM ingestion starts the scheduler.

Case Merge requests Result
Denied Only Project Members One info line from SchedulerService with merge_requests_private. No workload, and the due refresh is skipped
Throttled Only Project Members, within 5 minutes of the run above Only scheduler run throttled, so the rate limit check comes before the denial check
Defaults Everyone With Access, refresh due The refresh workload passes and the merge request gets a new commit. No denial line
Log lines: re-run
{"time":"2026-11-11T15:18:02.197Z","severity":"INFO","message":"DependencyManagement: service account cannot create merge requests","class_name":"DependencyManagement::SecurityUpdate::SchedulerService","gl_project_id":111,"denial_reason":"merge_requests_private"}
{"time":"2026-11-11T15:19:45.851Z","severity":"INFO","message":"DependencyManagement: scheduler run throttled","class_name":"DependencyManagement::SecurityUpdate::SchedulerService","gl_project_id":111,"rate_limit_key":"dependency_management_scheduler_run"}
{"time":"2026-11-11T15:24:29.166Z","severity":"INFO","message":"DependencyManagement: started an auto-remediation workload","class_name":"DependencyManagement::SecurityUpdate::UpdateService","gl_project_id":111,"gl_pipeline_id":1245}
{"time":"2026-11-11T15:30:50.092Z","severity":"INFO","message":"DependencyManagement::SecurityUpdate: merge request created or updated","merge_request_id":261}
Edited by Albina Yusupova

Merge request reports

Loading
Loading