Resumable workloads for DAP flows
## Summary
When your AI agent pauses to wait for your approval, it no longer loses its place — GitLab now suspends the agent's full environment (including file changes, installed dependencies, and git state) and resumes it in under 10 seconds once you respond. This means you get faster, more reliable Human-in-the-Loop workflows without the frustrating minutes-long cold-start delays or the agent having to redo work it already completed.
## Problem Statement
For the MVP version of [Approval Human In The Loop Node (HITL)](https://gitlab.com/groups/gitlab-org/-/work_items/20652), DAP flows handle resume actions by creating a brand new workload each time. This approach has significant drawbacks:
- **Dependency reinstall overhead**: Every resume re-installs DAP dependencies and re-sets up the git and SRT environment from scratch, adding minutes of cold-start delay.
- **Loss of WIP changes**: Any in-progress file changes made by the agent in the current environment are discarded. The agent must redo that work based on workflow checkpoints alone.
- **Resource inefficiency**: Keeping a workload alive while waiting for human input wastes compute; tearing it down and recreating it wastes time.
## Proposal
Leverage [Resumable Jobs for CI and Agent Sessions](https://gitlab.com/groups/gitlab-org/-/work_items/21159) to introduce true suspend/resume semantics for DAP workloads.
When a DAP flow reaches a `human_request` node, the underlying Runner job is **suspended** rather than terminated. The full environment state — git repo, file changes, installed dependencies, agent session context — is snapshotted and persisted. When `human_input` is received, the same workload **resumes** from exactly where it left off, with near-instant startup and zero rework.
This turns the current create-new-workload-on-every-resume pattern into a true pause/resume lifecycle:
```
human_request node hit → Suspend workload → Persist environment snapshot
human_input received → Resume workload → Restore environment, continue execution
```
## Infrastructure Dependency
Resumable Jobs require a modern Runner executor that supports suspend/resume semantics. The current fleet serving the `gitlab--duo` CI tag uses the legacy **docker+machine** executor, which does not support this. A dedicated fleet using a modern executor must be provisioned to unblock DAP teams.
Supported modern executors:
- **Docker Autoscaler** (via Fleeting + Taskscaler) — targeted for %19.0
- **Instance Autoscaler** (via Fleeting + Taskscaler)
- **Kubernetes** (planned, with gVisor isolation)
This is tracked in [Modern executor fleet on .com for Resumable Jobs and AI workloads](https://gitlab.com/gitlab-org/gitlab/-/issues/597039) and [Provision a runner fleet with a modern executor for the `duo` tag](https://gitlab.com/gitlab-org/gitlab/-/issues/597038).
## Key Constraints
- A resumed job must return to **exactly the runner that suspended it**, since only that runner's taskscaler holds the suspended acquisition. This routing enforcement is tracked in [Ensure resumed CI jobs return to the correct runner](https://gitlab.com/gitlab-org/gitlab/-/issues/596841).
- The DAP team is shipping an initial HITL MVC in %18.11 using a [stopgap that does not require runner-side suspend/resume](https://gitlab.com/groups/gitlab-org/-/epics/20652#note_3150140098). The full environment-preserving experience depends on Resumable Jobs being available by %19.0.
## Benefits Over Current Approach
| | Current (new workload per resume) | With Resumable Workloads |
|--|-----------------------------------|--------------------------|
| Resume startup time | 2-5 min (reinstall deps, re-clone) | \< 10s (volume reattach) |
| WIP file changes | Lost on suspend | Preserved across pause |
| Installed dependencies | Reinstalled every time | Persisted in snapshot |
| Git environment | Re-cloned every time | Preserved in snapshot |
| Compute during pause | Idle (wasted) or torn down | Released; no cost during pause |
## Success Metrics
| Metric | Target |
|--------|--------|
| Workload resume time | \< 10s from `human_input` received to agent continuing execution |
| Environment continuity | Git state, file changes, and installed dependencies fully preserved across pause/resume; 0 agent rework required |
| Compute efficiency | No compute consumed during the pause window between `human_request` and `human_input` |
| HITL flow shipped | 1 end-to-end DAP flow with HITL using suspend/resume delivered by %"19.10" |
## Related Epics and Issues
- [Resumable Jobs for CI and Agent Sessions](https://gitlab.com/groups/gitlab-org/-/work_items/21159) — the underlying Runner capability this epic depends on
- [Approval Human In The Loop Node (HITL)](https://gitlab.com/groups/gitlab-org/-/work_items/20652) — the DAP flow feature this epic unblocks
- [Remote Access into Jobs for Agents & Humans](https://gitlab.com/groups/gitlab-org/-/work_items/21160) — complementary capability enabling SSH access into live Runner environments
- [Blueprint - Resumable Jobs for CI and Agent Sessions](https://gitlab.com/gitlab-org/gitlab/-/issues/593314) — architecture blueprint
- [Ensure resumed CI jobs return to the correct runner](https://gitlab.com/gitlab-org/gitlab/-/issues/596841) — runner routing enforcement
- [Modern executor fleet on .com for Resumable Jobs and AI workloads](https://gitlab.com/gitlab-org/gitlab/-/issues/597039) — .com fleet provisioning
- [Provision a runner fleet with a modern executor for the `duo` tag](https://gitlab.com/gitlab-org/gitlab/-/issues/597038) — dedicated `duo` tag fleet
epic