Retry DAP source acquisition with exponential backoff

What does this MR do and why?

Transient Gitaly pack-objects load shedding can fail source acquisition before a DAP agent starts. Adds a retry at the startup stage when git access happens without restarting agent execution or replaying its side effects.

Behind the existing default-off, project-scoped dap_session_source_fetch_retry flag, configure:

  • GET_SOURCES_ATTEMPTS: 6 — six total source-acquisition attempts.

References

Screenshots or screen recordings

Before After
no_retry_job673 gitaly_retry_mid

How to set up and validate locally

  1. Insert the following diff in your local gitaly and restart it to simulate gitaly failure:
diff --git a/internal/gitaly/service/smarthttp/upload_pack.go b/internal/gitaly/service/smarthttp/upload_pack.go
index 2fdd6b0e5..5a3213bed 100644
--- a/internal/gitaly/service/smarthttp/upload_pack.go
+++ b/internal/gitaly/service/smarthttp/upload_pack.go
@@ -19,6 +19,9 @@ import (
 )

 func (s *server) PostUploadPackWithSidechannel(ctx context.Context, req *gitalypb.PostUploadPackWithSidechannelRequest) (*gitalypb.PostUploadPackWithSidechannelResponse, error) {
+       // TEMPORARY TEST INJECTION: always shed load to demo DAP get_sources retry
+       return nil, structerr.NewResourceExhausted("resource exhausted, please try again later")
+
        repoPath, gitConfig, err := s.validateUploadPackRequest(ctx, req)
        if err != nil {
                return nil, structerr.NewInvalidArgument("%w", err)
  1. Try a DAP job (e.g. mention Duo Developer), see it fail after a single attempt
  2. Turn on dap_session_job_retry for the project / instance
  3. Try a DAP job again, see it retry 6 times

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

  • Bounded retries remain before agent execution; whole-job retry removed.
  • Default-off project flag retained, with enabled/disabled and resume coverage.
  • Focused specs, RuboCop, and whitespace checks passed.
  • Verify Runner compatibility and fault-injection behavior before rollout.

Changelog note: this retry behavior is behind a default-off flag. When squashing, remove the first commit's Changelog: fixed trailer. No history rewrite is included here.

Edited by Sebastian Rehm

Merge request reports

Loading
Loading