Loading
Retry DAP source acquisition with exponential backoff
What does this MR do and why?
Transient Gitaly pack-objects load shedding can fail source acquisition before a DAP agent starts. Adds a retry at the startup stage when git access happens without restarting agent execution or replaying its side effects.
Behind the existing default-off, project-scoped dap_session_source_fetch_retry flag, configure:
GET_SOURCES_ATTEMPTS: 6— six total source-acquisition attempts.
References
- Source-acquisition failures: https://gitlab.com/gitlab-org/gitlab/-/work_items/627944
- Feature flag rollout: #628054
- Related work: #627739 (closed)
Screenshots or screen recordings
| Before | After |
|---|---|
![]() |
![]() |
How to set up and validate locally
- Insert the following diff in your local gitaly and restart it to simulate gitaly failure:
diff --git a/internal/gitaly/service/smarthttp/upload_pack.go b/internal/gitaly/service/smarthttp/upload_pack.go
index 2fdd6b0e5..5a3213bed 100644
--- a/internal/gitaly/service/smarthttp/upload_pack.go
+++ b/internal/gitaly/service/smarthttp/upload_pack.go
@@ -19,6 +19,9 @@ import (
)
func (s *server) PostUploadPackWithSidechannel(ctx context.Context, req *gitalypb.PostUploadPackWithSidechannelRequest) (*gitalypb.PostUploadPackWithSidechannelResponse, error) {
+ // TEMPORARY TEST INJECTION: always shed load to demo DAP get_sources retry
+ return nil, structerr.NewResourceExhausted("resource exhausted, please try again later")
+
repoPath, gitConfig, err := s.validateUploadPackRequest(ctx, req)
if err != nil {
return nil, structerr.NewInvalidArgument("%w", err)- Try a DAP job (e.g. mention Duo Developer), see it fail after a single attempt
- Turn on
dap_session_job_retryfor the project / instance - Try a DAP job again, see it retry 6 times
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
- Bounded retries remain before agent execution; whole-job retry removed.
- Default-off project flag retained, with enabled/disabled and resume coverage.
- Focused specs, RuboCop, and whitespace checks passed.
- Verify Runner compatibility and fault-injection behavior before rollout.
Changelog note: this retry behavior is behind a default-off flag. When squashing, remove the first commit's Changelog: fixed trailer. No history rewrite is included here.
Edited by Sebastian Rehm

