Retry CI trace archive if left in incomplete state

What does this MR do?

This aims to remove fix some of the cases that could allow Ci::Builds to have both a live trace & and archived trace.

What is changing?

The code change here allows an archived_trace that is incomplete to be removed, and rebuilt in a single run of Ci::Trace#unsafe_archive!. Previously, we would remove the archived_trace and raise an AlreadyArchivedError without rebuilding the archive file.

Scenario	Previous	New
trace_artifact exists, with file	trace chunks removed, AlreadyArchivedError raised	Same as previous
trace_artifact exists, no file	trace_artifact destroyed, AlreadyArchivedError raised	trace_artifact destroyed, archive process runs
trace_artifact does not exist	archive process runs	Same as previous

In what scenarios does this help?

Ci::ArchiveTracesCronWorker

1. Archive fails
2. The worker will retry.
3. Ci::ArchiveTraceService will be able to remove the old archive and attempt to create a new one in the same run.

Previous behaviour:

1. Archive fails
2. The worker will retry.
3. Ci::ArchiveTraceService will remove the archive, and raise AlreadyArchivedError which the Ci::ArchiveTraceService treats as success.
4. Will not be retried until Ci::ArchiveTracesCronWorker runs again (every 17 mins)

Screenshots or Screencasts (strongly suggested)

How to setup and validate locally (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.
• This change is backwards compatible across updates, or this does not apply.

Availability and Testing

• I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
• I have tested this MR in all supported browsers, or it's not needed.
• I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Related to #296616 (closed)

changed milestone to %14.3

added All SaaS Category:Continuous Integration Engineering Allocation FY22Q3 Verify candidate VerifyP1 backend corrective action devopsverify grouppipeline execution infradev missed:14.0 priority3 reliability sectionops severity3 typebug workflowready for development labels

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Alexandru Croitor (@acroitor) (UTC+3, 9 hours behind @seanarnold)	Vitali Tatarintev (@ck3g) (UTC+2, 10 hours behind @seanarnold)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

changed the description

marked this merge request as ready

changed title from Retry archive if left in incomplete state to Retry CI trace archive if left in incomplete state

marked this merge request as draft

changed the description

mentioned in issue #296616 (closed)

marked this merge request as ready

added 2 commits

• d2f675c7 - Retry archive if left in incomplete state
• d1050c1c - Scope builds by live trace, not no archived trace

Compare with previous version

@grzesiek are you able to have a look at this please?

The intention here is to ensure the jobs & service is more resilient to automatically handle failure situations.

• Ci::ArchiveTraceWorker will no process any build that has a live trace, which allows retrying
• #unsafe_archive! will now be able to retry an archive (delete an artefact and re-create it)

requested review from @grzesiek

assigned to @seanarnold

requested review from @mbobin

added 1 commit

• f5ff6d81 - Simply logic in unsafe_archive!

Compare with previous version

removed review request for @grzesiek

added 532 commits

• f5ff6d81...a149b49d - 528 commits from branch master
• 4e4e98ac - Retry archive if left in incomplete state
• b291faa2 - Scope builds by live trace, not no archived trace
• 1d2c9d17 - Simply logic in unsafe_archive!
• bb124a2f - Conditional to determine scope in archive worker

Compare with previous version

Thanks @mbobin. Are you able to continue the review? I made a change to put the scope a FF conditional. Let me know what you think.

removed review request for @mbobin

added 1 commit

• 366c39b9 - Rename method, remove change to worker scope

Compare with previous version

changed the description

requested review from @mbobin

approved this merge request

removed review request for @mbobin

@mbobin, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline has been started.

For more info, please refer to the following links:

• rspec minimal jobs
• merging a merge request.

added 1 commit

• 7a85e3e8 - Remove safe method call

Compare with previous version

@smcgivern would you please do the maintainer backend review here?

requested review from @smcgivern

marked the checklist item I have included changelog trailers, or none are needed. (Does this MR need a changelog?) as completed

marked the checklist item I have added/updated documentation, or it's not needed. (Is documentation required?) as completed

marked the checklist item I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?) as completed

marked the checklist item I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?) as completed

marked the checklist item This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) as completed

marked the checklist item I have self-reviewed this MR per code review guidelines. as completed

marked the checklist item I have followed the style guides. as completed

marked the checklist item This change is backwards compatible across updates, or this does not apply. as completed

mentioned in issue gitlab-com/www-gitlab-com#11848 (closed)

approved this merge request

requested review from @grzesiek and removed review request for @smcgivern

removed review request for @grzesiek

changed the description

added 1 commit

• 644773dc - Revert comment changes above destroy

Compare with previous version

requested review from @grzesiek

added 1309 commits

• 644773dc...fa457780 - 1302 commits from branch master
• 8deb1c70 - Retry archive if left in incomplete state
• a85209d6 - Scope builds by live trace, not no archived trace
• 99d5668c - Simply logic in unsafe_archive!
• dda6ed2b - Conditional to determine scope in archive worker
• 35b1c1a5 - Rename method, remove change to worker scope
• 93b92dbf - Remove safe method call
• bc509e2f - Revert comment changes above destroy

Compare with previous version

approved this merge request

resolved all threads

So it seems that we do not raise an exception when a trace artifact database entry exists, but a corresponding object in object storage has not been found.

It means that we can retry this operation immediately after we remove the tracking entry since it seems that it has ben erroneously created.

We are doing that in the write lock, so it looks good to me! MWPS set!

enabled an automatic merge when the pipeline for 0f211117 succeeds

merged

mentioned in commit 19043952

added workflowstaging-canary label and removed workflowready for development label

added workflowstaging label and removed workflowstaging-canary label

added workflowcanary label and removed workflowstaging label

added workflowproduction label and removed workflowcanary label

added releasedcandidate label

added releasedpublished label and removed releasedcandidate label

mentioned in merge request kubitus-project/kubitus-installer!236 (merged)