Skip to content

Skip copy job artifacts uploaded to final location

Erick Bajao requested to merge eb-skip-copy-job-artifact-upload into master

What does this MR do and why?

Resolves #285597 (closed)

Job artifacts directly uploaded to object storage don't need to be copied over by CarrierWave. We just need to give the final upload location to Workhorse instead of a temporary one.

This MR contains the bulk of the work. A separate MR will be opened to add the scheduled worker that would clean up old and stale pending remote upload records.

The feature is dependent on !106315 (merged) for it to fully work but is not a blocker for merging because this is disabled under a feature flag.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Setup a project that uploads an artifact. You can fork this sample project: https://gitlab.com/iamricecake/basic-artifact-test.
  2. Make sure you have direct upload and object storage enabled on your GDK.
  3. Now before we run our first pipeline, have a separate tab open to observe the minio trace (http://gdk.minio:9002/tools/trace). This is to confirm that enabling the flag will now skip copy/delete of objects. Click on "Start".
  4. Now run the first pipeline on the test project. This should upload an archive artifact. We can then see on the minio trace that there are calls to copy and delete: Screenshot_2023-05-03_at_6.26.20_PM
  5. Now, enable the :ci_artifacts_upload_to_final_location FF in the rails console:
    Feature.enable(:ci_artifacts_upload_to_final_location)
  6. Reload the minio trace page so that we have a clean table. Click on "Start" again.
  7. Then let's re-run the job. On complete, confirm that the artifacts are downloadable/browsable on the job details page.
  8. Now go back to the minio trace page and confirm that now there won't be copy/delete calls: Screenshot_2023-05-03_at_6.29.08_PM
  9. This means the artifact was uploaded directly to its final location.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Erick Bajao

Merge request reports