UI artifacts download button was lost after expire timeout for latest artifacts

changed the description

mentioned in issue gitlab-org/quality/triage-reports#1218 (closed)

added automation:ml grouppipeline execution labels

This issue was automatically tagged with the label ~"group::continuous integration" by TanukiStan, a machine learning classification model, with a probability of 0.42.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help me learn from my mistakes.

Setting label(s) devopsverify sectionops based on ~"group::continuous integration".

added devopsverify sectionops labels

mentioned in issue gitlab-org/quality/triage-reports#1278 (closed)

mentioned in issue gitlab-org/quality/triage-reports#1373 (closed)

changed milestone to %Backlog

added CI artifacts Category:Continuous Integration labels

added [deprecated] Accepting merge requests label

@thaoyeager,

Sorry for interrupting, but you are only one person, whose name I found in that issue.

It looks like some issue tags were lost (such as Reproduced on Gitlab.com and bug), isn't it?

Also, I cannot find any information about the severity.

With regards,
Alex

@AJIOB Thanks for submitting this issue; I don't see from the history that the labels reproduced on GitLab.com and ~bug were previously applied (and then lost) so I am adding those labels now. The severity/priority will be added when this issue is triaged with Engineering's input.

@pburdette Do you know of any recent work that might have introduced this?

/cc @v_mishra @samdbeckham

@cheryl.li I looked through the commit history here for the last 4 months and nothing stuck out that would have introduced a regression here...

I don't believe this is a regression but inconsistency between views. I took a look at the data the FE is getting back for an artifact that expires. The pipeline details response does not contain any artifact data in the response so we do not show the artifacts download button.

But for the job log view we use the jobs endpoint for a certain job ID and the artifact exists on the payload. So this might be some inconsistency between the endpoints. I'm going to put a backend label on this issue too until we get it in a milestone to further investigate.

@cheryl.li now that I found this #217511 (comment 343599596) this makes me wonder if this has to do with locked artifacts now and is on purpose.

Maybe @matteeyah can speak on the history of this.

I'm not sure how we ended up with this behavior. We changed the logic multiple times, so I can't really think of what could have caused this.

This definitely sounds like a ~bug.

@pburdette @matteeyah Thanks for the analysis here

added reproduced on GitLab.com typebug labels

added to epic &5125

added UX frontend priority3 severity3 labels

added backend label

Perhaps this is best resolved via another issue, but I will put it here for now until I find a better place to raise the concern:

In addition to the buttons disappearing on the UX Jobs and Pipelines views, there is another issue. Even for latest artifacts which should never expire, they will be removed from the Pages artifact storage location if Pages is enabled for that project:

https://<namespace>.<pages_domain>/-/<project_path>/-/jobs/<job_number>/artifacts/public/*

If Pages is enabled for a project then any jobs (not just the pages job) that store artifacts will store them in the above link. This lets you do thinks like enable Review Apps for MRs and see the rendered Pages content before running the actual pages job against the default branch.

The artifacts are still available at this location even though the buttons are removed from the two views mentioned above:

https://<gitlab_domain>/<namespace>/<project_path>/-/jobs/<job_number>/artifacts

For the most recent job that keeps artifacts around forever, I would expect its associated Pages artifacts to stay around forever too.

Note that I am not talking about the artifacts for the actual pages job itself. Those artifacts do stay around even after the job artifacts expire, as they appear to be copied to a separate space. The only way to remove them seems to be to remove the Pages content via the project's Pages settings.

cc @ogolowinski it seems like something has changed with Pages artifacts

@shinya.maeda Could you take a look?

@mclamb I think you'd be interested in this issue #228764. Pages artifacts should be able to be persisted for longer-term by explicitly setting long expiry period e.g expire_in: 1 year

@shinya.maeda Thanks for the link! The behavior I am seeing is that the most recent job's "direct access" artifacts are expiring even though they are supposed to persist forever or until a newer job occurs. The artifact .zip is still available to download on the job page. This is effectively the same thing that is happening with the disappearing buttons issue: The arfifacts zip archive and download button persists on the specific job page regardless of expire_in which is expected behavior. But certain things adhere to expire_in even though they should ignore it for the most recent job: the UI buttons on the Jobs page and Pipelines page, along with these "direct access"* (Pages-related) links.

I call "direct access" the artifacts made available at https://<namespace>.<pages_domain>/-/<project_path>/-/jobs/<job_number>/artifacts/public/* -- These are not exactly the Pages artifacts themselves as the pages job transfers the the public/ folder to the actual pages store, which is different from the above. However, when Pages is enable, the above artifacts become available for direct browsing. I make use of this to create Pages Review Apps for dev branches within environments. My most recent job's Review App URL contents will disappear after the expire_in time, even though I would expect it to persist.

@mclamb Thank you for the explanation. Let me clarify a couple of points.

The behavior I am seeing is that the most recent job's "direct access" artifacts are expiring even though they are supposed to persist forever or until a newer job occurs

I think you're relying on the "Keep artifacts from most recent successful jobs" feature, correct?

IIRC, we haven't fixed some DB query so that expired-but-locked-artifacts are still treated as simply expired (in some places). We should handle such artifacts as non-expired artifacts i.e. showing a download link instead of hiding. cc @matteeyah

My most recent job's Review App URL contents will disappear after the expire_in time, even though I would expect it to persist.

This sounds like the artifacts-in-pages are deleted when the artifacts-in-gitlab is expired and doesn't take into account locked status. @vshushlin Do you happen to know the artifacts-in-pages deletion policy?

@shinya.maeda Yes to all of the above! And thanks for the proper terminology. Indeed artifacts-in-pages not being locked sounds like what I am talking about here.

@shinya.maeda @mclamb I'm very confused here.

GitLab Pages daemon doesn't persist any artifacts by itself, it just proxies gitlab artifacts using the API. Maybe API just considers this artifact expired the same way it happens with buttons?

@vshushlin Ah, OK. So, the build.artifacts? checks the artifacts_expired?. This seems simply outdated.

We should fix it like:

diff --git a/app/models/ci/build.rb b/app/models/ci/build.rb
index db151126caf..844b408cb32 100644
--- a/app/models/ci/build.rb
+++ b/app/models/ci/build.rb
@@ -626,7 +626,7 @@ def artifacts_metadata
     end
 
     def artifacts?
-      !artifacts_expired? && artifacts_file&.exists?
+      (!artifacts_expired? || locked_artifacts?) && artifacts_file&.exists?
     end
 
     def locked_artifacts?

@matteeyah IIRC, we created a follow-up issue to fix such inconsistency when we were working on the locked status. Do you happen to know where it is?

I searched through issues and I couldn't find anything other than #217511 (closed). This was an issue specifically for fixing the UI quirks that started after implementing artifact locking.

These are the changes that I made !32787 (diffs)

What do you call these artifacts that I am referring to?

IIUC the only way to publish to Pages is via the pages job which has special handling in that it takes the public folder from artifacts and seems to copy it into what I am calling the "Pages store" which is at /var/opt/gitlab/gitlab-rails/shared/pages/<namespace>/<project_name_slug> on my Omnibus installation.

I can then of course view the site at https://<namespace>.<pages_domain>/<project_name_slug>

BUT, I can also view the same contents another way via https://<namespace>.<pages_domain>/-/<project_name_slug>/-/jobs/<job_number>/artifacts/public/

And the nice trick with the above URL is those artifacts are now available from any job, not just the pages job. So once the pages job has run at least once, I can then leverage the above in other jobs to do Review Apps and the like, without actually "publishing" to Pages.

And this URL is what becomes inaccessible independent of locked status. What do you call this and what is its relationship to artifacts and the Pages store?

changed epic to &5755

added grouppipeline security label and removed grouppipeline execution label

added Category:Build Artifacts label

added 1 deleted label

@AJIOB - are you still experiencing this?

changed epic to &4923

Hello @jreporter

Yes, you are right, bug is still reproducible.

You can find bug reproduction links at the top of the topic.

With regards, Alex

thanks @AJIOB for the confirmation!

added vintage label

Hi @AJIOB ,

Thanks for raising this bug.

Contributions like this are vital to help make GitLab a better product.

We would be grateful for your help in verifying whether your bug report requires further attention from the team. If you think this bug still exists, and is reproducible with the latest stable version of GitLab, please comment on this issue.

This issue has been inactive for more than 12 months now and based on the policy for inactive bugs, will be closed in 7 days.

Thanks for your contributions to make GitLab better!

added stale label

closed

UI artifacts download button was lost after expire timeout for latest artifacts

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Designs

Child items ...

Activity