Skip to content

Stop using ci_builds_metadata.has_exposed_artifacts to find exposed artifacts

Leaminn Marequested to merge
stop-using-has-exposed-artifacts-metadata into master

What does this MR do and why?

Context

A CI pipeline can be configured with "exposed artifacts" using the artifacts:expose_as keyword. These artifacts are listed as file links in the merge request UI with the name set to the expose_as value.

We currently use the column p_ci_builds_metadata.has_exposed_artifacts in the scope .with_exposed_artifacts to find jobs with exposed artifacts in FindExposedArtifactsService. This service iterates through each job where has_exposed_artifacts = true and fetches the applicable artifacts.

As part of our plan to migrate data away from the metadata table (https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/epics/36), we need update the code so it's no longer dependent on the has_exposed_artifacts column.

This MR

This MR updates the logic for finding exposed artifacts so it doesn't use the has_exposed_artifacts column. This means that we have to iterate through all latest builds of a pipelines and check each one for exposed artifacts.

This change is made behind a feature flag: ci_stop_using_has_exposed_artifacts_metadata_col

References

Database query plans

Batching queries

The following are the batching queries on p_ci_builds generated from this code:

pipeline.builds.latest.each_batch

Query 1

Raw SQL 
SELECT
    "p_ci_builds"."id"
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
ORDER BY
    "p_ci_builds"."id" ASC
LIMIT
    1;

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/40959/commands/125878

Query 2

Raw SQL 
SELECT
    "p_ci_builds"."id"
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
    AND "p_ci_builds"."id" >= 1
ORDER BY
    "p_ci_builds"."id" ASC
LIMIT
    1 OFFSET 1000;

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/40959/commands/125879

  • Returns zero rows because this pipeline I'm querying has fewer than 1000 jobs

Query 3

Raw SQL 
SELECT "p_ci_builds"."status", "p_ci_builds"."finished_at", "p_ci_builds"."created_at", "p_ci_builds"."updated_at", "p_ci_builds"."started_at", "p_ci_builds"."coverage", "p_ci_builds"."name", "p_ci_builds"."options", "p_ci_builds"."allow_failure", "p_ci_builds"."stage_idx", "p_ci_builds"."tag", "p_ci_builds"."ref", "p_ci_builds"."type", "p_ci_builds"."target_url", "p_ci_builds"."description", "p_ci_builds"."erased_at", "p_ci_builds"."artifacts_expire_at", "p_ci_builds"."environment", "p_ci_builds"."when", "p_ci_builds"."yaml_variables", "p_ci_builds"."queued_at", "p_ci_builds"."lock_version", "p_ci_builds"."coverage_regex", "p_ci_builds"."retried", "p_ci_builds"."protected", "p_ci_builds"."failure_reason", "p_ci_builds"."scheduled_at", "p_ci_builds"."token_encrypted", "p_ci_builds"."resource_group_id", "p_ci_builds"."waiting_for_resource_at", "p_ci_builds"."processed", "p_ci_builds"."scheduling_type", "p_ci_builds"."id", "p_ci_builds"."stage_id", "p_ci_builds"."partition_id", "p_ci_builds"."auto_canceled_by_partition_id", "p_ci_builds"."auto_canceled_by_id", "p_ci_builds"."commit_id", "p_ci_builds"."erased_by_id", "p_ci_builds"."project_id", "p_ci_builds"."runner_id", "p_ci_builds"."upstream_pipeline_id", "p_ci_builds"."user_id", "p_ci_builds"."execution_config_id", "p_ci_builds"."upstream_pipeline_partition_id" 
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
    AND "p_ci_builds"."id" >= 10486918170
    AND "p_ci_builds"."id" < 10486948170

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/40959/commands/125884

.any_with_exposed_artifacts?

Below are the queries executed by builds.latest.any_with_exposed_artifacts?.

Query 1

Raw SQL 
SELECT
    "p_ci_builds"."id"
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
ORDER BY
    "p_ci_builds"."id" ASC
LIMIT
    1

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/41189/commands/126584

Query 2

Raw SQL 
SELECT
    "p_ci_builds"."id"
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
    AND "p_ci_builds"."id" >= 1242
ORDER BY
    "p_ci_builds"."id" ASC
LIMIT
    1 OFFSET 1000

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/41189/commands/126589

Query 3

Raw SQL 
SELECT
    "p_ci_builds"."id",
    "p_ci_builds"."partition_id",
    "p_ci_builds"."project_id",
    "p_ci_builds"."options"
FROM
    "p_ci_builds"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "p_ci_builds"."commit_id" = 1892868760
    AND "p_ci_builds"."partition_id" = 104
    AND (
        "p_ci_builds"."retried" = FALSE
        OR "p_ci_builds"."retried" IS NULL
    )
    AND "p_ci_builds"."id" >= 10486918170
    AND "p_ci_builds"."id" < 10486948170

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/41189/commands/126590

Query 4

Raw SQL 
SELECT "projects"."id", "projects"."name", "projects"."path", "projects"."description", "projects"."created_at", "projects"."updated_at", "projects"."creator_id", "projects"."namespace_id", "projects"."last_activity_at", "projects"."import_url", "projects"."visibility_level", "projects"."archived", "projects"."avatar", "projects"."merge_requests_template", "projects"."star_count", "projects"."merge_requests_rebase_enabled", "projects"."import_type", "projects"."import_source", "projects"."approvals_before_merge", "projects"."reset_approvals_on_push", "projects"."merge_requests_ff_only_enabled", "projects"."issues_template", "projects"."mirror", "projects"."mirror_last_update_at", "projects"."mirror_last_successful_update_at", "projects"."mirror_user_id", "projects"."shared_runners_enabled", "projects"."runners_token", "projects"."build_allow_git_fetch", "projects"."build_timeout", "projects"."mirror_trigger_builds", "projects"."pending_delete", "projects"."public_builds", "projects"."last_repository_check_failed", "projects"."last_repository_check_at", "projects"."only_allow_merge_if_pipeline_succeeds", "projects"."has_external_issue_tracker", "projects"."repository_storage", "projects"."repository_read_only", "projects"."request_access_enabled", "projects"."has_external_wiki", "projects"."ci_config_path", "projects"."lfs_enabled", "projects"."description_html", "projects"."only_allow_merge_if_all_discussions_are_resolved", "projects"."repository_size_limit", "projects"."printing_merge_request_link_enabled", "projects"."auto_cancel_pending_pipelines", "projects"."service_desk_enabled", "projects"."cached_markdown_version", "projects"."delete_error", "projects"."last_repository_updated_at", "projects"."disable_overriding_approvers_per_merge_request", "projects"."storage_version", "projects"."resolve_outdated_diff_discussions", "projects"."remote_mirror_available_overridden", "projects"."only_mirror_protected_branches", "projects"."pull_mirror_available_overridden", "projects"."jobs_cache_index", "projects"."external_authorization_classification_label", "projects"."mirror_overwrites_diverged_branches", "projects"."pages_https_only", "projects"."external_webhook_token", "projects"."packages_enabled", "projects"."merge_requests_author_approval", "projects"."pool_repository_id", "projects"."runners_token_encrypted", "projects"."bfg_object_map", "projects"."detected_repository_languages", "projects"."merge_requests_disable_committers_approval", "projects"."require_password_to_approve", "projects"."max_pages_size", "projects"."max_artifacts_size", "projects"."pull_mirror_branch_prefix", "projects"."remove_source_branch_after_merge", "projects"."marked_for_deletion_at", "projects"."marked_for_deletion_by_user_id", "projects"."autoclose_referenced_issues", "projects"."suggestion_commit_message", "projects"."project_namespace_id", "projects"."hidden", "projects"."organization_id"
FROM "projects"
WHERE "projects"."id" = 278964

Query plan: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/41190/commands/126591

Query 5

Raw SQL 
SELECT
    "p_ci_builds_metadata"."build_id",
    "p_ci_builds_metadata"."partition_id",
    "p_ci_builds_metadata"."config_options"
FROM
    "p_ci_builds_metadata"
WHERE
    "p_ci_builds_metadata"."partition_id" = 100
    AND "p_ci_builds_metadata"."build_id" IN (1243, 1244, 1242)

Query plan: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/41189/commands/126592

How to set up and validate locally

These steps will lead you through creating an exposed artifact and verifying that the feature still works as before.

Prerequisite: Ensure you have a GitLab Runner set up with your gdk.

  1. Enable the feature flag in Rails console: Feature.enable(:ci_stop_using_has_exposed_artifacts_metadata_col)

  2. Create a new blank project with the following pipeline config:

.gitlab-ci.yml

generate-exposed-artifact:
  script:
    - echo "My test artifact" > my_artifact.txt
  artifacts:
    paths:
      - my_artifact.txt
    expose_as: 'My Test Artifact'

control-job:
  script: echo
  1. Commit (anything) to a new branch and open a merge request. In that MR, wait for the pipeline to complete. Then observe that the artifact is correctly exposed in the UI:

Screenshot_2025-06-26_at_8.25.50_PM

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #550223 (closed)

Edited by Leaminn Ma

Merge request reports