Skip to content

Implement segmented export for dependency list export

Mehmet Emin INAC requested to merge minac_implement_scaleable_dependency_list_export into master

What does this MR do and why?

This MR implements the segmented export protocol for dependency list exports. The feature will be behind a feature flag to gradually release it.

Database review

In addition to the index introduced in this MR, there are some new queries.

Iterating over all the `sbom_occurrences` associated with the group 
SELECT
    "sbom_occurrences".*
FROM
    "sbom_occurrences"
WHERE
    "sbom_occurrences"."traversal_ids" >= '{9970}'
    AND "sbom_occurrences"."traversal_ids" < '{9971}'
    AND "sbom_occurrences"."archived" = FALSE
ORDER BY
    "sbom_occurrences"."traversal_ids" ASC,
    "sbom_occurrences"."id" ASC
LIMIT 25000

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/29630/commands/91997

Iterating through `sbom_occurrences` for an export part 
SELECT
    "sbom_occurrences".*
FROM
    "sbom_occurrences"
WHERE
    "sbom_occurrences"."traversal_ids" >= '{9970}'
    AND "sbom_occurrences"."traversal_ids" < '{9971}'
    AND ("sbom_occurrences"."traversal_ids",
        "sbom_occurrences"."id") >= ('{9970}',
        5714955234)
    AND ("sbom_occurrences"."traversal_ids",
        "sbom_occurrences"."id") <= ('{9971}',
        571495523400)
    AND "sbom_occurrences"."archived" = FALSE
ORDER BY
    "sbom_occurrences"."traversal_ids" ASC,
    "sbom_occurrences"."id" ASC
LIMIT 1000

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/29630/commands/92001

Edited by Mehmet Emin INAC

Merge request reports