Skip updates for existing components and versions

Brian Williamsrequested to merge
bwill/reduce-updates-for-sbom-components-and-versions into master

What does this MR do and why?

Skip updates for existing components and versions

Currently, these records are created by upserts, which causes many needless updates that contribute to WAL volume on the database. This change fires a separate query to determine if records already exist instead. I estimate that this should result in a WAL volume reduction of several Mb/s on GitLab.com.

Fixes: https://gitlab.com/gitlab-org/gitlab/-/issues/474480

Database Queries

This MR adds two new database queries to retrieve existing records. Both these queries are covered by a unique index. These queries may contain no more than 10 OR operators, because we process in batches of 10.

Find existing components: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/30285/commands/93948

SELECT
    sbom_components.*
FROM
    sbom_components
WHERE
    sbom_components.component_type = 0 AND
    (
        (
            sbom_components.name = 'rails' AND
            sbom_components.purl_type = 3
        ) OR
        (
            sbom_components.name = '@types/babel-types' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = '@types/babylon' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'accepts' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'acorn' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'acorn-globals' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'align-text' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'amdefine' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'array-flatten' AND
            sbom_components.purl_type = 6
        ) OR
        (
            sbom_components.name = 'asap' AND
            sbom_components.purl_type = 6
        )
    );

Find existing versions: https://console.postgres.ai/gitlab/gitlab-production-main/sessions/30285/commands/93949

SELECT
    sbom_component_versions.*
FROM
    sbom_component_versions
WHERE
    sbom_component_versions.component_id = 154 AND
    (
        sbom_component_versions.version = '8.12.1' OR
        sbom_component_versions.version = '8.12.0' OR
        sbom_component_versions.version = '8.11.3' OR
        sbom_component_versions.version = '8.11.1' OR
        sbom_component_versions.version = '8.11.0' OR
        sbom_component_versions.version = '8.9.0' OR
        sbom_component_versions.version = '8.8.2' OR
        sbom_component_versions.version = '8.8.1' OR
        sbom_component_versions.version = '8.8.0' OR
        sbom_component_versions.version = '8.7.1'
    );

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Brian Williams

Merge request reports