Skip to content

Prepare async index for project vulnerability_reads on common filters

Mehmet Emin INAC requested to merge minac_413697_index_vulnerability_reads into master

What does this MR do and why?

This MR creates an index for the vulnerability_reads table on project_id, state, report_type, severity, and vulnerability_id columns asynchronously to fix a timing out query.

The original query will be overridden by the UnnestedInFilters abstraction automatically once we introduce this index as the index can be utilized for the specialized query.

Related to #413697 (closed)

Problemmatic query
SELECT
    "vulnerability_reads".*
FROM
    "vulnerability_reads"
WHERE
    "vulnerability_reads"."project_id" = 278964
    AND "vulnerability_reads"."report_type" IN (6, 2, 5, 3, 1, 0, 4, 99)
    AND "vulnerability_reads"."severity" = 4
    AND "vulnerability_reads"."state" IN (1, 4)
ORDER BY
    "vulnerability_reads"."severity" DESC,
    "vulnerability_reads"."vulnerability_id" DESC
LIMIT 21
New query after the index
SELECT
    "vulnerability_reads".*
FROM
    "vulnerability_reads"
WHERE
    "vulnerability_reads"."vulnerability_id" IN (
        SELECT
            "vulnerability_reads"."vulnerability_id"
        FROM
            unnest('{6,2,5,3,1,0,4,99}'::smallint[]) AS "report_types" ("report_type"),
            unnest('{1,4}'::smallint[]) AS "states" ("state"),
            LATERAL (
                SELECT
                    "vulnerability_reads"."project_id",
                    "vulnerability_reads"."report_type",
                    "vulnerability_reads"."severity",
                    "vulnerability_reads"."state",
                    "vulnerability_reads"."vulnerability_id"
                FROM
                    "vulnerability_reads"
                WHERE
                    "vulnerability_reads"."severity" = 4
                    AND "vulnerability_reads"."project_id" = 278964
                    AND (vulnerability_reads."report_type" = "report_types"."report_type")
                    AND (vulnerability_reads."state" = "states"."state")
                ORDER BY
                    "vulnerability_reads"."severity" DESC,
                    "vulnerability_reads"."vulnerability_id" DESC
                LIMIT 21) AS vulnerability_reads
        ORDER BY
            "vulnerability_reads"."severity" DESC,
            "vulnerability_reads"."vulnerability_id" DESC
        LIMIT 21)
ORDER BY
    "vulnerability_reads"."severity" DESC,
    "vulnerability_reads"."vulnerability_id" DESC
LIMIT 21

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19183/commands/63327

rake command outputs
up
main: == [advisory_lock_connection] object_id: 228420, pg_backend_pid: 55811
main: == 20230619134106 PrepareIndexForVulnerabilityReadsOnCommonProjectFilters: migrating
main: -- index_exists?(:vulnerability_reads, [:project_id, :state, :report_type, :severity, :vulnerability_id], {:order=>{:severity=>:desc, :vulnerability_id=>:desc}, :name=>"index_project_vulnerability_reads_common_finder_query_desc", :algorithm=>:concurrently})
main:    -> 0.0156s
main: -- add_index_options(:vulnerability_reads, [:project_id, :state, :report_type, :severity, :vulnerability_id], {:order=>{:severity=>:desc, :vulnerability_id=>:desc}, :name=>"index_project_vulnerability_reads_common_finder_query_desc", :algorithm=>:concurrently})
main:    -> 0.0006s
main: == 20230619134106 PrepareIndexForVulnerabilityReadsOnCommonProjectFilters: migrated (0.1872s)

main: == [advisory_lock_connection] object_id: 228420, pg_backend_pid: 55811
down
main: == [advisory_lock_connection] object_id: 228480, pg_backend_pid: 55274
main: == 20230619134106 PrepareIndexForVulnerabilityReadsOnCommonProjectFilters: reverting
main: == 20230619134106 PrepareIndexForVulnerabilityReadsOnCommonProjectFilters: reverted (0.1316s)

main: == [advisory_lock_connection] object_id: 228480, pg_backend_pid: 55274

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Thiago Figueiró

Merge request reports