Skip to content

Add updated_at index to improve Merge Request API performance

What does this MR do?

This MR is an attempt to improve the performance issue that was discussed in #336545 (closed).

According to the requests I've inspected via https://log.gprd.gitlab.net/goto/7020d2f63c45790ccdda417a09a3900e, the customer utilizes updated_after with a relatively recent timestamp. It seems to be that they query a week worth of data. It seems that we don't have index on updated_at to perform that query efficiently.

Database

Example query

SELECT
  "merge_requests".*
FROM
  "merge_requests"
  INNER JOIN "projects" ON "projects"."id" = "merge_requests"."target_project_id"
  LEFT JOIN project_features ON projects.id = project_features.project_id
WHERE
  "projects"."namespace_id" IN (
    SELECT
      "namespaces"."id"
    FROM
      "namespaces"
    WHERE
      (traversal_ids @ > ('{4249178}'))
  )
  AND (
    EXISTS (
      SELECT
        1
      FROM
        "project_authorizations"
      WHERE
        "project_authorizations"."user_id" = 3880924
        AND (project_authorizations.project_id = projects.id)
        AND (project_authorizations.access_level >= 20)
    )
    OR projects.visibility_level IN (10, 20)
  )
  AND (
    "project_features"."merge_requests_access_level" IS NULL
    OR "project_features"."merge_requests_access_level" IN (20, 30)
    OR (
      "project_features"."merge_requests_access_level" = 10
      AND EXISTS (
        SELECT
          1
        FROM
          "project_authorizations"
        WHERE
          "project_authorizations"."user_id" = 3880924
          AND (project_authorizations.project_id = projects.id)
          AND (project_authorizations.access_level >= 20)
      )
    )
  )
  AND "merge_requests"."updated_at" >= '2021-07-13 18:33:34'
  AND "projects"."archived" = FALSE
ORDER BY
  "merge_requests"."created_at" ASC,
  "merge_requests"."id" ASC
LIMIT
  100 OFFSET 1400

Before

1st execution (straight after starting a new database.ai session)

Explain: https://explain.depesz.com/s/EoKj

Time: 1.088 min

3rd execution

Explain: https://explain.depesz.com/s/cAWT

Time: 5.362 s

After

1st execution (straight after starting a new database.ai session)

Explain: https://explain.depesz.com/s/657Y

Time: 34.425 s

3rd execution

Explain: https://explain.depesz.com/s/9rby

Time: 400.536 ms

Index Creation: Duration: 6.950 min

CREATE INDEX index_merge_requests_on_target_project_id_and_updated_at_and_id ON merge_requests USING btree (target_project_id, updated_at, id)

Up

== 20210722055217 AddUpdatedAtIndexOnMergeRequests: migrating =================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:merge_requests, [:target_project_id, :updated_at, :id], {:name=>"index_merge_requests_on_target_project_id_and_updated_at_and_id", :algorithm=>:concurrently})
   -> 0.0151s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:merge_requests, [:target_project_id, :updated_at, :id], {:name=>"index_merge_requests_on_target_project_id_and_updated_at_and_id", :algorithm=>:concurrently})
   -> 0.0206s
-- execute("RESET ALL")
   -> 0.0016s
== 20210722055217 AddUpdatedAtIndexOnMergeRequests: migrated (0.0388s) ========

Down

== 20210722055217 AddUpdatedAtIndexOnMergeRequests: reverting =================
-- transaction_open?()
   -> 0.0000s
-- indexes(:merge_requests)
   -> 0.0184s
-- execute("SET statement_timeout TO 0")
   -> 0.0007s
-- remove_index(:merge_requests, {:algorithm=>:concurrently, :name=>"index_merge_requests_on_target_project_id_and_updated_at_and_id"})
   -> 0.0034s
-- execute("RESET ALL")
   -> 0.0007s
== 20210722055217 AddUpdatedAtIndexOnMergeRequests: reverted (0.0245s) ========

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #336545 (closed)

Edited by Dylan Griffith

Merge request reports