How do we keep the main DB responsive during the first import from License DB?

Topic to Evaluate

On GitLab instances that import data from License DB for the first time, PackageMetadata::SyncWorker inserts millions of row into the main Postgres database using INSERT queries. This causes important performance issues.

  • This heavily degrades the initial user experience.
  • It makes QA tests fail due to performance slowdowns in the main DB. #396649 (comment 1319465309)

We need to evaluate options to solve this performance problem.

Tasks to Evaluate

For each option considered in this issue:

  • Check feasibility
  • Check compatibility with all supported deployments
  • Assess cost of implementation
  • Measure performance gain

Then,

  • Compare options, and choose a first step
  • Create implementation issues

Risks and Implementation Considerations

Team

/cc @gonzoyumo @sam.white @brytannia @stanhu

Auto-Summary 🤖

Discoto Usage

Points

Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive) point:. For example, the following are all valid points:

  • #### POINT: This is a point
  • * point: This is a point
  • + Point: This is a point
  • - pOINT: This is a point
  • point: This is a **point**

Note that any markdown used in the point text will also be propagated into the topic summaries.

Topics

Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.

Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive) topic:. For example, the following are all valid topics:

  • # Topic: Inline discussion topic 1
  • ## TOPIC: **{+A Green, bolded topic+}**
  • ### tOpIc: Another topic

Quick Actions

Action Description
/discuss sub-topic TITLE Create an issue for a sub-topic. Does not work in epics
/discuss link ISSUABLE-LINK Link an issuable as a child of this discussion

Last updated by this job

Discoto Settings 
---
summary:
  max_items: -1
  sort_by: created
  sort_direction: ascending

See the settings schema for details.

Implementation Plan

Edited by Lucas Charles