Set p_ci_build_trace_metadata to partitioned with detach_archived: true

What to do

Define the p_ci_build_trace_metadata table with this option:

partitioned: { detach_archived: true }

This lets GitLab detach a partition when it archives the pipeline. GitLab does not delete each record one by one.

Why

The p_ci_build_trace_metadata table grows too fast on GitLab.com. The capacity planning team found this growth in gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com#3071. The database size Rapid Action tracks this work in gitlab-org&16520.

The table holds processing data. GitLab uses it to check that it archives each trace correctly. The code processes the data and then deletes it after the upload to Object Storage. After the upload, no code uses the data.

The growth pattern is not normal. There can be two causes:

  • GitLab makes many bad uploads to Object Storage.
  • GitLab does not erase some records.

The table is already partitioned. GitLab does not need this data after the upload. Therefore, GitLab can keep the records until it archives the pipeline. Then GitLab can truncate the whole partition. This step saves database capacity.

Related

Edited by 🤖 GitLab Bot 🤖