Load only needed snippet columns when backing up repositories
What does this MR do and why?
gitlab-backup create loads every column of every snippet while it enumerates snippet repositories, including content. Single-file snippet content is still stored in snippets.content, up to snippet_size_limit (50 MB by default) per row. On instances with many large snippets, one batch of 1,000 rows can hold tens of GB, and the backup process gets OOM-killed. This MR selects only id, type, and project_id, which is all the backup needs to locate each snippet repository.
Query before:
SELECT "snippets".* FROM "snippets" ORDER BY "snippets"."id" ASC LIMIT 1000Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/64808/commands/169775
Query after:
SELECT "snippets"."id", "snippets"."type", "snippets"."project_id" FROM "snippets" ORDER BY "snippets"."id" ASC LIMIT 1000Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/64808/commands/169776
The filters, ordering, and limit don't change, so the query plan doesn't change either.
The code is the same on all supported stable branches, so this fix is a backport candidate.
References
- Resolves: #606539 (closed)
Screenshots or screen recordings
This is a backend-only change, so screenshots are not applicable.
How to set up and validate locally
-
Seed 1,000 personal snippets with 2 MB of content each. The content compresses well, so this uses about 24 MB of disk:
-- gdk psql INSERT INTO snippets (title, content, file_name, author_id, organization_id, type, visibility_level, created_at, updated_at) SELECT 'oom-repro-' || g, repeat(md5(g::text), 2 * 1024 * 1024 / 32), 'big.json', (SELECT id FROM users WHERE username = 'root'), (SELECT id FROM organizations ORDER BY id LIMIT 1), 'PersonalSnippet', 0, now(), now() FROM generate_series(1, 1000) g; -
Run only the snippet enumeration and print the peak RSS, on
masterand on this branch:bin/rails runner ' rss = -> { `ps -o rss= -p #{Process.pid}`.to_i / 1024 } peak = rss.call Thread.new { loop { peak = [peak, rss.call].max; sleep 0.05 } } strategy = Object.new def strategy.enqueue(container, type) = type.repository_for(container).relative_path Backup::Targets::Repositories.new($stdout, strategy: strategy, options: Backup::Options.new) .send(:enqueue_consecutive_snippets) puts "peak RSS: #{peak} MB" 'Expected result: on
masterthe peak RSS is roughly 2 GB higher than on this branch, because the batch loads the content of all 1,000 snippets. -
Remove the seeded snippets:
DELETE FROM snippets WHERE title LIKE 'oom-repro-%';
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.