Load only needed snippet columns when backing up repositories

What does this MR do and why?

gitlab-backup create loads every column of every snippet while it enumerates snippet repositories, including content. Single-file snippet content is still stored in snippets.content, up to snippet_size_limit (50 MB by default) per row. On instances with many large snippets, one batch of 1,000 rows can hold tens of GB, and the backup process gets OOM-killed. This MR selects only id, type, and project_id, which is all the backup needs to locate each snippet repository.

Query before:

SELECT "snippets".* FROM "snippets" ORDER BY "snippets"."id" ASC LIMIT 1000

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/64808/commands/169775

Query after:

SELECT "snippets"."id", "snippets"."type", "snippets"."project_id" FROM "snippets" ORDER BY "snippets"."id" ASC LIMIT 1000

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/64808/commands/169776

The filters, ordering, and limit don't change, so the query plan doesn't change either.

The code is the same on all supported stable branches, so this fix is a backport candidate.

References

Screenshots or screen recordings

This is a backend-only change, so screenshots are not applicable.

How to set up and validate locally

  1. Seed 1,000 personal snippets with 2 MB of content each. The content compresses well, so this uses about 24 MB of disk:

    -- gdk psql
    INSERT INTO snippets (title, content, file_name, author_id, organization_id, type, visibility_level, created_at, updated_at)
    SELECT 'oom-repro-' || g, repeat(md5(g::text), 2 * 1024 * 1024 / 32), 'big.json',
           (SELECT id FROM users WHERE username = 'root'),
           (SELECT id FROM organizations ORDER BY id LIMIT 1),
           'PersonalSnippet', 0, now(), now()
    FROM generate_series(1, 1000) g;
  2. Run only the snippet enumeration and print the peak RSS, on master and on this branch:

    bin/rails runner '
      rss = -> { `ps -o rss= -p #{Process.pid}`.to_i / 1024 }
      peak = rss.call
      Thread.new { loop { peak = [peak, rss.call].max; sleep 0.05 } }
      strategy = Object.new
      def strategy.enqueue(container, type) = type.repository_for(container).relative_path
      Backup::Targets::Repositories.new($stdout, strategy: strategy, options: Backup::Options.new)
        .send(:enqueue_consecutive_snippets)
      puts "peak RSS: #{peak} MB"
    '

    Expected result: on master the peak RSS is roughly 2 GB higher than on this branch, because the batch loads the content of all 1,000 snippets.

  3. Remove the seeded snippets:

    DELETE FROM snippets WHERE title LIKE 'oom-repro-%';

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Chloe Fons

Merge request reports

Loading
Loading