Skip to content

Speed up delete operations

Dmitry Gruzd requested to merge speed-up-deletions into main

What does this MR do and why?

This MR offloads delete operations to a goroutine which performs periodic cleanup. This is a corrective action from 2024-04-02: Zoekt disk usage saturation (gitlab-com/gl-infra/production#17770 - closed). During the incident we noticed that deletes are extremely slow:

image

source

The main problem is that scanning the index directory to delete one repoID at a time is very expensive. After this change is merged, we queue all deletes and actually remove index files every 10 minutes.

Profiling

image

source

Benchmarking

$ make bench
go test -bench=. -benchmem -run ^$ -benchtime=1s gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner
goos: darwin
goarch: arm64
pkg: gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner
BenchmarkMarkRepoForDeletion-10               12          92101451 ns/op          386234 B/op       6578 allocs/op
BenchmarkRemoveShardsFor-10                    1        6972158707 ns/op        621597464 B/op   7839591 allocs/op
BenchmarkRemoveFiles-10                       18          78941604 ns/op         1229045 B/op      13420 allocs/op
PASS
ok      gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner        16.809s
[Done]

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Dmitry Gruzd

Merge request reports