Speed up delete operations
What does this MR do and why?
This MR offloads delete operations to a goroutine which performs periodic cleanup. This is a corrective action from 2024-04-02: Zoekt disk usage saturation (gitlab-com/gl-infra/production#17770 - closed). During the incident we noticed that deletes are extremely slow:
The main problem is that scanning the index directory to delete one repoID at a time is very expensive. After this change is merged, we queue all deletes and actually remove index files every 10 minutes.
Profiling
Benchmarking
$ make bench
go test -bench=. -benchmem -run ^$ -benchtime=1s gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner
goos: darwin
goarch: arm64
pkg: gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner
BenchmarkMarkRepoForDeletion-10 12 92101451 ns/op 386234 B/op 6578 allocs/op
BenchmarkRemoveShardsFor-10 1 6972158707 ns/op 621597464 B/op 7839591 allocs/op
BenchmarkRemoveFiles-10 18 78941604 ns/op 1229045 B/op 13420 allocs/op
PASS
ok gitlab.com/gitlab-org/gitlab-zoekt-indexer/internal/file_cleaner 16.809s
[Done]
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Edited by Dmitry Gruzd