gitlab-zoekt-indexer has an unbounded memory usage problem

See gitlab-com/gl-infra/production#16090 (comment 1494684690) . When trying to index the project https://gitlab.com/gitlab-org/frontend/playground/webpack-bundle-sizes we see memory grow continuously until K8s just OOM kills it.

This project contains tonnes of small text files that amount to a total of 18GiB on disk storage so it's not suprising it takes a while to index. But it seems the indexer writes out small tmp shard files while it's running in 280MiB chunks. The indexer should also be capable of clearing out memory as it goes with indexing in batches. This would allow us to index arbitrarily large repos. Instead it seems to just keep everything in memory so perhaps we have a design issue here where we aren't freeing up memory as we batch through a project.