Revert catfile cache refactor

This reverts !3853 (merged) since we believe it's leading to a Goroutine leak in gitlab-com/gl-infra/production#5566 (closed).

I was able to reproduce the leak with version:

Gitaly, version 14.3.0-rc2-45-ge6374eaf6

With this revert, the leak was not there.