Geo creates larger repos in secondary

Summary

While on a call with a customer we saw that the Geo secondary node was returning a 502. When we checked the primary instance Sidekiq had about 940k background jobs in the process_commitqueue, about 3k in system_hookand about 40 in geo_repository_update. Most (95%) of these jobs where regarding the same project. The system_hook and some other process_commit jobs where failing and reporting FIPS integrity verification test failed.

This was about 10hrs after the project was initially pushed to primary. The project is around 4GB and the secondary node ran out of disk space because the same project had taken up around 187GB already.

Steps to reproduce

It might be specific to this case but all that was required to reproduced was to push the same project to primary. This behavior was reproduced on a clean secondary node.

Expected behavior

Secondary node should reflect the same project as primary.

Actual behavior

Secondary node is stuck

Relevant logs and/or screenshots

Pending

Results of GitLab application Check

I don't have this available but did check status on our call and instance is healthy.

Results of GitLab environment info

Pending

Possible fixes

@brodock mentioned this on a previous conversation Perhaps it's not pruning or requires GC run?

Links

Slack conversation https://gitlab.slack.com/archives/support/p1482781613003455

---

/cc @brodock