Gitlab::Git::CommandTimedOut errors popping up when Indexer does force-push checking

Context

From slack:

Enabling Incremental Indexing caused an incident: #inc-3768-the-rails_request-sli-of-the-web-service-cny-stage-has-an-error-ra.

There were no errors related to the Incremental Indexing itself, but other Git operations called from Rails were running into Gitlab::Git::CommandTimedOut errors (see Sentry errors).

Looking at the duration for the RepositoryIndexWorker (see Kibana search), there was a job that took 300s, and other jobs took from 5s to 11s. Comparing this to the performance of the Zoekt Indexing Worker and the Advanced Search Indexing Worker (both running for < 1s), this is obviously not acceptable.

Possible cause

Based on this series of events:

Action	Details
Initial Indexing was ran	NO Gitlab::Git::CommandTimedOut
Incremental Indexing was merged	this did NOT add new calls to git
Force-push checking was merged	this added new calls to git from rails
Incremental Indexing was enabled	this caused Gitlab::Git::CommandTimedOut errors in other areas of the code

It's likely that the force_push? checking is what caused the slowness / timeouts.

Proposal

TBA