Index the diff after a force push when the last commit exists

Description

After a force push, the code indexer always ran a full re-index from the empty tree, also when the last indexed commit still exists. On an extra large repository, the full re-index stops at the timeout every time, so the repository never catches up.

The indexer compares two trees, so it does not need an ancestor. Now a full re-index runs only when the last indexed commit no longer exists. Otherwise the run indexes the changes from that commit.

The fix for the Sidekiq retries of a failed run moved to !258664 (merged).

Merge order

Merge after gitlab-elasticsearch-indexer!942 (merged) is released and GitLab uses that indexer version. A full re-index also removed some stale chunks as a side effect. That MR removes them in the diff too:

• chunks of files that became binary or empty
• chunks of paths that changed type, such as a file that became a symlink or a submodule

Known gaps

• Chunks that Elasticsearch rejected in an earlier run are not indexed again: gitlab-elasticsearch-indexer#204 (closed). This gap also exists without this MR. Before, only a successful full re-index repaired it.
• When the last indexed commit no longer exists, an extra large repository still runs a full re-index on each push.

How to set up and validate locally

In GDK with Elasticsearch:

1. Index a project and set the repository to ready.
2. Force push the default branch to a commit that is not a descendant of the indexed commit.
3. Run Ai::ActiveContext::Code::RepositoryIndexWorker.new.perform(repository.id).

The indexer uses the old commit as from_sha with force_reindex: false. The index has only the files of the new commit. If the old commit does not exist, the indexer runs a full re-index.

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/631250.