Use streaming results instead of OFFSET/LIMIT on Geo backfilling workers

What does this MR do?

Proof of concept to streaming results on Geo backfilling workers instead of OFFSET/LIMIT.

TODO

  • Control the maximum concurrency of repository backfill for a secondary node
  • Put this feature behind a feature flag
  • Add tests

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related issues

#12565 (closed)

Edited by Michael Kozono

Merge request reports

Loading