Add a GitLab Geo download scheduler for LFS files
The download scheduler works as follows:
- Load a batch of IDs that we need to download from the primary (DB_RETRIEVE_BATCH) into a pending list.
- Schedule them so that at most MAX_CONCURRENT_DOWNLOADS are running at once.
- When a slot frees, schedule another download.
- When we have drained the pending list, load another batch into memory, and schedule the remaining files, excluding ones in progress.
- Quit when we have scheduled all downloads or exceeded an hour.
Originally from !1359 (closed)