BuildStream doesn't retry on Git HTTPS with error 429

Summary

Noticed in tracking that if remote server returns 429 (Too Many Requests), BuildStream bails out immediately. It will not retry even with grace period. Noticeable in eg https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/526479997

Steps to reproduce

Hard to repro, you have to have a backend that returns 429. Mock server that always returns 429 might work.

What is the current bug behavior?

BuildStream fails with no retry.

What is the expected correct behavior?

BuildStream retries with eg incremental backoff jitter if no further sources available. Retry time should probably be configurable.

Relevant logs and/or screenshots

        [--:--:--] STATUS  bootstrap/build/rsync.bst-0: Running host command
            /usr/sbin/git fetch origin --prune --tags --force
        fatal: unable to access 'https://git.samba.org/rsync.git/': The requested URL returned error: 429

Possible fixes

Other relevant information