update index from any available mirror
Right now, the IndexV1Updater always checks the canonical URL first, and only fails over to the mirrors if the canonical URL fails with any error. UpdateService polls each repo with an HTTP HEAD request to see if the index has changed, and only downloads a new index file if there is a new one. This is based on the HTTP Etag as received from the server.
The problem is that Apache and nginx use separate algorithms to calculate the ETag, and the mirrors are currently a mix of Apache and nginx servers. The current ETag check just does a simple check of whether the stored ETag matches the ETag the server sent in the HTTP HEAD request. So switching to a mirror running on a different webserver will mean that the index file is downloaded again, even if it is the same file.
If we can generate the Apache and nginx ETags in fdroidclient, then it can properly check the file status on mirrors running either Apache or nginx. I found the nginx algorithm for generating the Etag, it is quite simple:
from email.utils import parsedate
import requests
r = requests.get('https://fdroid.tetaneutral.net/fdroid/repo/index-v1.jar')
last_modified = int(datetime(*parsedate(r.headers['Last-Modified'])[:6]).timestamp())
content_length = int(r.headers['Content-Length'])
etag = '"%x-%x"' % (last_modified, content_length)
I suppose another approach would be to do entirely manual configuration of the mirrors, e.g. something more like #1696 (closed). That will require lots of people to do manual configuration before it lightens the load on f-droid.org. So I think sticking with the more automated approach still makes sense.
I stuck more info in the mirror monitor to help with this: https://fdroid.gitlab.io/mirror-monitor/
@Bubu @IzzySoft @krombel FYI since you've been following the mirroring stuff.