Skip to content

Handle HTTP 429 Responses Better

Tim Way requested to merge wayt/osinfo-db:improve-http-429-handling into main
  • Honor Retry-After headers that contain an integer
  • Back-off trying a URL starting at 1 second and double with each attempt
  • Do not handle 301 Moved-Permanently based Retry-After
  • This does not handle Retry-After with a timestamp currently and those cases will fallback to the second doubling based back-off

I encountered a number of 429 reponses when trying to run the test_urls test locally. This addresses that while keeping the normal 3 retries from Session.

I found using pytest-xdist with psutil to be particularly useful when testing URLs I plan to work on like so:

pytest -n logical --dist worksteal tests/test_urls.py

I couldn't enable it in the Makefile globally because some of the other tests have issues and I wasn't able to quickly solve the parameetize cases with sorting. I also looked at pytest-asyncio but by design it won't speed up the calls for the same test case so was moot for this.

Edited by Tim Way

Merge request reports