Skip to content

Add check-urls to check for broken URLs

Benjamin Drung requested to merge bdrung1/osinfo-db:check-urls into master

URLs to ISOs can become broken (e.g. when Ubuntu releases a new point release image). Add scripts/check-urls for checking that all URLs are not returning HTTP 4xx.

Example run:

$ scripts/check-urls 
scripts/check-urls 
2022-08-12 15:11:53,928 INFO: Checking 1728 URLs from 865 files...
2022-08-12 15:13:54,807 ERROR: URL http://mirror-master.dragonflybsd.org/iso-images/dfly-x86_64-4.8.0_REL.iso.bz2 gave status code 404.
2022-08-12 15:13:55,136 ERROR: URL http://mirror-master.dragonflybsd.org/iso-images/dfly-x86_64-4.8.1_REL.iso.bz2 gave status code 404.
2022-08-12 15:19:33,616 INFO: Checked 1728 URLs. 2 URLs had issues.

You can limit the URLs to check. Example (before !497 (merged) got merged):

$ scripts/check-urls -i ubuntu 
2022-08-12 13:56:22,511 INFO: Checking 270 URLs from 40 files...
2022-08-12 13:56:50,892 ERROR: URL https://cdimage.ubuntu.com/ubuntu/releases/22.04/release/ubuntu-22.04-live-server-arm64.iso gave status code 404.
2022-08-12 13:56:51,238 ERROR: URL https://cdimage.ubuntu.com/ubuntu/releases/22.04/release/ubuntu-22.04-live-server-ppc64el.iso gave status code 404.
2022-08-12 13:56:51,586 ERROR: URL https://cdimage.ubuntu.com/ubuntu/releases/22.04/release/ubuntu-22.04-live-server-s390x.iso gave status code 404.
2022-08-12 13:57:02,586 ERROR: URL https://releases.ubuntu.com/22.04/ubuntu-22.04-desktop-amd64.iso gave status code 404.
2022-08-12 13:57:02,931 ERROR: URL https://releases.ubuntu.com/22.04/ubuntu-22.04-live-server-amd64.iso gave status code 404.
2022-08-12 13:57:03,941 INFO: Checked 270 URLs. 5 URLs had issues.
Edited by Benjamin Drung

Merge request reports