Skip to content

Port to reqwest, and use its connection pooling

David Laban requested to merge alsuren/cargo-info:reqwests into master

Context

I wanted to fetch the Cargo.toml for all crates on crates.io, so I got the full list of package names from https://github.com/rust-lang/crates.io-index and then used xargs and cargo info --repository to fetch the repo urls. I then had a guess where the Cargo.toml would live based on the repo url and used wget to fetch them all.

This took a while. While I was waiting, I thought "I wonder if cargo-info is re-using the ssl connection to crates.io or spinning up a new connection for each crate". It turns out you don't, so I spent my idle time writing this patch.

Performance Numbers

$ time cargo info -jv reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest > /dev/null

real	0m15.578s
user	0m0.132s
sys	0m0.045s

$ time ./target/release/cargo-info info -jv reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest reqwest > /dev/null

real	0m3.849s
user	0m0.056s
sys	0m0.023s

I don't think I've made any difference to the performance of a single query though:

$ time cargo info -jv reqwest > /dev/null

real	0m0.850s
user	0m0.035s
sys	0m0.020s

$ time  ./target/debug/cargo-info info -jv reqwest > /dev/null

real	0m0.735s
user	0m0.038s
sys	0m0.014s

See also

I didn't notice that someone has already written !6 (closed) to port to reqwest. I can rebase on top of that if you prefer.

Merge request reports