fix(df): stream large binary downloads safely
What does this MR do?
The dependency firewall proxy decided whether to stream a response straight to the client or buffer-and-rewrite it in isBinaryDownload, which only streamed application/octet-stream or .tgz. A pip/uv wheel from files.pythonhosted.org is served as binary/octet-stream with a .whl path, so it took the buffered path (io.ReadAll of the whole body). On the buffered path time-to-first-byte equals the entire download, so large wheels (for example botocore's ~16 MB) tripped the client's read timeout and packages failed to download.
This renames the check to isStreamable: stream any recognized binary artifact (by extension or Content-Type) that is self-delimiting (known Content-Length and not transport-decompressed); everything else - metadata/index responses and the PEP 658 .whl.metadata sidecar - takes the buffer-and-rewrite path that repairs framing after the transport's automatic decompression.
It also closes the same gap for Maven: mavenPrimaryExts recognizes .jar, .pom, .aar and .war, but only .jar was covered and repo1.maven.org serves these as application/java-archive, which was absent from the content-type set. A 29.7 MB solr-4.10.4.war is larger than the wheel that already broke pip, and glab df maven/glab df gradle reach it today. .pom deliberately stays on the buffered path as small XML metadata.
The broadened set also covers .crate, .gem, .nupkg, .zip and gzip content-types, fixing the same stall class across ecosystems that ship binary artifacts, not just PyPI.
Scope
Minimal fix extracted from the feat/dependency-firewall integration branch (commit f1db553cd), deliberately excluding the cargo sparse-index matcher half of that commit since there is no glab df cargo command on main yet.
Testing
Verified end-to-end against a GDK with an enforced dependency firewall policy:
main |
this branch | |
|---|---|---|
glab df pip install boto3 |
ReadTimeoutError: Read timed out after 18s |
16 MB wheel at 93 MB/s, 4.3s |
glab df uv pip install boto3 |
- | 1.8s |
Measured time-to-first-byte through the real proxy for a 4 MiB artifact on a throttled upstream: a buffered .war returned its first byte at 559 ms (the full download time) before this change and 59 ms after, with a byte-complete body.
TestIsStreamable covers every binaryArtifactExts and binaryContentTypes entry plus the buffered cases (unknown length, transport-decompressed, JSON/HTML metadata, non-2xx).