Makefile: allow proxying buf's BSR traffic via BUF_PROXY

Bitol Mayarequested to merge
kukulkan/buf-proxy-passthrough into develop

Problem

buf.build serves HTTP 403 at their edge to the Finland CI runner's egress IP (185.100.86.150). Every job that resolves BSR dependencies dies on the same line:

Failure: 403 Forbidden

This takes out generate, proto-lint and proto-check-breaking. Since that runner is the only one online for this project, it takes out 100% of protobuf CI.

Evidence

Reproduced directly on the runner, independent of our protos: a throwaway buf.yaml with a single dep plus buf mod update fails identically (buf 1.20.0).

  • It discriminates on source IP. The same request from three other IPs returns 200, all hitting the same anycast edge (34.95.71.106) with the same client.
  • The 403 is generated before it reaches buf's application: their app answers application/json and stamps x-ratelimit-* headers, while the 403 is a 134-byte text/html page with none of them.
  • It is specific to buf.build. From that same runner, google.com, cloud.google.com, github.com and cloudflare.com all return 200.
  • It is not the documented BSR rate limiter, which returns 429 with Retry-After.

Why buf blocks that IP is still unconfirmed, and is being chased separately with them.

Change

One optional variable. When BUF_PROXY is set, it is forwarded into the proto-builder container as HTTPS_PROXY/HTTP_PROXY. When unset, which is the default, nothing changes: local development and unaffected runners behave exactly as before.

The proxy itself lives entirely in runner infrastructure, not in this repo. On the Finland runner a systemd unit holds an outbound SSH tunnel to a host with clean egress and exposes SOCKS5 on the docker bridge. No new public listener is exposed on either machine, and the remote account is shell-less with restrict,port-forwarding. BUF_PROXY is set as a project CI/CD variable pointing at it.

Note for anyone reusing this: socks5h:// does not work, buf rejects it with Are you sure "socks5h" is a valid remote address?. Use socks5://.

Validation

On a scratch branch with this Makefile change and BUF_PROXY set, all three affected targets pass on the blocked runner:

job result
proto-lint success
proto-gen success, 159 .proto files processed
proto-check-breaking success

Before the change, proto-gen died on the very first file (bifrost/p2p/messages/signature_notifier.proto).

Control on the runner host, with the real buf:

with proxy:     buf mod update -> exit=0, buf.lock generated
without proxy:  Failure: 403 Forbidden -> exit=1

Follow-ups, not in this MR

  • This is a workaround for an external block, not a fix for the cause. The durable fix is to vendor the five buf.lock deps so CI never depends on buf.build. Worth doing regardless of how the block resolves.
  • Setting BUF_TOKEN is worth doing on its own merits, since every BSR call from CI is anonymous today. It may also be sufficient by itself: buf counts unauthenticated requests per source IP and authenticated ones per user.
  • The same runner also serves node-launcher and thornode-private, so those pipelines are hit by the same block and need the same treatment.

Merge request reports