Cannot pick a target / .cargo/config.toml target is ignored

on my repo I have

repo/testbins/vsock-bin$ cat .cargo/config.toml 
[build]
target = "x86_64-unknown-linux-musl"

but this gets ignored by test-binary, and it's not possible to select a target with TestBinary either