Skip to content

Support specifying a default compression algorithm for gRPC transport compression

Jeremiah Bonney requested to merge jbonney/transport-compression into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

This PR adds support for configuring gRPC compression, which is applied transparently to outgoing requests. It defaults to "NoCompression" and will apply to all responses unless a particular RPC overrides it (none do currently as this is a brand new feature).

I'm still working on some tests/example config/enabling compression for the client channels, so opening this up in draft. I've added compression in the docker-compose.yaml for the storage service to allow testing and demoing of the functionality. I'll add compression support for the various remote storage/AC in a separate MR since this one is nice and self-contained.

Validation

Specify compression to either Deflate or Gzip in a BuildGrid config and start up a server with GRPC_TRACE=compression and GRPC_VERBOSITY=debug set in the environment. When sending requests you should see in the log output that compression is enabled and how much space it saved (or whether it decided not to compress to begin with).

With compression you should see something like the following output:

storage_1         | 2023-06-23 21:47:09,926:[58014dc0-8b82-442b-945e-9885bbb9e5f8][   buildgrid.server.cas.storage.disk][DEBUG][gRPC_Executor_0]: Getting blob: [hash: "140ed3e3f35c4d1aae2079f112edfebc72a68a8094f34cff6cd3ea28bba09c91"
storage_1         | size_bytes: 24397651
storage_1         | ]
storage_1         | E0623 21:47:09.928760997      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:09.979100087      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 418826 bytes (60.06% savings)
storage_1         | E0623 21:47:09.981270643      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.026479497      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 451976 bytes (56.90% savings)
storage_1         | E0623 21:47:10.027526872      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.082754138      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 550177 bytes (47.53% savings)
storage_1         | E0623 21:47:10.084066524      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.139570699      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 634930 bytes (39.45% savings)
storage_1         | E0623 21:47:10.142522171      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.203238105      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 606750 bytes (42.14% savings)
storage_1         | E0623 21:47:10.204507068      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.261103370      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 629224 bytes (39.99% savings)
storage_1         | E0623 21:47:10.262281410      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.304739071      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 376666 bytes (64.08% savings)
storage_1         | E0623 21:47:10.306130561      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.349990577      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 440050 bytes (58.03% savings)
storage_1         | E0623 21:47:10.351028116      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.399864590      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 525255 bytes (49.91% savings)
storage_1         | E0623 21:47:10.400964344      21 compression_filter.cc:114]            CompressMessage: len=1048580 alg=2 flags=0
storage_1         | I0623 21:47:10.457476887      21 compression_filter.cc:146]            Compressed[gzip] 1048580 bytes vs. 628167 bytes (40.09% savings)
...
Edited by Jeremiah Bonney

Merge request reports