Skip to content

feat: switch to content negotiation library

feistel requested to merge feistel/gitlab-pages:feat/content-encoding-lib into master

What does this MR do?

I've been working on a few performance improvements and I noticed the content negotiation code is doing a lot more work than it needs to.

Right now we're probing different versions (.gz and .br) of the same file to negotiate with the accepted encodings sent by the client. We're then passing the list of supported encoding to httputil.NegotiateContentEncoding.

Since we're reusing code from gddo I decided to write my own library. The library is ready: it has 100% code coverage, it is compliant to the spec, it has a decent amount of tests and it's heavily optimized.

Performance has been a primary concern, the library has zero allocations and it outperforms the current content negotiation code:

I've run a few benchmarks using some common test cases: https://gitlab.com/feistel/go-contentencoding/-/blob/main/encoding/negotiate_bench_test.go

Before (internal/httputil):

 BenchmarkNegotiate/simple-16             13314931               127.7 ns/op            24 B/op          1 allocs/op
 BenchmarkNegotiate/firefox-16             2432739               524.8 ns/op           168 B/op          3 allocs/op
 BenchmarkNegotiate/value-16               2998654               412.4 ns/op            72 B/op          2 allocs/op
 BenchmarkNegotiate/big-16                 1897549               639.3 ns/op           168 B/op          3 allocs/op

After (go-contentencoding):

 BenchmarkNegotiate/simple-16              11483326               109.0 ns/op             0 B/op          0 allocs/op
 BenchmarkNegotiate/firefox-16              4185826               240.0 ns/op             0 B/op          0 allocs/op
 BenchmarkNegotiate/value-16                3983143               294.6 ns/op             0 B/op          0 allocs/op
 BenchmarkNegotiate/big-16                  3548016               341.9 ns/op             0 B/op          0 allocs/op

Keep in mind the primary benefit is reducing the number of requests to the vfs. The performance gain from optimizing the code is marginal if compared to that.

Overview

The library has one important method:

// Negotiate returns a slice of accepted content encodings for the
// request's Accept-Encoding header.
// The offer earlier in the list is preferred. If no offers are
// acceptable, then an empty slice is returned.
func (pref Preference) Negotiate(acceptHeader string, wildcardStrategy WildcardResolutionStrategy) ([]string, error)

Since it returns a slice of strings we can always pass the full list of supported encodings of the server and then iterate the slice of string to verify the file exists. As result we only need to query the vfs once (best-case scenario) instead of three times.
If the Accept-Header is missing it doesn't even need to query for compressed content and it's even faster.

There are a few more optimizations but they are related to the library and don't affect Pages directly.
I want to wait for feedback on this MR before releasing v1.0.0.

TODO

Edited by feistel

Merge request reports