### Summary When a Workhorse route enforces a request body limit, a request whose body is exactly the limit size is answered with `413 Request Entity Too Large`, although it is within the limit. The routes that enforce a limit on their own are the Terraform state lock and unlock routes (4 KB, [`routes.go#L498-500`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/upstream/routes.go#L498-500)) and `remote_mirrors` (100 KB, [`routes.go#L577`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/upstream/routes.go#L577)). With `WORKHORSE_REQUEST_LIMIT=ENFORCED` ([`roundtripper.go#L91-104`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/upstream/roundtripper/roundtripper.go#L91-104)), the requests Workhorse proxies on every other route are held to their route's limit too, 100 MB unless the route sets another ([`routes.go#L203`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/upstream/routes.go#L203)), and the same happens there. A body sent with `Content-Length` is refused every time. A chunked body is refused when its terminating chunk has not reached Workhorse by the time the data is read. Through the real router on `master`, a chunked 4 KB body whose terminating chunk was sent 100 ms after the data got `413` in 20 of 20 runs, and one whose terminating chunk followed the data straight away got it only occasionally (7 of 200 runs). ### Steps to reproduce 1. On `master`, add this subtest to `TestTerraformStateLockUnlockBodyLimit` in `workhorse/internal/upstream/routes_test.go`, after `small body is allowed`: ```go t.Run("body of exactly the limit is allowed", func(t *testing.T) { req, err := http.NewRequest(tt.method, url, bytes.NewReader(bytes.Repeat([]byte("a"), 4*1024))) require.NoError(t, err) req.Header.Set("Content-Type", "application/json") resp, err := http.DefaultClient.Do(req) require.NoError(t, err) defer resp.Body.Close() require.Equal(t, http.StatusOK, resp.StatusCode) }) ``` 1. Run `go test ./internal/upstream/ -run TestTerraformStateLockUnlockBodyLimit -count=1 -v` in `workhorse/`. ### Example Project Not needed: the route test reproduces it without a GitLab instance. ### What is the current *bug* behavior? Workhorse answers `413 Request Entity Too Large` to a body of exactly 4096 bytes, on the lock route and on the unlock route. The existing subtests in the same test show that a small body is proxied and a 5 KB body is refused. ### What is the expected *correct* behavior? A body of exactly the limit is proxied to Rails like any smaller body. Only a body larger than the limit gets `413`. ### Relevant logs and/or screenshots On `master` at f6995befbec8df5626b977eb2b83ee60b5c0d22c, with Go 1.26.9: ```plaintext === RUN TestTerraformStateLockUnlockBodyLimit/lock/body_of_exactly_the_limit_is_allowed routes_test.go:246: Error: Not equal: expected: 200 actual : 413 === RUN TestTerraformStateLockUnlockBodyLimit/unlock/body_of_exactly_the_limit_is_allowed routes_test.go:246: Error: Not equal: expected: 200 actual : 413 ``` ### Output of checks Not applicable: I reproduced it on the `master` source, not on an installation. ### Possible fixes [`countingReadCloser.Read`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/bodylimit/bodylimit.go#L56-74) returns `RequestBodyTooLargeError` as soon as the byte count has reached the limit, without checking whether the body has ended, and the round tripper turns that error into the `413` ([`bodylimit.go#L126-142`](https://gitlab.com/gitlab-org/gitlab/-/blob/f6995befbec8df5626b977eb2b83ee60b5c0d22c/workhorse/internal/bodylimit/bodylimit.go#L126-142)). Go's HTTP transport reads the body again at that point when it forwards the request: it copies a `Content-Length` body through `io.LimitReader` and then reads once more to detect extra data, and it copies a chunked body until EOF ([`transferWriter.writeBody`](https://cs.opensource.google/go/go/+/refs/tags/go1.25.12:src/net/http/transfer.go;l=356-378)). A chunked body avoids the extra read only when the incoming body returns the end of the body together with the last data, which Go's chunked reader does only when the terminating chunk is already buffered. Once the limit is reached, read one more byte from the underlying body. If it returns no data, return its result (the end of the body, or a read error); if it returns a byte, return `RequestBodyTooLargeError` as today, and keep returning it on every later read. I have a merge request ready for it.