## What Enable the `msgsize_failed_precondition` feature flag, which reports a message exceeding a gRPC size limit as `FAILED_PRECONDITION` instead of `RESOURCE_EXHAUSTED`. grpc-go reports both the send-side and receive-side message size limits as `RESOURCE_EXHAUSTED`, which is what [the cross-language spec mandates](https://github.com/grpc/grpc/blob/master/doc/statuscodes.md). That code is conventionally treated as retryable — it is how server-side throttling is surfaced, so it is the natural thing to list in a retry policy's `retryableStatusCodes`. A message that is too large, however, is rejected identically on every attempt, so retrying it can only burn the retry budget. `FAILED_PRECONDITION` carries the intended meaning: per the status code documentation, use it "if the client should not retry until the system state has been explicitly fixed". The conversion lives in `internal/grpc/middleware/msgsizehandler` and is installed on the Gitaly server's stream interceptor chain and on the client dial options. Every interceptor is a pass-through while the flag is disabled. ### Scope and known limitations - The server side of a **unary** RPC is not interceptable: grpc-go unmarshals the request before the unary interceptor runs and marshals the response after it returns, and it skips the stream interceptor for unary RPCs entirely. In practice these are still converted, because the status message crosses the wire and the client-side unary interceptor converts it there. - The flag is read from incoming request metadata. That is always present server-side, but client-side it only takes effect for calls whose context descends from an inbound RPC. A standalone client such as `gitaly-hooks` has no incoming metadata and will keep seeing `RESOURCE_EXHAUSTED` even with the flag fully enabled. - Detection matches on grpc-go's error strings, because grpc-go exposes no sentinel error or status detail for size violations. A future grpc-go bump that rewords those messages would silently stop the conversion; the middleware's tests cover all known wordings so such a bump fails loudly instead. - Errors that are genuinely about scarce resources are left alone, including anything carrying a `LimitError` detail from the concurrency limiter, and the HTTP/2 `FLOW_CONTROL`/`ENHANCE_YOUR_CALM` mappings. ## Owners - Team: Gitaly - Most appropriate slack channel to reach out to: `#g_gitaly` - Best individual to reach out to: @jcaigitlab ## Expectations ### What release does this feature occur in first? 19.2 ### What are we expecting to happen? No change in behaviour other than the gRPC status code reported when a message exceeds a configured size limit, which moves from `RESOURCE_EXHAUSTED` (8) to `FAILED_PRECONDITION` (9). Gitaly does not currently configure any message size limits itself, so only grpc-go's defaults (4 MiB receive) are in play. ### What might happen if this goes wrong? Any client or test that matches on `RESOURCE_EXHAUSTED` to detect an oversized message will stop matching. Concretely: - **Rails tests force-enable all feature flags** via `GITALY_TESTING_ENABLE_ALL_FEATURE_FLAGS`, so this flag is effectively on in Rails CI as soon as a Gitaly version containing it is picked up, regardless of the production flag state. Any Rails spec asserting `ResourceExhausted` for an oversized message will fail. This is the scenario the rollout template's header warns about, and it may require the "update `GITALY_SERVER_VERSION` and the assertions in the same Rails MR" dance. - Server-side observability still reports the pre-conversion code for the unary paths described above, so logs and metrics may show a mix of both codes during rollout. - Interceptors that branch on the status code see the new code. Both `housekeeping/middleware.go` and the server's log-level function already treat `FAILED_PRECONDITION` and `RESOURCE_EXHAUSTED` the same way, so no behaviour change is expected there, but this has not been verified under load. ### What can we monitor to detect problems with this? - `gitaly_feature_flag_checks_total{flag="msgsize_failed_precondition"}` to confirm the flag is being evaluated. - The error-code breakdown on the [Gitaly feature status dashboard](https://dashboards.gitlab.net/d/000000199/gitaly-feature-status), watching for a shift from `ResourceExhausted` to `FailedPrecondition` and for any unexpected rise in total error rate. ## Roll Out Steps - [ ] Enable on staging - [ ] Is the required code deployed on staging? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed)) - [ ] Enable on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-on-staging)) - [ ] Add ~"featureflag::staging" to this issue ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flag-labels)) - [ ] Test on staging ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#test-on-staging)) - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22jom%22:%7B%22datasource%22:%22mimir-gitlab-gstg%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22rate%28gitaly_feature_flag_checks_total%5B$__rate_interval%5D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gstg%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1&g0.expr=sum%20by%20%28flag%29%20%28rate%28gitaly_feature_flag_checks_total%5B5m%5D%29%29&g0.tab=1&g0.stacked=0&g0.range_input=1h) - [ ] Enable on production - [ ] Is the required code deployed on production? ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#is-the-required-code-deployed)) - [ ] Progressively enable in production ([howto](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#enable-in-production)) - [ ] Add ~"featureflag::production" to this issue - [ ] Verify the feature flag was used by checking Prometheus metric [`gitaly_feature_flag_checks_total`](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22jom%22:%7B%22datasource%22:%22mimir-gitlab-gprd%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22rate%28gitaly_feature_flag_checks_total%5B$__rate_interval%5D%29%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22mimir-gitlab-gprd%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1&g0.expr=sum%20by%20%28flag%29%20%28rate%28gitaly_feature_flag_checks_total%5B5m%5D%29%29&g0.tab=1&g0.stacked=0&g0.range_input=1h) - [ ] Create subsequent issues - [ ] To default enable the feature flag (optional, only required if backwards-compatibility concerns exist) - [ ] [Create issue](https://gitlab.com/gitlab-org/gitaly/-/issues/new?issuable_template=Feature%20Flag%20Default%20Enable) using the `Feature Flag Default Enable` template. - [ ] Set milestone to current+1 release - [ ] To Remove feature flag - [ ] [Create issue](https://gitlab.com/gitlab-org/gitaly/-/issues/new?issuable_template=Feature%20Flag%20Removal) using the `Feature Flag Removal` template. - [ ] Set milestone to current+1 (+2 if we created an issue to default enable the flag). Please refer to the [documentation of feature flags](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/PROCESS.md#feature-flags) for further information.