2025-01-18: WebServiceLoadbalancerErrorSLOViolation - increased error rate on GPRD cny and main stage

added an incident timeline event

added IncidentActive ServiceWeb Source::IMAIncidentDeclare a:WebServiceLoadbalancerErrorSLOViolation incident severity3 labels

assigned to @astarovoytov

changed the description

changed the severity to Medium - S3

added a resource link

mentioned in issue on-call-handovers#5690 (closed)

added IncidentMitigated label and removed IncidentActive label

This fired again:

source

The 500 errors seem to come from workhorse since that is the next step after loadbalancer SLI:

source

Most of the errors seem to be .git/info/refs based:

source

All of the errors are response body exceeded maximum buffer size (32768 bytes)

source

What I'm slightly confused is that these all seem git requests, but they are actually under the web service

The majority is certainly git as we can see below:

source

So a few questions come to mind:

Do we have different request buffers in web and git?
Why are these requests going to web and not git?
Is web actually serving and successful git requests?

Is web actually serving and successful git requests?

Yes it seems like it is

source

Why are these requests going to web and not git?

Looking at the Gitaly RPC they seem to be things like FindCommit but I still don't get why it's using web

source

Do we have different request buffers in web and git?

No looking at the code the limit is hard coded

All of the IPs seem to be Google-owned IPs

In #19136 (comment 2304089008) (internal URL) we've identified that it's because of malformed URLs that should be returning 404. The question is now did something change in the application that is causing:

Generation of these malformed URLs
404s to hit the wrong workhorse endpoint resulting into the 500 error

Starting to look at some of the requests, they all seem to have started on 2025-01-17, and they are all coming from the same user agent

source

Given that they all seem to be coming from google IP address it makes me wonder if this is a single actor sending these malformed requests

Spot-checking some of these projects they all seem to be using public projects, it would have been private projects we could have pin pointed the common member.

The diff below should fix it, but I'm having trouble testing it locally because my GDK installation is un-happy

$ git diff
diff --git a/workhorse/internal/api/api.go b/workhorse/internal/api/api.go
index 388bcd1c7014..368e8b09019a 100644
--- a/workhorse/internal/api/api.go
+++ b/workhorse/internal/api/api.go
@@ -5,6 +5,7 @@ import (
        "bytes"
        "context"
        "encoding/json"
+       "errors"
        "fmt"
        "io"
        "net/http"
@@ -33,6 +34,8 @@ const (
        geoProxyEndpointPath = "/api/v4/geo/proxy"
 )

+var errResponseLimit = fmt.Errorf("response body exceeded maximum buffer size (%d bytes)", failureResponseLimit)
+
 // API represents a client for interacting with an external API.
 type API struct {
        Client  *http.Client
@@ -177,7 +180,7 @@ type Response struct {
        GitalyServer GitalyServer
        // Repository object for making gRPC requests to Gitaly.
        Repository gitalypb.Repository
-       // For git-http, does the requestor have the right to view all refs?
+       // For git-http, does the requester have the right to view all refs?
        ShowAllRefs bool
        // Detects whether an artifact is used for code intelligence
        ProcessLsif bool
@@ -462,6 +465,10 @@ func passResponseBack(httpResponse *http.Response, w http.ResponseWriter, r *htt
        // the entire response body in memory before sending it on.
        responseBody, err := bufferResponse(httpResponse.Body)
        if err != nil {
+               if errors.Is(err, errResponseLimit) {
+                       fail.Request(w, r, err, fail.WithStatus(http.StatusRequestEntityTooLarge))
+                       return
+               }
                fail.Request(w, r, err)
                return
        }
@@ -492,7 +499,7 @@ func bufferResponse(r io.Reader) (*bytes.Buffer, error) {
        }

        if n == failureResponseLimit {
-               return nil, fmt.Errorf("response body exceeded maximum buffer size (%d bytes)", failureResponseLimit)
+               return nil, errResponseLimit
        }

        return responseBody, nil

After fixing my GDK I'm getting a different response

Returns 404

$ git clone http://gdk.test:3000/root/gitlab-runner/-/tree/v17.8.1.git
Cloning into 'v17.8.1'...
fatal: repository 'http://gdk.test:3000/root/gitlab-runner/-/tree/v17.8.1.git/' not found

Same request returns a 500 in production/staging

$ git clone https://gitlab.com/gitlab-org/gitlab-runner/tree/v17.3.0.git
Cloning into 'v17.3.0'...
remote: Internal Server Error
fatal: unable to access 'https://gitlab.com/gitlab-org/gitlab-runner/tree/v17.3.0.git/': The requested URL returned error: 500

changed the description

mentioned in issue on-call-handovers#5691 (closed)

mentioned in incident #19138 (closed)

mentioned in issue gitlab-org/release/tasks#16445 (closed)

2025-01-18: WebServiceLoadbalancerErrorSLOViolation - increased error rate on GPRD cny and main stage

Customer Impact

Current Status

Next Items

References and helpful links

Deployment Guidance

Child items ...

Activity