Getting responses with Status 500 for any API request proxied through our platform

Problem Statement

  • We are seeing responses with status code 500 for nearly all of the api requests we are proxying to Gitlab
  • Since 14:00 UTC on Nov 14th 2019
  • The responses
    • do not contain a response body
    • do not contain a X-Request-Id header
    • do contain a gitlab-lb header
  • The requests
    • are originating from one cluster, we are able to give you request IPs if needed
    • do not have a User-Agent header

Implementation

This is how our service works:

  • Users authorize our Gitlab OAuth app and we store the Gitlab access token in our service
  • User's apps can make requests via our service to the Gitlab API
    • requests are authorized via a JWT issued by an identity service
    • the request is passed on to gitlab and authorized with the token we stored

Reach

There are 118 Sites currently using the integration with Gitlab and getting failing requests in the past 48 hours.

Personas possibly affected are Software Developers or System administrator that setup the service on our platform and are getting support requests from their Users (usually content creators).

Problem quantification: 0.5 = Minimal reach (Less than ~5%).

Impact

1.0 = Medium Impact

People that are using the service to author content are not able to do that right now on Gitlab.

Confidence

80% = Medium confidence

Effort

Unsure?

/cc @fool @mraerino

Edited by Netlify Bot