GitLab Next

Summary

Requests to the gitlab.com API with some id_before ranges produce "500 Internal Server Error"

Steps to reproduce

to reproduce, query https://gitlab.com/api/v4/projects?id_before=2115100&order_by=id&pagination=keyset&per_page=20

the same query works fine for higher ids, eg curl "https://gitlab.com/api/v4/projects?id_before=2115400&order_by=id&pagination=keyset&per_page=20"

also, no error when querying lower ids: curl "https://gitlab.com/api/v4/projects?id_before=2114900&order_by=id&pagination=keyset&per_page=20"

Example Project

What is the current bug behavior?

Instead of a project list, the API returns {"message":"500 Internal Server Error"}

What is the expected correct behavior?

It should return a list of projects

Relevant logs and/or screenshots

The three mentioned requests, response headers only:

~ $ http "https://gitlab.com/api/v4/projects?id_before=2115400&order_by=id&pagination=keyset&per_page=20" --headers
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 5ddf0df56e24d45b-HAM
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 06 Oct 2020 11:31:10 GMT
Etag: W/"e12fb215ee840880a2cc3b93c0d69e7b"
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
GitLab-LB: fe-07-lb-gprd
GitLab-SV: api-01-sv-gprd
Link: <https://gitlab.com/api/v4/projects?id_before=2115001&membership=false&order_by=id&owned=false&page=1&pagination=keyset&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next"
Links: <https://gitlab.com/api/v4/projects?id_before=2115001&membership=false&order_by=id&owned=false&page=1&pagination=keyset&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next"
Referrer-Policy: strict-origin-when-cross-origin
Server: cloudflare
Set-Cookie: __cfduid=dfc7a630c84a157aacc2639f0f0a2b6551601983870; expires=Thu, 05-Nov-20 11:31:10 GMT; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: sZpTo7CJBY8
X-Runtime: 0.234673
cf-request-id: 059f470d620000d45b74a55200000001

~ $ http "https://gitlab.com/api/v4/projects?id_before=2115100&order_by=id&pagination=keyset&per_page=20" --headers
HTTP/1.1 500 Internal Server Error
CF-Cache-Status: DYNAMIC
CF-RAY: 5ddf0e84ffa2d45b-HAM
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 39
Content-Type: application/json
Date: Tue, 06 Oct 2020 11:32:03 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
GitLab-LB: fe-11-lb-gprd
GitLab-SV: api-14-sv-gprd
Server: cloudflare
Set-Cookie: __cfduid=dd4d763efd03831b7fe6aae869924dc321601983893; expires=Thu, 05-Nov-20 11:31:33 GMT; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
Vary: Origin
X-Request-Id: RkjYz0JIF03
X-Runtime: 30.110572
cf-request-id: 059f47671a0000d45b8f2da200000001

~ $ http "https://gitlab.com/api/v4/projects?id_before=2114900&order_by=id&pagination=keyset&per_page=20" --headers
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 5ddf0f89ea25d45f-HAM
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 06 Oct 2020 11:32:15 GMT
Etag: W/"eff664feccf86d80471bf220be82c92d"
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
GitLab-LB: fe-04-lb-gprd
GitLab-SV: api-16-sv-gprd
Link: <https://gitlab.com/api/v4/projects?id_before=2114568&membership=false&order_by=id&owned=false&page=1&pagination=keyset&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next"
Links: <https://gitlab.com/api/v4/projects?id_before=2114568&membership=false&order_by=id&owned=false&page=1&pagination=keyset&per_page=20&repository_checksum_failed=false&simple=false&sort=desc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next"
Referrer-Policy: strict-origin-when-cross-origin
Server: cloudflare
Set-Cookie: __cfduid=d9c0af482c962856318e51805b5bf86ad1601983935; expires=Thu, 05-Nov-20 11:32:15 GMT; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: dnGn6abREt6
X-Runtime: 0.250222
cf-request-id: 059f480a320000d45f4fbf5200000001

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes