Introduce Keyset pagination for `GET /api/:version/projects/:id/jobs` API endpoint

Problem to solve

Currently, it appears that a PG::QueryCanceled: ERROR: canceling statement due to statement timeout error is occurring on the ci_builds table with the SQL being below:

/*application:web,correlation_id:01G22STTS1JN4Z4HDGZG4HQBMA,endpoint_id:GET /api/:version/projects/:id/jobs,db_config_name:ci_replica*/ SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" = $1 AND "ci_builds"."project_id" = $2 AND "ci_builds"."status" IN ($3, $4, $5) ORDER BY id DESC LIMIT $7 OFFSET $6 

This triggered grouppipeline execution error budget to go unhealthy recently.

Upon further investigation, most of the errors were coming from bots that scrape the jobs: https://log.gprd.gitlab.net/goto/d29aa460-d06f-11ec-b73f-692cc1ae8214

Example: One of the requests was trying to access the page: 354 with a per_page setting of 100.

Proposal

To help improve performance and reduce failing requests due to timeouts, we should add keyset pagination along side offset pagination. And deprecate(not remove) offset pagination in 15.7. Please see the deprecation guidelines and create a deprecation entry before closing this issue.

In another issue in %16.0 we will remove offset pagination but it will be a breaking change.

Benefit of proposal

This would allow a longer runway for evaluating this pagination method being used (however it can be accessed in a non-default way) before promoting this as the next iteration to become the default in 16.0.

Investigation table

Description	Link	Milestone	Timeframe
Assign custom Apdex targets for CI jobs endpoints	#344667 (closed)	%14.9	FY22Q4
Set low urgency as Apdex targets for remaining PE endpoints	#360273 (closed)	%15.0	FY23Q2
Remove ignored_components from PE error budget	#348552 (closed)	%15.0	FY23Q2
Introduce Keyset pagination for GET /api/:version/projects/:id/jobs API endpoint	👈 You are here	TBD	FY23Q4
Backend: Improve performance of GraphqlController#execute	#361377	TBD	TBD
Backend: Improve performance of PATCH /api/:version/jobs/:id/trace	#353802	TBD	TBD
Assign custom Apdex targets for remaining PE endpoints	#348554	TBD	TBD

Deprecation Steps

Group	Issue Link
backend	👈 You are here
backend	Remove offset pagination from `GET /api/:versio... (#367537 - closed)