Assign custom Apdex targets for CI jobs endpoints
# Context
We need to set custom Apdex targets for Rails controller and REST API endpoints related to CI jobs as part of our Q4 scalability initiative for our ~SaaS offering: https://gitlab.com/gitlab-org/gitlab/-/issues/343561, https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1315
The default SLI is 1 second. We need to determine where we will set a custom SLI (up to 5 seconds). Ideally, high traffic endpoints used directly by users (not as a part of a background process) should have a high priority set (< 1 second). We may need to set low traffic, slow endpoints to a low priority. Follow these steps to find the right priority for each endpoint: https://docs.gitlab.com/ee/development/application_slis/rails_request_apdex.html#how-to-adjust-the-urgency
# Technical details
REST API CI jobs endpoints: https://docs.gitlab.com/ee/api/jobs.html
Controller endpoints exist both in job specific controllers like `Projects::BuildsController` and in the actions of other CI controllers such as `Projects::PipelinesController#builds` and `Ci::ExternalPullRequests::CreatePipelineWorker`
Instructions on setting custom Apdex targets: https://docs.gitlab.com/ee/development/application_slis/rails_request_apdex.html#how-to-adjust-the-urgency
## Implementation table
| Description | Link | Milestone | Timeframe |
| ------ | ------ | ------ | ------ |
| **Assign custom Apdex targets for CI jobs endpoints** | :point_left: You are here | %"14.9" | ~"FY22::Q4" |
| Set low urgency as Apdex targets for remaining PE endpoints | https://gitlab.com/gitlab-org/gitlab/-/issues/360273 | %"15.0" | ~"FY23::Q1" |
| Remove ignored_components from PE error budget | https://gitlab.com/gitlab-org/gitlab/-/issues/348552 | %"15.0" | ~"FY23::Q2" |
| Introduce Keyset pagination for `GET /api/:version/projects/:id/jobs` API endpoint | https://gitlab.com/gitlab-org/gitlab/-/issues/362172 | %"15.5" | ~"FY23::Q3" |
| Backend: Improve performance of GraphqlController#execute | https://gitlab.com/gitlab-org/gitlab/-/issues/361377 | TBD | ~"FY23::Q3" |
| Backend: Improve performance of PATCH /api/:version/jobs/:id/trace | https://gitlab.com/gitlab-org/gitlab/-/issues/353802 | TBD | ~"FY23::Q3" |
| Assign custom Apdex targets for remaining PE endpoints | https://gitlab.com/gitlab-org/gitlab/-/issues/348554 | TBD | ~"FY23::Q3" |
## Steps
- [x] 1. Locate all CI jobs REST API and controller endpoints
- For now, using the [Elastic dashboard](https://log.gprd.gitlab.net/app/visualize#/create?type=table&indexPattern=7092c4e2-4eb5-46f2-8305-a7da2edad090&_a=h@304d83b&_g=(time:(from:'now-7d',to:'now'))) is the easiest way to get to the list of endpoints. As written in the [documentation](https://docs.gitlab.com/ee/development/application_slis/rails_request_apdex.html#error-budget-attribution-and-ownership). So from the Stage Group Dashboard, in the error budget attribution row, follow the "Puma Apdex" link to see a list of endpoints and the duration targets they meet or exceed.
- [ ] 2. Select a target duration for each endpoint by consulting [the duration table](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1315#what-target-duration-options-are-available) and following the [steps for determining a custom duration](https://docs.gitlab.com/ee/development/application_slis/rails_request_apdex.html#how-to-adjust-the-urgency)
- [x] 3. Where the duration we want differs from the default (1 second), add a custom target duration to the endpoint. **Involve a [Scalability team member](https://about.gitlab.com/handbook/engineering/infrastructure/team/scalability/#team-members) in the review**
issue