GraphQL query times out with 503 when loading environment and variables
### Problem
Attemping to load https://gitlab.com/gitlab-org/gitlab-foss/-/settings/ci_cd results in 503 timeout errors:

It looks like this `GraphQL` query times out on occasion:

Kibana shows this `SQL` timeout (https://log.gprd.gitlab.net/goto/5be38d60-5962-11ed-8d37-e9a2f393ea2a):
```sql
SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = $1 ORDER BY "environments"."id" DESC LIMIT $2
```
This stems from this `GraphQL` query:
```json
[
{
"operationName": "getProjectEnvironments",
"variables": "[FILTERED]",
"query": "query getProjectEnvironments($fullPath: ID!) {
project(fullPath: $fullPath) {
id
environments {
nodes {
id
name
__typename
}
__typename
}
__typename
}
}
"
},
{
"operationName": "getProjectVariables",
"variables": "[FILTERED]",
"query": "query getProjectVariables($after: String, $first: Int = 100, $fullPath: ID!) {
project(fullPath: $fullPath) {
id
ciVariables(after: $after, first: $first) {
pageInfo {
...PageInfo
__typename
}
nodes {
...BaseCiVariable
environmentScope
masked
protected
__typename
}
__typename
}
__typename
}
}
fragment BaseCiVariable on CiVariable {
__typename
id
key
value
variableType
}
fragment PageInfo on PageInfo {
hasNextPage
hasPreviousPage
startCursor
endCursor
__typename
}
"
}
]
```
##### Backtrace
```ruby
"exception.backtrace": [
"lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'",
"lib/gitlab/graphql/pagination/keyset/connection.rb:135:in `block in limited_nodes'",
"lib/gitlab/utils/strong_memoize.rb:44:in `strong_memoize'",
"lib/gitlab/graphql/pagination/keyset/connection.rb:117:in `limited_nodes'",
"lib/gitlab/graphql/pagination/keyset/connection.rb:96:in `nodes'",
"lib/gitlab/graphql/connection_redaction.rb:23:in `block in nodes'",
"lib/gitlab/graphql/connection_redaction.rb:16:in `redacted'",
"lib/gitlab/graphql/connection_redaction.rb:23:in `nodes'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"app/graphql/gitlab_schema.rb:51:in `multiplex'",
"app/controllers/graphql_controller.rb:160:in `execute_multiplex'",
"app/controllers/graphql_controller.rb:57:in `execute'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
```
## UX Goal
The list appears in a dropdown in a modal on the `project settings >> CI/CD >> Variables` section. Users should be able to easily discover an environment without having to scroll through lengthy lists or leaving the page.
| Image |
|-------|
|  |
### Proposal
Show only the first set of elements (less than 100) and not paginate and rely on the search instead for users who have more to prevent encountering a 503 error.
**NOTE:** Environments scope will require a search bar and won't be paginated whereas variables will be paginated.
Pending: UX proposal for the experience.
## Implementation Table
<!--
_NOTE: If the issue is not part of an epic, the implementation table can be removed. If it is part of an epic, make sure that the implementation table below mirrors the corresponding epic's implementation table content._
-->
| Group | Issue Link |
| ------ | ------ |
| gitlab~2492649 | https://gitlab.com/gitlab-org/gitlab/-/issues/385767 |
| gitlab~2492649 | https://gitlab.com/gitlab-org/gitlab/-/issues/387803 |
| gitlab~3412464 | :point_left: You are here |
#### Implementation
| header | header |
| ------ | ------ |
| https://gitlab.com/gitlab-org/gitlab/-/issues/388952 / https://gitlab.com/gitlab-org/gitlab/-/issues/394750 | Limit fetch/search for environments |
| https://gitlab.com/gitlab-org/gitlab/-/issues/388827 | Implement key-set pagination for CI Variables. |
| https://gitlab.com/gitlab-org/gitlab/-/issues/392947 / https://gitlab.com/gitlab-org/gitlab/-/issues/393882 | Add search ability for CI Variables
| https://gitlab.com/gitlab-org/gitlab/-/issues/392948 | Show confirmation toast message when adding, editing and deleting a CI variable. |
Rollout issue for Environment Scope changes: https://gitlab.com/gitlab-org/gitlab/-/issues/395003
Rollout issue for CI Variable Pagination: https://gitlab.com/gitlab-org/gitlab/-/issues/392874
epic