GraphQL query times out with 503 when loading environment and variables
### Problem Attemping to load https://gitlab.com/gitlab-org/gitlab-foss/-/settings/ci_cd results in 503 timeout errors: ![image](/uploads/33c5886122983dece2003065649ca602/image.png) It looks like this `GraphQL` query times out on occasion: ![image](/uploads/41e164e1023b211df356852484fd1320/image.png) Kibana shows this `SQL` timeout (https://log.gprd.gitlab.net/goto/5be38d60-5962-11ed-8d37-e9a2f393ea2a): ```sql SELECT "environments".* FROM "environments" WHERE "environments"."project_id" = $1 ORDER BY "environments"."id" DESC LIMIT $2 ``` This stems from this `GraphQL` query: ```json [ { "operationName": "getProjectEnvironments", "variables": "[FILTERED]", "query": "query getProjectEnvironments($fullPath: ID!) { project(fullPath: $fullPath) { id environments { nodes { id name __typename } __typename } __typename } } " }, { "operationName": "getProjectVariables", "variables": "[FILTERED]", "query": "query getProjectVariables($after: String, $first: Int = 100, $fullPath: ID!) { project(fullPath: $fullPath) { id ciVariables(after: $after, first: $first) { pageInfo { ...PageInfo __typename } nodes { ...BaseCiVariable environmentScope masked protected __typename } __typename } __typename } } fragment BaseCiVariable on CiVariable { __typename id key value variableType } fragment PageInfo on PageInfo { hasNextPage hasPreviousPage startCursor endCursor __typename } " } ] ``` ##### Backtrace ```ruby "exception.backtrace": [ "lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'", "lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'", "lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'", "lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'", "lib/gitlab/graphql/pagination/keyset/connection.rb:135:in `block in limited_nodes'", "lib/gitlab/utils/strong_memoize.rb:44:in `strong_memoize'", "lib/gitlab/graphql/pagination/keyset/connection.rb:117:in `limited_nodes'", "lib/gitlab/graphql/pagination/keyset/connection.rb:96:in `nodes'", "lib/gitlab/graphql/connection_redaction.rb:23:in `block in nodes'", "lib/gitlab/graphql/connection_redaction.rb:16:in `redacted'", "lib/gitlab/graphql/connection_redaction.rb:23:in `nodes'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "app/graphql/gitlab_schema.rb:51:in `multiplex'", "app/controllers/graphql_controller.rb:160:in `execute_multiplex'", "app/controllers/graphql_controller.rb:57:in `execute'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", ``` ## UX Goal The list appears in a dropdown in a modal on the `project settings >> CI/CD >> Variables` section. Users should be able to easily discover an environment without having to scroll through lengthy lists or leaving the page. | Image | |-------| | ![Screenshot_2022-12-16_at_10.45.46_AM](/uploads/f681ef6ec4013819e153bde74f3877ba/Screenshot_2022-12-16_at_10.45.46_AM.png) | ### Proposal Show only the first set of elements (less than 100) and not paginate and rely on the search instead for users who have more to prevent encountering a 503 error. **NOTE:** Environments scope will require a search bar and won't be paginated whereas variables will be paginated. Pending: UX proposal for the experience. ## Implementation Table <!-- _NOTE: If the issue is not part of an epic, the implementation table can be removed. If it is part of an epic, make sure that the implementation table below mirrors the corresponding epic's implementation table content._ --> | Group | Issue Link | | ------ | ------ | | gitlab~2492649 | https://gitlab.com/gitlab-org/gitlab/-/issues/385767 | | gitlab~2492649 | https://gitlab.com/gitlab-org/gitlab/-/issues/387803 | | gitlab~3412464 | :point_left: You are here | #### Implementation | header | header | | ------ | ------ | | https://gitlab.com/gitlab-org/gitlab/-/issues/388952 / https://gitlab.com/gitlab-org/gitlab/-/issues/394750 | Limit fetch/search for environments | | https://gitlab.com/gitlab-org/gitlab/-/issues/388827 | Implement key-set pagination for CI Variables. | | https://gitlab.com/gitlab-org/gitlab/-/issues/392947 / https://gitlab.com/gitlab-org/gitlab/-/issues/393882 | Add search ability for CI Variables | https://gitlab.com/gitlab-org/gitlab/-/issues/392948 | Show confirmation toast message when adding, editing and deleting a CI variable. | Rollout issue for Environment Scope changes: https://gitlab.com/gitlab-org/gitlab/-/issues/395003 Rollout issue for CI Variable Pagination: https://gitlab.com/gitlab-org/gitlab/-/issues/392874
epic