Skip to content

getPipeline() securityReportFindings query times out due to slow response from db

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

GitLab Dedicated customer seeing high amount of 503s due to PG::QueryCanceled: ERROR: canceling statement due to statement timeout

Steps to reproduce

Happening on a GitLab Dedicated customer

Example Project

What is the current bug behavior?

The user makes this query

query getPipelines() {
  project(fullPath: "abc/xyz/123") {
    pipeline(id: "gid://gitlab/Ci::Pipeline/xxxx") {
      securityReportFindings(first: 100, after: "NDAw") {
        edges {
          node {
            dismissalReason
            dismissedAt
            falsePositive
            reportType
            severity
            state
            title
            uuid
          }
        }
        pageInfo {
          endCursor
          hasNextPage
        }
      }
    }
  }
}

Which leads to

exception.cause_class PG::QueryCanceled
	
exception.class ActiveRecord::QueryCanceled
	
exception.message PG::QueryCanceled: ERROR:  canceling statement due to statement timeout
	
exception.sql
/*application:web,correlation_id:01K6PWYA9ZVAG2ZN82G5XVE571,endpoint_id:GraphqlController#execute,db_config_database:gitlabhq_production,db_config_name:main*/
SELECT
    security_findings.*
FROM
    security_scans,
    unnest( $1::pg_catalog.int2[] ) AS severities ( severity ),
    LATERAL (
        SELECT
            security_findings.*
        FROM
            security_findings
            LEFT JOIN vulnerability_occurrences ON vulnerability_occurrences.uuid = security_findings.uuid
        WHERE
            security_findings.scan_id = security_scans.id AND
            COALESCE(
                vulnerability_occurrences.severity,
                security_findings.severity
            ) = severities.severity AND
            security_findings.partition_number = $2 AND
            security_findings.deduplicated = $3 AND
            (
                NOT EXISTS (
                    SELECT
                        $4
                    FROM
                        vulnerabilities
                        JOIN vulnerability_occurrences ON vulnerability_occurrences.vulnerability_id = vulnerabilities.id
                    WHERE
                        vulnerabilities.state = $5 AND
                        vulnerability_occurrences.uuid = security_findings.uuid
                )
            )
        ORDER BY
            security_findings.severity DESC,
            security_findings.id ASC
        LIMIT $6
    ) AS security_findings
WHERE
    security_scans.pipeline_id = $7 AND
    security_scans.latest = $8 AND
    security_scans.status = $9
ORDER BY
    security_findings.severity DESC,
    security_findings.id ASC;

graphql 
{
  "complexity": 23,
  "depth": 6,
  "used_fields": [
    "PipelineSecurityReportFinding.dismissalReason",
    "PipelineSecurityReportFinding.dismissedAt",
    "PipelineSecurityReportFinding.falsePositive",
    "PipelineSecurityReportFinding.reportType",
    "PipelineSecurityReportFinding.severity",
    "PipelineSecurityReportFinding.state",
    "PipelineSecurityReportFinding.title",
    "PipelineSecurityReportFinding.uuid",
    "PipelineSecurityReportFindingEdge.node",
    "PipelineSecurityReportFindingConnection.edges",
    "PageInfo.endCursor",
    "PageInfo.hasNextPage",
    "PipelineSecurityReportFindingConnection.pageInfo",
    "Pipeline.securityReportFindings",
    "Project.pipeline",
    "Query.project"
  ],
  "used_deprecated_fields": [],
  "used_deprecated_arguments": [],
  "variables": "{}",
  "operation_name": "getPipelines"
}

What is the expected correct behavior?

The query returns a successful 200 response

Relevant logs and/or screenshots

Exception backtrace ``` activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `exec_params', activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `block (2 levels) in exec_no_cache', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract_adapter.rb:1027:in `block in with_raw_connection', activesupport (7.1.5.1) lib/active_support/concurrency/null_lock.rb:9:in `synchronize', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract_adapter.rb:999:in `with_raw_connection', activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:893:in `block in exec_no_cache', activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract_adapter.rb:1142:in `log', activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:892:in `exec_no_cache', activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear', marginalia (1.11.1) lib/marginalia.rb:91:in `execute_and_clear_with_marginalia', activerecord (7.1.5.1) lib/active_record/connection_adapters/postgresql/database_statements.rb:64:in `internal_exec_query', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `block in select_all', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/query_cache.rb:152:in `block in cache_sql', activesupport (7.1.5.1) lib/active_support/concurrency/null_lock.rb:9:in `synchronize', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/query_cache.rb:147:in `cache_sql', activerecord (7.1.5.1) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `select_all', lib/gitlab/database/load_balancing/connection_proxy.rb:111:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:111:in `block in read_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:65:in `read', lib/gitlab/database/load_balancing/connection_proxy.rb:110:in `read_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all', activerecord (7.1.5.1) lib/active_record/querying.rb:62:in `_query_by_sql', activerecord (7.1.5.1) lib/active_record/relation.rb:1008:in `block in exec_main_query', activerecord (7.1.5.1) lib/active_record/relation.rb:1028:in `skip_query_cache_if_necessary', activerecord (7.1.5.1) lib/active_record/relation.rb:994:in `exec_main_query', activerecord (7.1.5.1) lib/active_record/relation.rb:972:in `block in exec_queries', activerecord (7.1.5.1) lib/active_record/relation.rb:1028:in `skip_query_cache_if_necessary', activerecord (7.1.5.1) lib/active_record/relation.rb:966:in `exec_queries', gems/activerecord-gitlab/lib/active_record/gitlab_patches/rescue_from.rb:31:in `exec_queries', activerecord (7.1.5.1) lib/active_record/relation.rb:752:in `load', activerecord (7.1.5.1) lib/active_record/relation.rb:264:in `records', activerecord (7.1.5.1) lib/active_record/relation/delegation.rb:100:in `each', ee/app/graphql/resolvers/pipeline_security_report_findings_resolver.rb:35:in `block in resolve', :90:in `tap', ee/app/graphql/resolvers/pipeline_security_report_findings_resolver.rb:35:in `resolve', graphql (2.5.11) lib/graphql/schema/resolver.rb:118:in `public_send', graphql (2.5.11) lib/graphql/schema/resolver.rb:118:in `call_resolve', graphql (2.5.11) lib/graphql/schema/resolver.rb:105:in `block (3 levels) in resolve_with_support', graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy', graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy', graphql (2.5.11) lib/graphql/schema/resolver.rb:93:in `block (2 levels) in resolve_with_support', graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy', graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy', graphql (2.5.11) lib/graphql/schema/resolver.rb:84:in `block in resolve_with_support', graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy', graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy', graphql (2.5.11) lib/graphql/schema/resolver.rb:72:in `resolve_with_support', graphql (2.5.11) lib/graphql/schema/field.rb:758:in `public_send', graphql (2.5.11) lib/graphql/schema/field.rb:758:in `block (2 levels) in resolve', graphql (2.5.11) lib/graphql/schema/field.rb:903:in `block in with_extensions', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block (2 levels) in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:942:in `run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field_extension.rb:134:in `resolve', graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve', lib/gitlab/graphql/present/field_extension.rb:18:in `resolve', graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve', lib/gitlab/graphql/extensions/externally_paginated_array_extension.rb:7:in `resolve', graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:939:in `block in run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field_extension.rb:134:in `resolve', graphql (2.5.11) lib/graphql/schema/field.rb:926:in `run_extensions_before_resolve', graphql (2.5.11) lib/graphql/schema/field.rb:898:in `with_extensions', graphql (2.5.11) lib/graphql/schema/field.rb:729:in `block in resolve', graphql (2.5.11) lib/graphql/schema.rb:1647:in `after_lazy', graphql (2.5.11) lib/graphql/query.rb:30:in `after_lazy', graphql (2.5.11) lib/graphql/schema/field.rb:727:in `resolve', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:462:in `block (2 levels) in evaluate_selection_with_resolved_keyword_args', graphql (2.5.11) lib/graphql/tracing/trace.rb:90:in `execute_field', batch-loader (2.0.5) lib/batch_loader/graphql.rb:7:in `execute_field', graphql (2.5.11) lib/graphql/schema/timeout.rb:92:in `execute_field', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:461:in `block in evaluate_selection_with_resolved_keyword_args', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:809:in `call_method_on_directives', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:449:in `evaluate_selection_with_resolved_keyword_args', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:427:in `block in evaluate_selection_with_args', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:936:in `after_lazy', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:374:in `evaluate_selection_with_args', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:368:in `block in evaluate_selection', graphql (2.5.11) lib/graphql/execution/interpreter/arguments_cache.rb:41:in `dataload_for', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:366:in `evaluate_selection', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:315:in `block (3 levels) in evaluate_selections', graphql (2.5.11) lib/graphql/dataloader/null_dataloader.rb:22:in `append_job', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:314:in `block (2 levels) in evaluate_selections', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:301:in `each', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:301:in `block in evaluate_selections', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:809:in `call_method_on_directives', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:300:in `evaluate_selections', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:728:in `block (2 levels) in continue_field', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:209:in `each_gathered_selections', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:717:in `block in continue_field', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:936:in `after_lazy', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:712:in `continue_field', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:481:in `block (2 levels) in evaluate_selection_with_resolved_keyword_args', graphql (2.5.11) lib/graphql/execution/interpreter/runtime.rb:919:in `block in after_lazy', graphql (2.5.11) lib/graphql/execution/lazy.rb:30:in `value', graphql (2.5.11) lib/graphql/execution/interpreter/resolve.rb:27:in `block (2 levels) in resolve_each_depth', graphql (2.5.11) lib/graphql/dataloader/null_dataloader.rb:22:in `append_job', graphql (2.5.11) lib/graphql/execution/interpreter/resolve.rb:27:in `block in resolve_each_depth', graphql (2.5.11) lib/graphql/execution/interpreter/resolve.rb:26:in `each', graphql (2.5.11) lib/graphql/execution/interpreter/resolve.rb:26:in `resolve_each_depth', graphql (2.5.11) lib/graphql/execution/interpreter/resolve.rb:31:in `resolve_each_depth', graphql (2.5.11) lib/graphql/execution/interpreter.rb:97:in `block (3 levels) in run_all', graphql (2.5.11) lib/graphql/tracing/trace.rb:72:in `execute_query_lazy', graphql (2.5.11) lib/graphql/execution/interpreter.rb:96:in `block (2 levels) in run_all', graphql (2.5.11) lib/graphql/dataloader/null_dataloader.rb:22:in `append_job', graphql (2.5.11) lib/graphql/execution/interpreter.rb:94:in `block in run_all', graphql (2.5.11) lib/graphql/tracing/trace.rb:64:in `execute_multiplex', lib/gitlab/graphql/tracers/instrumentation_tracer.rb:23:in `execute_multiplex', graphql (2.5.11) lib/graphql/schema/timeout.rb:68:in `execute_multiplex', graphql (2.5.11) lib/graphql/execution/interpreter.rb:42:in `run_all', graphql (2.5.11) lib/graphql/schema.rb:1603:in `multiplex', app/graphql/gitlab_schema.rb:47:in `multiplex', graphql (2.5.11) lib/graphql/schema.rb:1578:in `execute', app/controllers/graphql_controller.rb:269:in `execute_query', app/controllers/graphql_controller.rb:66:in `execute', actionpack (7.1.5.1) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action', actionpack (7.1.5.1) lib/abstract_controller/base.rb:224:in `process_action', actionpack (7.1.5.1) lib/action_controller/metal/rendering.rb:165:in `process_action', actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:259:in `block in process_action', activesupport (7.1.5.1) lib/active_support/callbacks.rb:121:in `block in run_callbacks', lib/gitlab/auth/current_user_mode.rb:45:in `bypass_session!', app/controllers/concerns/sessionless_authentication.rb:40:in `sessionless_bypass_admin_mode!', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', lib/gitlab/auth/current_user_mode.rb:75:in `with_current_admin', app/controllers/application_controller.rb:488:in `set_current_admin', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', app/controllers/application_controller.rb:474:in `set_session_storage', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', lib/gitlab/i18n.rb:116:in `with_locale', lib/gitlab/i18n.rb:122:in `with_user_locale', app/controllers/application_controller.rb:467:in `set_locale', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', app/controllers/application_controller.rb:456:in `set_current_context', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', lib/gitlab/ip_address_state.rb:11:in `with', app/controllers/application_controller.rb:462:in `set_current_ip_address', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action', sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span', sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span', sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action', activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks', activesupport (7.1.5.1) lib/active_support/callbacks.rb:141:in `run_callbacks', actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:258:in `process_action', actionpack (7.1.5.1) lib/action_controller/metal/rescue.rb:25:in `process_action', actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action', activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `block in instrument', activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument', activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `instrument', actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:73:in `process_action', actionpack (7.1.5.1) lib/action_controller/metal/params_wrapper.rb:261:in `process_action', activerecord (7.1.5.1) lib/active_record/railties/controller_runtime.rb:32:in `process_action', actionpack (7.1.5.1) lib/abstract_controller/base.rb:160:in `process', actionview (7.1.5.1) lib/action_view/rendering.rb:40:in `process', actionpack (7.1.5.1) lib/action_controller/metal.rb:227:in `dispatch', actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `block in dispatch', lib/gitlab/middleware/action_controller_static_context.rb:23:in `call', actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `dispatch', actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch', actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve', actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:51:in `block in serve', config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes', config/initializers/action_dispatch_journey_router.rb:25:in `map!', config/initializers/action_dispatch_journey_router.rb:25:in `find_routes', actionpack (7.1.5.1) lib/action_dispatch/journey/rou ter.rb:32:in `serve', actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:882:in `call', gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call', omniauth (2.1.3) lib/omniauth/strategy.rb:472:in `call_app!', omniauth-saml (2.2.4) lib/omniauth/strategies/saml.rb:83:in `other_phase', omniauth (2.1.3) lib/omniauth/strategy.rb:195:in `call!', omniauth (2.1.3) lib/omniauth/strategy.rb:169:in `call', flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call', flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call', lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call', lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled', lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call', lib/gitlab/middleware/memory_report.rb:13:in `call', lib/gitlab/middleware/speedscope.rb:13:in `call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call', lib/gitlab/middleware/rails_queue_duration.rb:33:in `call', lib/gitlab/etag_caching/middleware.rb:21:in `call', lib/gitlab/metrics/rack_middleware.rb:16:in `block in call', lib/gitlab/metrics/web_transaction.rb:46:in `run', lib/gitlab/metrics/rack_middleware.rb:16:in `call', lib/gitlab/middleware/go.rb:21:in `call', lib/gitlab/middleware/query_analyzer.rb:11:in `block in call', lib/gitlab/database/query_analyzer.rb:83:in `within', lib/gitlab/middleware/query_analyzer.rb:11:in `call', lib/ci/job_token/middleware.rb:11:in `call', batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call', rack-attack (6.7.0) lib/rack/attack.rb:103:in `call', apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call', lib/gitlab/middleware/multipart.rb:174:in `call', rack-attack (6.7.0) lib/rack/attack.rb:127:in `call', warden (1.2.9) lib/warden/manager.rb:36:in `block in call', warden (1.2.9) lib/warden/manager.rb:34:in `catch', warden (1.2.9) lib/warden/manager.rb:34:in `call', rack-cors (2.0.2) lib/rack/cors.rb:102:in `call', rack (2.2.17) lib/rack/tempfile_reaper.rb:15:in `call', rack (2.2.17) lib/rack/etag.rb:27:in `call', rack (2.2.17) lib/rack/conditional_get.rb:40:in `call', rack (2.2.17) lib/rack/head.rb:12:in `call', actionpack (7.1.5.1) lib/action_dispatch/http/permissions_policy.rb:36:in `call', actionpack (7.1.5.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call', lib/gitlab/middleware/read_only/controller.rb:40:in `call', lib/gitlab/middleware/read_only.rb:18:in `call', lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call', rack (2.2.17) lib/rack/session/abstract/id.rb:266:in `context', rack (2.2.17) lib/rack/session/abstract/id.rb:260:in `call', lib/gitlab/middleware/secure_headers.rb:11:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/cookies.rb:689:in `call', lib/gitlab/middleware/same_site_cookies.rb:27:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call', activesupport (7.1.5.1) lib/active_support/callbacks.rb:101:in `run_callbacks', actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call', sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call', lib/gitlab/middleware/path_traversal_check.rb:40:in `call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call', lib/gitlab/middleware/json_validation.rb:179:in `allow_if_validated', lib/gitlab/middleware/json_validation.rb:160:in `call', sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call', sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking', sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking', sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call', sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope', sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope', sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call', lib/gitlab/middleware/basic_health_check.rb:25:in `call', lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app', railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `block in call', activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `block in tagged', activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:39:in `tagged', activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `tagged', activesupport (7.1.5.1) lib/active_support/broadcast_logger.rb:241:in `method_missing', railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/remote_ip.rb:92:in `call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call', lib/gitlab/middleware/request_context.rb:15:in `call', lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call', request_store (1.7.0) lib/request_store/middleware.rb:19:in `call', rack (2.2.17) lib/rack/method_override.rb:24:in `call', rack (2.2.17) lib/rack/runtime.rb:22:in `call', rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call', rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout', rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call', config/initializers/fix_local_cache_middleware.rb:11:in `call', lib/gitlab/middleware/compressed_json.rb:44:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call', lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call', gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:22:in `block in call', gitlab-labkit (0.39.0) lib/labkit/context.rb:35:in `with_context', gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:21:in `call', rack (2.2.17) lib/rack/sendfile.rb:110:in `call', actionpack (7.1.5.1) lib/action_dispatch/middleware/request_id.rb:28:in `call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call', railties (7.1.5.1) lib/rails/engine.rb:536:in `call', railties (7.1.5.1) lib/rails/railtie.rb:226:in `public_send', railties (7.1.5.1) lib/rails/railtie.rb:226:in `method_missing', lib/gitlab/middleware/release_env.rb:12:in `call', rack (2.2.17) lib/rack/urlmap.rb:74:in `block in call', rack (2.2.17) lib/rack/urlmap.rb:58:in `each', rack (2.2.17) lib/rack/urlmap.rb:58:in `call', puma (6.6.1) lib/puma/configuration.rb:279:in `call', puma (6.6.1) lib/puma/request.rb:99:in `block in handle_request', puma (6.6.1) lib/puma/thread_pool.rb:390:in `with_force_shutdown', puma (6.6.1) lib/puma/request.rb:98:in `handle_request', puma (6.6.1) lib/puma/server.rb:472:in `process_client', puma (6.6.1) lib/puma/server.rb:254:in `block in run', puma (6.6.1) lib/puma/thread_pool.rb:167:in `block in spawn_thread' ```

Output of checks

Results of GitLab environment info

GitLab version: 18.3.3

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)

(we will only investigate if the tests are passing)

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖