Skip to content

(CI job) GraphQL Timeout on validation of query

Some query validations are timing out. The timeout is 0.2 seconds (defined in https://gitlab.com/gitlab-org/gitlab/blob/cd54700c365c3fcfb4b28d8dfe62095291f673d8/app/graphql/gitlab_schema.rb#L38-38). Recently (week of 15th March onwards) we're getting some timeouts during query validations. Initially it errored out, but that was fixed. Now we see the timeout message.

Here are the recent errors:

Date (UTC) job pipeline slack link note
2023-03-29 12:37 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/4019788804 https://gitlab.com/gitlab-org/gitlab/-/pipelines/820951163 https://gitlab.slack.com/archives/CR6QH3D7C/p1680050245708209 app/assets/javascripts/incidents/graphql/queries/get_count_by_status.query.graphql
2023-03-28 10:23 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/3985547772 https://gitlab.com/gitlab-org/gitlab/-/pipelines/815143057 https://gitlab.slack.com/archives/CR6QH3D7C/p1679999004230229 app/assets/javascripts/alert_management/graphql/queries/get_count_by_status.query.graphql
2023-03-24 10:01 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3998992447 https://gitlab.com/gitlab-org/gitlab/-/pipelines/817350150 https://gitlab.slack.com/archives/CR6QH3D7C/p1679691679358239 app/assets/javascripts/alerts_settings/graphql/mutations/create_http_integration.mutation.graphql
2023-03-24 7:18 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3998230424 https://gitlab.com/gitlab-org/gitlab/-/pipelines/817235763 https://gitlab.slack.com/archives/CR6QH3D7C/p1679681924898339 app/assets/javascripts/alerts_settings/graphql/mutations/create_http_integration.mutation.graphql
2023-03-23 5:30 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3990905672 https://gitlab.com/gitlab-org/gitlab/-/pipelines/816024583 https://gitlab.slack.com/archives/CR6QH3D7C/p1679592626653749 app/assets/javascripts/alert_management/graphql/queries/get_count_by_status.query.graphql
2023-03-23 5:13 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3990738151 https://gitlab.com/gitlab-org/gitlab/-/pipelines/815997114 https://gitlab.slack.com/archives/CR6QH3D7C/p1679591606894899 app/assets/javascripts/alerts_settings/graphql/mutations/create_prometheus_integration.mutation.graphql
2023-03-24 4:15 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/3990136457 https://gitlab.com/gitlab-org/gitlab/-/pipelines/815901887 https://gitlab.slack.com/archives/CR6QH3D7C/p1679588116947659 ee/app/assets/javascripts/security_dashboard/graphql/queries/security_report_finding.query.graphql
2023-03-22 11:27 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3984691778 https://gitlab.com/gitlab-org/gitlab/-/pipelines/814995164 https://gitlab.slack.com/archives/CR6QH3D7C/p1679524031000579 app/assets/javascripts/admin/users/graphql/queries/get_users_group_counts.query.graphql
2023-03-22 3:46 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3982067164 https://gitlab.com/gitlab-org/gitlab/-/pipelines/814596886 https://gitlab.slack.com/archives/CR6QH3D7C/p1679499971376439 app/assets/javascripts/alerts_settings/graphql/mutations/create_http_integration.mutation.graphql
2023-03-22 08:25 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3980527013 https://gitlab.com/gitlab-org/gitlab/-/pipelines/814353073 https://gitlab.slack.com/archives/CR6QH3D7C/p1679488712390989 app/assets/javascripts/alerts_settings/graphql/mutations/create_http_integration.mutation.graphql
2023-03-20 11:53 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3965095499 https://gitlab.com/gitlab-org/gitlab/-/pipelines/811775183 https://gitlab.slack.com/archives/CR6QH3D7C/p1679313213265589 ee/app/assets/javascripts/iterations/queries/iteration_issues.query.graphql
2023-03-20 4:35 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/3963091666 https://gitlab.com/gitlab-org/gitlab/-/pipelines/811413845 https://gitlab.slack.com/archives/CR6QH3D7C/p1679286922049039 app/assets/javascripts/vue_shared/components/issuable_blocked_icon/graphql/blocking_epics.query.graphql
2023-03-17 10:06 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3958245887 https://gitlab.com/gitlab-org/gitlab/-/pipelines/810403989 https://gitlab.slack.com/archives/CR6QH3D7C/p1679090940352119 app/assets/javascripts/vue_merge_request_widget/queries/toggle_draft.mutation.graphql
2023-03-16 5:29 PM https://gitlab.com/gitlab-org/gitlab/-/jobs/3949344378 https://gitlab.com/gitlab-org/gitlab/-/pipelines/808934265 https://gitlab.slack.com/archives/CR6QH3D7C/p1678987768443019 ee/app/assets/javascripts/security_dashboard/graphql/queries/instance_vulnerabilities.query.graphql
2023-03-15 8:55 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/3942725978 https://gitlab.com/gitlab-org/gitlab/-/pipelines/807806631 https://gitlab.slack.com/archives/CR6QH3D7C/p1678913708325829 app/assets/javascripts/work_items/graphql/create_work_item.mutation.graphql?
2023-03-15 4:41 AM https://gitlab.com/gitlab-org/gitlab/-/jobs/3935926760 https://gitlab.com/gitlab-org/gitlab-foss/-/pipelines/807203732 https://gitlab.slack.com/archives/CR6QH3D7C/p1678855274336589 app/assets/javascripts/work_items/graphql/create_work_item_from_task.mutation.graphql?

After the fix in !114809 (merged), we now see :

ERROR /builds/gitlab-org/gitlab/ee/app/assets/javascripts/security_dashboard/graphql/queries/instance_vulnerabilities.query.graphql: Timeout on validation of query 

##########
#
# 1 GraphQL query out of 758 failed validation:
# - /builds/gitlab-org/gitlab/ee/app/assets/javascripts/security_dashboard/graphql/queries/instance_vulnerabilities.query.graphql
#
##########

https://gitlab.com/gitlab-org/gitlab/-/jobs/3949344378


Job #3942725978 failed for 55bba418:

OK    /builds/gitlab-org/gitlab/app/assets/javascripts/vue_shared/components/runner_instructions/graphql/get_runner_setup.query.graphql
OK    /builds/gitlab-org/gitlab/app/assets/javascripts/vue_shared/security_reports/graphql/queries/security_report_merge_request_download_paths.query.graphql
OK    /builds/gitlab-org/gitlab/app/assets/javascripts/vue_shared/security_reports/graphql/queries/security_report_pipeline_download_paths.query.graphql
SKIP  /builds/gitlab-org/gitlab/app/assets/javascripts/work_items/graphql/add_hierarchy_child.mutation.graphql: client query
rake aborted!
NoMethodError: undefined method `join' for nil:NilClass
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:90:in `block (5 levels) in <main>'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:88:in `each'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:88:in `block (4 levels) in <main>'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:80:in `each'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:80:in `flat_map'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/graphql.rake:80:in `block (3 levels) in <main>'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli/exec.rb:58:in `load'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli/exec.rb:58:in `kernel_load'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli/exec.rb:23:in `run'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli.rb:492:in `exec'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli.rb:34:in `dispatch'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/cli.rb:28:in `start'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/exe/bundle:45:in `block in <top (required)>'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/builds/gitlab-org/gitlab/vendor/ruby/3.0.0/gems/bundler-2.4.8/exe/bundle:33:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:graphql:validate
(See full trace by running task with --trace)
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1
Edited by charlie ablett