ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR: canceling statement due to statement timeout

https://sentry.gitlab.net/gitlab/staginggitlabcom/issues/1083356/

PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

  active_record/connection_adapters/postgresql_adapter.rb:611:in `async_exec_params'
    @connection.exec_params(sql, type_casted_binds)
  active_record/connection_adapters/postgresql_adapter.rb:611:in `block (2 levels) in exec_no_cache'
    @connection.exec_params(sql, type_casted_binds)
  active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    yield
  active_support/concurrency/share_lock.rb:187:in `yield_shares'
    yield
  active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    @lock.yield_shares(compatible: [:load]) do
...
(83 additional frame(s) were not displayed)

ActiveRecord::QueryCanceled: PG::QueryCanceled: ERROR:  canceling statement due to statement timeout
: SELECT  "projects"."id" FROM "projects" INNER JOIN "project_repository_states" ON "project_repository_states"."project_id" = "projects"."id" WHERE "project_repository_states"."repository_verification_checksum" IS NOT NULL AND "project_repository_states"."last_repository_verification_failure" IS NULL AND ("project_repository_states"."last_repository_verification_ran_at" IS NULL OR "project_repository_states"."last_repository_verification_ran_at" <= '2019-11-19 11:02:56.610774') AND "projects"."repository_storage" = 'nfs-file21' ORDER BY projects.last_repository_updated_at ASC NULLS LAST LIMIT 938

PG::QueryCanceled: ERROR:  canceling statement due to statement timeout
: SELECT  "projects"."id" FROM "projects" INNER JOIN "project_repository_states" ON "project_repository_states"."project_id" = "projects"."id" WHERE "project_repository_states"."repository_verification_checksum" IS NOT NULL AND "project_repository_states"."last_repository_verification_failure" IS NULL AND ("project_repository_states"."last_repository_verification_ran_at" IS NULL OR "project_repository_states"."last_repository_verification_ran_at" <= '2019-11-19 11:02:56.610774') AND "projects"."repository_storage" = 'nfs-file21' ORDER BY projects.last_repository_updated_at ASC NULLS LAST LIMIT 938

Plain SQL:

SELECT
    "projects"."id"
FROM
    "projects"
    INNER JOIN "project_repository_states" ON "project_repository_states"."project_id" = "projects"."id"
WHERE
    "project_repository_states"."repository_verification_checksum" IS NOT NULL
    AND "project_repository_states"."last_repository_verification_failure" IS NULL
    AND ("project_repository_states"."last_repository_verification_ran_at" IS NULL
        OR "project_repository_states"."last_repository_verification_ran_at" <= '2019-11-19 11:02:56.610774')
    AND "projects"."repository_storage" = 'nfs-file21'
ORDER BY
    projects.last_repository_updated_at ASC NULLS LAST
LIMIT 938

Here's the query plan: https://explain.depesz.com/s/GIn5

It seems filtering on projects.repository_storage is very inefficient.