Backend: Investigate the root cause and fix Internal server error: undefined method `bsearch'

Summary

This typebug was encountered on gitlab.com, and then on GDK locally, where it was identified where it was coming from.

When using a remote include like in the description, we get an error: Internal server error: undefined method bsearch' for nil:NilClass", which comes from [the exists` keyword](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/build/rules/rule/clause/exists.rb#L37).

The config shows up like this as well, which isn't very helpful: Screen_Shot_2022-06-02_at_8.44.39_PM

NoMethodError (undefined method `bsearch' for nil:NilClass):
  lib/gitlab/ci/build/rules/rule/clause/exists.rb:37:in `block in exact_matches?'
  lib/gitlab/ci/build/rules/rule/clause/exists.rb:37:in `any?'
  lib/gitlab/ci/build/rules/rule/clause/exists.rb:37:in `exact_matches?'
  lib/gitlab/ci/build/rules/rule/clause/exists.rb:21:in `satisfied_by?'
  lib/gitlab/ci/build/rules/rule.rb:27:in `block in matches?'
  lib/gitlab/ci/build/rules/rule.rb:27:in `all?'
  lib/gitlab/ci/build/rules/rule.rb:27:in `matches?'
  lib/gitlab/ci/config/external/rules.rb:25:in `block in match_rule'
  lib/gitlab/ci/config/external/rules.rb:25:in `each'
  lib/gitlab/ci/config/external/rules.rb:25:in `find'
  lib/gitlab/ci/config/external/rules.rb:25:in `match_rule'
  lib/gitlab/ci/config/external/rules.rb:19:in `evaluate'
  lib/gitlab/ci/config/external/mapper.rb:76:in `verify_rules_without_instrumentation'
  lib/gitlab/ci/config/external/mapper.rb:71:in `block in verify_rules'
  lib/gitlab/ci/pipeline/logger.rb:33:in `instrument'
  lib/gitlab/ci/config/external/mapper.rb:70:in `verify_rules'
  lib/gitlab/ci/config/external/mapper.rb:45:in `each'
  lib/gitlab/ci/config/external/mapper.rb:45:in `filter_map'
  lib/gitlab/ci/config/external/mapper.rb:45:in `process_without_instrumentation'
  lib/gitlab/ci/config/external/mapper.rb:31:in `block in process'
  lib/gitlab/ci/pipeline/logger.rb:33:in `instrument'
  lib/gitlab/ci/config/external/mapper.rb:30:in `process'
  lib/gitlab/ci/config/external/processor.rb:14:in `initialize'
  lib/gitlab/ci/config/external/file/base.rb:116:in `new'
  lib/gitlab/ci/config/external/file/base.rb:116:in `expand_includes'
  lib/gitlab/ci/config/external/file/base.rb:79:in `block in expanded_content_hash'
  lib/gitlab/utils/strong_memoize.rb:44:in `strong_memoize'
  lib/gitlab/ci/config/external/file/base.rb:78:in `expanded_content_hash'
  lib/gitlab/ci/config/external/file/base.rb:46:in `to_hash'
  lib/gitlab/ci/config/external/file/base.rb:110:in `validate_hash!'
  lib/gitlab/ci/config/external/file/base.rb:54:in `block in validate!'
  lib/gitlab/ci/pipeline/logger.rb:33:in `instrument'
  lib/gitlab/ci/config/external/file/base.rb:50:in `validate!'
  lib/gitlab/ci/config/external/mapper.rb:130:in `verify!'
  lib/gitlab/ci/config/external/mapper.rb:50:in `each'
  lib/gitlab/ci/config/external/mapper.rb:50:in `process_without_instrumentation'
  lib/gitlab/ci/config/external/mapper.rb:31:in `block in process'
  lib/gitlab/ci/pipeline/logger.rb:33:in `instrument'
  lib/gitlab/ci/config/external/mapper.rb:30:in `process'
  lib/gitlab/ci/config/external/processor.rb:14:in `initialize'

Reproduce

# .gitlab-ci.yml of the project

include:
  - remote: http://example.com/random-address/include.yml
# http://example.com/random-address/include.yml

include:
  - local: sub-include.yml
    rules:
      - exists:
        - "Dockerfile"

This bug only occurs when the remote include has an include with an exist rule.

Problem

However, if the remote file has rules inside of their includes, we cannot process them because we don't have enough context to know file changes, variables, etc.

Currently, when you define;

# http://example.com/random-address/include.yml

include:
  - remote: http://example.com/random-address/builds.yml
    rules:
      - if: $CI_COMMIT_BRANCH == "master"

the rule is always evaluated as false and the file will not be included.

However, when you define the rule with exists;

# http://example.com/random-address/include.yml

include:
  - remote: http://example.com/random-address/builds.yml
    rules:
      - exists: [Dockerfile]

we get an error.

We have this error because in lib/gitlab/ci/build/rules/rule/clause/exists.rb;

def satisfied_by?(_pipeline, context)
  paths = worktree_paths(context)

  exact_matches?(paths) || pattern_matches?(paths)
end

private

def worktree_paths(context)
  return unless context.project

  if @top_level_only
    context.top_level_worktree_paths
  else
    context.all_worktree_paths
  end
end

def exact_matches?(paths)
  @exact_globs.any? { |glob| paths.bsearch { |path| glob <=> path } }
end
  • we don't have context, so we don't have context.project
  • worktree_paths returns nil
  • and we try to call methods on paths, which is nil.

Proposal

I think what we need to do here is to return [] from worktree_paths instead of nil, so that exists will also behave as if for the includes.

Links/References

Edited by Mark Nuzzo