Using !reference in a YAML merge key causes server-side exception

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Using YAML <<: merge key with !reference results in server-side exception.

Steps to reproduce

  1. Open the CI/CD Linter page.
  2. Paste the example YAML document below.
  3. Attempt to lint the document.

or

  1. Edit .gitlab-ci.yml in web UI.
  2. Paste the example YAML below.
  3. Save the commit.

or

  1. Add example YAML to local repo .gitlab-ci.yml
  2. Push repo changes.

Example Project

.prod-env:
    variables: &prod
        K8S_CLUSTER: production
        K8S_NAMESPACE: default

.qa-env:
    variables: &qa
        K8S_CLUSTER: development
        K8S_NAMESPACE: qa

.stage-env:
    variables: &stage
        K8S_CLUSTER: production
        K8S_NAMESPACE: staging

.dev-env:
    variables: &dev
        K8S_CLUSTER: development
        K8S_NAMESPACE: testing

deploy:
    rules:
        # Anchor works fine
        - if: $CI_COMMIT_TAG && $CI_COMMIT_REF_PROTECTED
          variables: *prod

        # Merge + anchor works fine
        - if: $CI_COMMIT_TAG
          variables:
              <<: *qa
              NOTIFY: "qa@example.com"

        # Reference works fine
        - if: $CI_COMMIT_REF_PROTECTED
          variables: !reference [.stage-env, variables]
        
        # Merge + reference causes server error
        - when: manual
          variables:
              <<: !reference [.dev-env, variables]
              NOTIFY: "dev@example.com"

    script:
        - env | sort

The above is a minimized example so it may appear a bit contrived. In reality, the environment configs are saved in a separate file and are included, hence the need for !reference.

What is the current bug behavior?

Using !reference in a YAML merge key causes a server-side exception to be thrown.

What is the expected correct behavior?

I expect !reference + merge to work the same as anchor + merge.

Relevant logs and/or screenshots

Processing by Projects::Ci::LintsController#create as JSON
  Parameters: {"content"=>"[FILTERED]", "dry_run"=>false, "namespace_id"=>"redacted", "project_id"=>"redacted", "lint"=>{"content"=>"[FILTERED]", "dry_run"=>false}}
Completed 500 Internal Server Error in 96ms (ActiveRecord: 6.6ms | Elasticsearch: 0.0ms | Allocations: 28496)
  
NoMethodError (undefined method `reverse_each' for #<Gitlab::Ci::Config::Yaml::Tags::Reference:0x00007f47738d2298>):
  
lib/gitlab/config/loader/yaml.rb:13:in `initialize'
lib/gitlab/ci/config/yaml.rb:13:in `new'
lib/gitlab/ci/config/yaml.rb:13:in `load!'
lib/gitlab/ci/config.rb:104:in `block in build_config'
lib/gitlab/ci/pipeline/logger.rb:27:in `instrument'
lib/gitlab/ci/config.rb:103:in `build_config'
ee/lib/ee/gitlab/ci/config_ee.rb:18:in `build_config'
lib/gitlab/ci/config.rb:91:in `expand_config'
lib/gitlab/ci/config.rb:35:in `block in initialize'
lib/gitlab/ci/pipeline/logger.rb:27:in `instrument'
lib/gitlab/ci/config.rb:34:in `initialize'
lib/gitlab/ci/yaml_processor.rb:23:in `new'
lib/gitlab/ci/yaml_processor.rb:23:in `execute'
lib/gitlab/ci/lint.rb:73:in `block in yaml_processor_result'
lib/gitlab/ci/pipeline/logger.rb:27:in `instrument'
lib/gitlab/ci/lint.rb:69:in `yaml_processor_result'
lib/gitlab/ci/lint.rb:56:in `static_validation'
lib/gitlab/ci/lint.rb:33:in `validate'
app/controllers/projects/ci/lints_controller.rb:20:in `create'
...

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	go1.8.3 linux/amd64

GitLab information
Version:	14.8.2-ee
Revision:	20a7fdf52c9
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://redacted
HTTP Clone URL:	https://redacted/some-group/some-project.git
SSH Clone URL:	git@redacted:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: google_oauth2, saml

GitLab Shell
Version:	13.23.2
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Possible fixes

The Psych library only calls reverse_each on sequence nodes, so I think !reference is being interpreted as a sequence type instead of a mapping.

Alternately, if there's correct way to use !reference + merge, I'd be ok with using that instead.

Edited by 🤖 GitLab Bot 🤖