Pipeline showing error: `yaml invalid: undefined error` when `git fsck` reports problems with associated Git repository
Summary
A pipeline instance has yaml invalid: undefined error
, but a previous pipeline instance with the exact same Git commit ID, on the same branch, has no yaml invalid
error. The YAML file validates in public validators just fine, and the YAML is exactly the same between pipeline instances.
Likely cause
The corresponding repository for the project with the broken Pipeline doesn't have a master
branch, and the Git repository on the disk is erroneously pointing to it:
root@gitlab:/var/opt/gitlab/git-data/repositories/BigTerminal/scala/invoice-sender-worker.git
$ git fsck
notice: HEAD points to an unborn branch (master)
Checking object directories: 100% (256/256), done.
Checking objects: 100% (75/75), done.
When I push a master
branch, it fixes the problem.
Important: Note that this repository was imported from GitHub.com, and never had a master
branch, and previously worked fine on GitLab. So the GitLab server got the repository in to a state where it was erroneously pointing to master
, and I suspect that it was after creating an "Environment", or otherwise playing with that feature, but I can't be sure. You should try to mess with Environments when reproducing this error.
Steps to reproduce
- (Probably): Make a new Environment with the same name as a branch that hasn't yet been pushed
- (Probably): Push that branch for the first time
- Push a pipeline
- You will probably get the error
- Delete all environments for this project with the API because someone was too lazy to make a button
- Delete all branches corresponding to those deleted environments, except the default branch
- Push a non-default branch to the remote, which creates an new instance of that branch remotely
- Notice that the same stupid
yaml invalid: undefined error
occurs for the same commit ID that used to work - Delete the environment and remote branch again, then create the environment before pushing, then push, then notice that the same error still happens
- Make a new project and copy the same YAML file over, commit to the main branch, then push for the first time, and notice that the same thing happens. No environments were ever created.
After some action, which may be the creation of environments, the entire GitLab server goes in to a state where any YAML .gitlab-ci.yml
file causes a yaml invalid: undefined error
, and its irreversible. It's questionable whether the .gitlab-ci.yml
file is actually the YAML file in question.
Example Project
n/a
What is the current bug behavior?
The pipeline fails with a yaml invalid: undefined error
.
It happens for all projects, with any valid, simple .gitlab-ci.yml
file that was working before. The GitLab server gets in to an irreversible state where it breaks for any configuration. Removing the .gitlab-ci.yml
file from a project entirely ceases to trigger a Pipeline creation.
What is the expected correct behavior?
The pipeline instance should successfully execute like it did before with the same Git commit ID.
Relevant logs and/or screenshots
I removed a catchall rescue
from the code and got the app to print a stack trace:
WARN: NoMethodError: undefined method `sha' for nil.NilClass`
Expand for stack trace
2018-05-04T15:40:50.112Z 470 TID-os3t4h6f4 WARN: NoMethodError: undefined method `sha' for nil:NilClass
2018-05-04T15:40:50.112Z 470 TID-os3t4h6f4 WARN: /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:30:in `process_external_files'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:21:in `build_config'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config.rb:11:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:9:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/yaml_processor.rb:11:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/yaml_processor.rb:11:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/ci/pipeline.rb:40:in `new'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/ci/pipeline.rb:40:in `initialize_yaml_processor'
/opt/gitlab/embedded/service/gitlab-rails/app/models/ci/pipeline.rb:416:in `config_processor'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/pipeline/chain/remove_unwanted_chat_jobs.rb:8:in `perform!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:17:in `block in build!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:14:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:14:in `build!'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_pipeline_service.rb:41:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/app/services/git_push_service.rb:144:in `execute_related_hooks'
/opt/gitlab/embedded/service/gitlab-rails/app/services/git_push_service.rb:59:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:43:in `block in process_project_changes'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:22:in `block in changes_refs'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:19:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:19:in `changes_refs'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:32:in `process_project_changes'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:22:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/server/active_record.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:36:in `block in '
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
Another stack trace
undefined method `sha' for nil:NilClass["/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:30:in `process_external_files'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:21:in `build_config'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config.rb:11:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/config.rb:9:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/yaml_processor.rb:11:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/yaml_processor.rb:11:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/ci/pipeline.rb:40:in `new'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/ee/ci/pipeline.rb:40:in `initialize_yaml_processor'
/opt/gitlab/embedded/service/gitlab-rails/app/models/ci/pipeline.rb:416:in `config_processor'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/ci/pipeline/chain/remove_unwanted_chat_jobs.rb:8:in `perform!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:17:in `block in build!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:14:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/pipeline/chain/sequence.rb:14:in `build!'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_pipeline_service.rb:41:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/app/services/git_push_service.rb:144:in `execute_related_hooks'
/opt/gitlab/embedded/service/gitlab-rails/app/services/git_push_service.rb:59:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:43:in `block in process_project_changes'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:22:in `block in changes_refs'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:19:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git_post_receive.rb:19:in `changes_refs'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:32:in `process_project_changes'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/post_receive.rb:22:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:188:in `execute_job'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:170:in `block (2 levels) in process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/server/active_record.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:169:in `block in process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:141:in `block (6 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:97:in `local'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:140:in `block (5 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:36:in `block in '
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:136:in `block (4 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:204:in `stats'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:131:in `block (3 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_logger.rb:7:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:130:in `block (2 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/job_retry.rb:72:in `global'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:129:in `block in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:44:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/logging.rb:38:in `with_job_hash_context'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:128:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:168:in `process'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:85:in `process_one'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'"]
You can see here that the job with commit ID e3ed717a
at the top of the list has a yaml invalid
error (which says undefined error
on mouseover), but the job with the exact same commit ID, and therefore the exact same .gitlab-ci.yml
file, has no yaml invalid
error at all:
A previously complex YAML file was used, which worked for a while as mentioned above, and now the entire GitLab server is in a state where all YAML files in all projects trigger a yaml invalid: undefined error
regardless of their content. This simple YAML file causes the error, and the same version without "
characters does as well:
stages:
- "compile"
- "test"
compile:
stage: "compile"
script: "echo compile"
test:
stage: "test"
script: "echo test"
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
root@gitlab:/# gitlab-rake gitlab:env:infoSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.7.2-ee Revision: a4f29081 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.8 URL: http://${GITLAB_HOST} HTTP Clone URL: http://${GITLAB_HOST}/some-group/some-project.git SSH Clone URL: git@${GITLAB_HOST}:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 4/1 ... ok 4/2 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 4/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 2 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Possible fixes
This only seems to happen when this issue is also present: https://gitlab.com/gitlab-org/gitlab-ee/issues/5860
GitLab Enterprise Edition seems to get in to a globally fucked-up state, since this now happens to new projects where environments weren't created.