Skip to content

ERROR: column ci_builds.commands does not exist following after upgrade to 13.10.0

Summary

Upgrading the GitLab docker container from version 13.9.3 to 13.10.0 results in the following error being logged every 15 seconds in postgresql/current:

ERROR:  column ci_builds.commands does not exist at character 825
STATEMENT:  SELECT
	  subquery.namespace_id,
	  subquery.shared_runners_enabled,
	  subquery.project_id,
	  subquery.status,
	  MAX(subquery.count) as count
	FROM (
	  SELECT
	    projects.namespace_id,
	    projects.shared_runners_enabled,
	    ci_builds.project_id,
	    ci_builds.commit_id,
	    ci_builds.status,
	    COUNT(*) AS count
	  FROM ci_builds
	  JOIN projects
	    ON projects.id = ci_builds.project_id
	  JOIN namespaces
	    ON namespaces.id = projects.namespace_id
	  WHERE ci_builds.type = 'Ci::Build'
	    AND ci_builds.status IN ('running', 'pending')
	  -- The created_at filter has been introduced for performance reasons only
	  AND ci_builds.created_at > NOW() - INTERVAL '7 days'
	  GROUP BY
	    projects.namespace_id,
	    projects.shared_runners_enabled,
	    ci_builds.project_id,
	    ci_builds.commit_id,
	    ci_builds.status,
	    ci_builds.commands
	  HAVING COUNT(*) > 2
	) AS subquery
	GROUP BY
	  subquery.namespace_id,
	  subquery.shared_runners_enabled,
	  subquery.project_id,
	  subquery.commit_id,
	  subquery.status

Originally reported in: https://forum.gitlab.com/t/upgrade-13-1-11-13-10-0-error-column-ci-builds-commands-does-not-exist/50688/3

Steps to reproduce

  1. Install and set up GitLab using docker container gitlab/gitlab-ee:13.9.3-ee.0
  2. Verify there are no errors in the container's /var/log/gitlab/postgresql/current file
  3. Upgrade GitLab docker container to gitlab/gitlab-ee:13.10.0-ee.0
  4. Verify that the error is now being logged every 15 seconds to /var/log/gitlab/postgresql/current

What is the current bug behavior?

Upgrading GitLab from 13.9.3 to 13.10.0 introduces an error in the GitLab logs that can't be addressed.

What is the expected correct behavior?

Upgrading does not introduce errors in the postgresql log about missing database columns.

Relevant logs and/or screenshots

postgresql/current

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
$ docker exec -it gitlab gitlab-rake gitlab:env:info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.3
Redis Version:	6.0.10
Git Version:	2.29.0
Sidekiq Version:5.2.9
Go Version:	unknown


GitLab information
Version:	13.10.0-ee
Revision:	0d2d9558dba
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.5
URL:		http://docker.sr.gitlab.support
HTTP Clone URL:	http://docker.sr.gitlab.support/some-group/some-project.git
SSH Clone URL:	git@docker.sr.gitlab.support:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	13.17.0
Repository storage paths:
default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
$ docker exec -it gitlab gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
GitLab Instance / Monitoring ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

This database migration is relevant: https://gitlab.com/gitlab-org/gitlab/blob/master/db/post_migrate/20210218105431_remove_deprecated_ci_builds_columns.rb

Workaround

Run the following command to revert the database migration that introduced this error:

sudo gitlab-rake db:migrate:down VERSION=20210218105431 RAILS_ENV=production
Edited by Greg Myers