NoMethodError undefined method instance for #<GithubService:0x00007f2e7852cb48>
Summary
We had a few customers reporting that when going from a version 12.7 to 12.9.3 and another customer upgrading to 12.7.5 (not sure from which version) and now after upgrade to version 12.9.1 the same error occurs on either GitHubService
or JiraService
. It looks like the column instance
is missing in the table services
.
It was added here eaf34116 but not sure why migrations did not add it to the services table.
Output from migrate:status
shows that all the migrations are up. Tried Rerunning below migrations, but received an error
https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/migrate/20200310132654_add_instance_to_services.rb
https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/migrate/20200310135823_add_index_to_service_unique_instance_per_type.rb
Error:
[root@cxpxgit11mst001 ~#] gitlab-rake db:migrate:redo VERSION=20200310132654
== 20200310132654 AddInstanceToServices: reverting ============================
-- remove_column(:services, :instance)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedColumn: ERROR: column "instance" of relation "services" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200310132654_add_instance_to_services.rb:15:in `down'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "instance" of relation "services" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200310132654_add_instance_to_services.rb:15:in `down'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::UndefinedColumn: ERROR: column "instance" of relation "services" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200310132654_add_instance_to_services.rb:15:in `down'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate:down
(See full trace by running task with --trace)
services table dump
CREATE TABLE public.services (
id integer NOT NULL,
type character varying(255),
title character varying(255),
project_id integer,
created_at timestamp without time zone,
updated_at timestamp without time zone,
active boolean DEFAULT false NOT NULL,
properties text,
template boolean DEFAULT false,
push_events boolean DEFAULT true,
issues_events boolean DEFAULT true,
merge_requests_events boolean DEFAULT true,
tag_push_events boolean DEFAULT true,
note_events boolean DEFAULT true NOT NULL,
category character varying DEFAULT 'common'::character varying NOT NULL,
"default" boolean DEFAULT false,
wiki_page_events boolean DEFAULT true,
pipeline_events boolean DEFAULT false NOT NULL,
confidential_issues_events boolean DEFAULT true NOT NULL,
commit_events boolean DEFAULT true NOT NULL,
job_events boolean DEFAULT false NOT NULL,
confidential_note_events boolean DEFAULT true,
deployment_events boolean DEFAULT false NOT NULL,
description character varying(500),
comment_on_event_enabled boolean DEFAULT true NOT NULL
);
Steps to reproduce
- Could not reproduce on my test instance.
What is the current bug behavior?
Bug (500 error) happens in 2 scenarios
-
When accessing admin/application_settings/services
-
When updating Jira integration settings, project level
Relevant logs and/or screenshots
NoMethodError (undefined method `instance?' for #<GithubService:0x00007f2e7852cb48>
Did you mean? instance_of?):
app/models/service.rb:39:in `block in <class:Service>'
app/controllers/admin/services_controller.rb:37:in `block in services_templates'
app/controllers/admin/services_controller.rb:35:in `map'
app/controllers/admin/services_controller.rb:35:in `services_templates'
app/controllers/admin/services_controller.rb:10:in `index'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'
app/controllers/application_controller.rb:479:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:470:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:464:in `set_locale'
lib/gitlab/error_tracking.rb:34:in `with_context'
app/controllers/application_controller.rb:555:in `sentry_context'
lib/gitlab/application_context.rb:46:in `block in use'
lib/gitlab/application_context.rb:46:in `use'
lib/gitlab/application_context.rb:19:in `with_context'
app/controllers/application_controller.rb:455:in `set_current_context'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:122:in `call'
lib/gitlab/middleware/read_only/controller.rb:53:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Completed 304 Not Modified in 0ms (Allocations: 5)
This seems to also impact trying to enable services for certain project, for example trying to enable the jira service for a project: /-/services/jira/edit
NoMethodError (undefined method `instance?' for #<JiraService:0x00007f2e7734c5b8>
Did you mean? instance_of?):
app/models/service.rb:35:in `block in <class:Service>'
app/controllers/projects/services_controller.rb:55:in `service_test_response'
app/controllers/projects/services_controller.rb:46:in `test'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'
app/controllers/application_controller.rb:479:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:470:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:464:in `set_locale'
lib/gitlab/error_tracking.rb:34:in `with_context'
app/controllers/application_controller.rb:555:in `sentry_context'
lib/gitlab/application_context.rb:46:in `block in use'
lib/gitlab/application_context.rb:46:in `use'
lib/gitlab/application_context.rb:19:in `with_context'
app/controllers/application_controller.rb:455:in `set_current_context'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
ee/lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:122:in `call'
lib/gitlab/middleware/read_only/controller.rb:53:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Completed 304 Not Modified in 0ms (Allocations: 5)
Results of GitLab environment info
v12.9.4 omnibus installation
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
To fix this follow the below steps:
## Enter the database
sudo gitlab-rails dbconsole
## While inside the database
ALTER TABLE services ADD COLUMN instance boolean DEFAULT false NOT NULL;
Check whether the issue is still present.
You would also want to try to remove and rerun the migration to add the index:
## Enter the Postgres console
sudo gitlab-rails dbconsole
## Delete the 20200310135823 migration
DELETE FROM schema_migrations WHERE schema_migrations.version = '20200310135823';
DELETE FROM schema_migrations WHERE schema_migrations.version = '20200310132654';
## Exit the console and then run
sudo gitlab-rake db:migrate:up VERSION=20200310135823
sudo gitlab-rake db:migrate:up VERSION=20200310132654
Tickets: https://gitlab.zendesk.com/agent/tickets/154362 https://gitlab.zendesk.com/agent/tickets/153126