missing column epic.relative_position (instances that migrated from CE)
Summary
We've had a couple of tickets from EE customers that are unable to create epics. The cause is that their epics
table is missing the relative_position
column, but this only appears to have come to light in %14.4.
The column was added in in 11.9 MR: !9283 (merged), d7417e42
ee/db/migrate/20190128104236_add_relative_position_to_epics.rb
ee/db/migrate/20190218031401_set_default_position_for_child_epics.rb
Then in 12.0, the backport_enterprise_schema migration did: create_table_if_not_exists "epics" do
It might be the case that these instances had been 'EE' at some point in the past, and switched to 'CE' - and so already had epics
tables ... but if they weren't 'EE' between 11.9 and 12.0, they'd not have the relative_position
column created.
Ticket links are for GitLab team members to find out more.
- customer migrated from community edition at some point.
- issue reported at GitLab 14.4.2
- error 'something went wrong when fetching child epics' or 'failed to set epic on this issue, please try again'
-
egrep '20190128104236|20190218031401'
on their migration list did not show them as present
500 error
RuntimeError ( Failed to implement Epic.relativePosition, tried: - `Types::EpicType#relative_position`, which did not exist - `EpicPresenter#relative_position`, which did not exist - Looking up hash key `:relative_position` or `"relative_position"` on `#`, but it wasn't a Hash To implement this field, define one of the methods above (and check for typos) ): lib/gitlab/graphql/present/field_extension.rb:18:in `resolve' lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing' lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace' lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing' lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace' app/graphql/gitlab_schema.rb:40:in `multiplex' app/controllers/graphql_controller.rb:118:in `execute_multiplex' app/controllers/graphql_controller.rb:44:in `execute' ee/lib/gitlab/ip_address_state.rb:10:in `with' ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address' app/controllers/application_controller.rb:503:in `set_current_admin' lib/gitlab/session.rb:11:in `with_session' app/controllers/application_controller.rb:494:in `set_session_storage' lib/gitlab/i18n.rb:105:in `with_locale' lib/gitlab/i18n.rb:111:in `with_user_locale' app/controllers/application_controller.rb:488:in `set_locale' app/controllers/application_controller.rb:482:in `set_current_context' lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call' lib/gitlab/middleware/rails_queue_duration.rb:33:in `call' lib/gitlab/middleware/speedscope.rb:13:in `call' lib/gitlab/request_profiler/middleware.rb:17:in `call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call' lib/gitlab/metrics/rack_middleware.rb:16:in `block in call' lib/gitlab/metrics/web_transaction.rb:21:in `run' lib/gitlab/metrics/rack_middleware.rb:16:in `call' lib/gitlab/jira/middleware.rb:19:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:21:in `call' lib/gitlab/middleware/multipart.rb:173:in `call' lib/gitlab/middleware/read_only/controller.rb:50:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/same_site_cookies.rb:27:in `call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call' lib/gitlab/middleware/request_context.rb:21:in `call' config/initializers/fix_local_cache_middleware.rb:11:in `call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call' lib/gitlab/middleware/release_env.rb:12:in `call'Completed 500 Internal Server Error in 57ms (Views: 0.1ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 28142)
- GitLab 14.4.2
- unable to use the epics featue
- attach an issue to an epic: 500 error on graphql endpoint
- viewing / editing epics - error
postgresql error
==> /var/log/gitlab/postgresql/current <== ERROR: column "relative_position" does not exist at character 214 STATEMENT: /*application:web,correlation_id:xx,endpoint_id:Projects::IssuesController#create, db_config_name:main*/ SELECT "epic_issues"."epic_id", MIN(relative_position) AS position FROM ((SELECT id, relative_position, parent_id, parent_id as epic_id, 'epic' as object_type FROM "epics" WHERE "epics"."parent_id" = 13)
exceptions log - creating issue from epic
"exception.message": " Failed to implement Epic.relativePosition, tried:\n\n - `Types::EpicType#relative_position`, which did not exist\n - `EpicPresenter#relative_position`, which did not exist\n - Looking up hash key `:relative_position` or `\"relative_position\"` on `#`, but it wasn't a Hash\n\n To implement this field, define one of the methods above (and check for typos)\n", "exception.backtrace": [ "lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'", "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'", "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'", "app/graphql/gitlab_schema.rb:40:in `multiplex'", "app/controllers/graphql_controller.rb:125:in `execute_query'", "app/controllers/graphql_controller.rb:44:in `execute'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'", "app/controllers/application_controller.rb:503:in `set_current_admin'", "lib/gitlab/session.rb:11:in `with_session'", "app/controllers/application_controller.rb:494:in `set_session_storage'", "lib/gitlab/i18n.rb:105:in `with_locale'", "lib/gitlab/i18n.rb:111:in `with_user_locale'", "app/controllers/application_controller.rb:488:in `set_locale'", "app/controllers/application_controller.rb:482:in `set_current_context'", "lib/gitlab/middleware/speedscope.rb:13:in `call'", "lib/gitlab/request_profiler/middleware.rb:17:in `call'", "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'", "lib/gitlab/jira/middleware.rb:19:in `call'", "lib/gitlab/middleware/go.rb:20:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/multipart.rb:173:in `call'", "lib/gitlab/middleware/read_only/controller.rb:50:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:21:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'"
exceptions log - active record error
"exception.class": "ActiveRecord::StatementInvalid", "exception.message": "PG::UndefinedColumn: ERROR: column \"relative_position\" does not exist\nLINE 1: ...(relative_position) AS position FROM ((SELECT id, relative_p...\n ^\n", "exception.backtrace": [ "lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'", "lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'", "lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'", "lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'", "lib/gitlab/database/load_balancing/connection_proxy.rb:98:in `read_using_load_balancer'", "lib/gitlab/database/load_balancing/connection_proxy.rb:47:in `select_all'", "lib/gitlab/relative_positioning/item_context.rb:108:in `calculate_relative_position'", "lib/gitlab/relative_positioning/item_context.rb:30:in `block in min_relative_position'", "lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'", "lib/gitlab/relative_positioning/item_context.rb:30:in `min_relative_position'", "lib/gitlab/relative_positioning/mover.rb:22:in `move_to_start'", "app/models/concerns/relative_positioning.rb:165:in `move_to_start'", "ee/app/models/concerns/epic_tree_sorting.rb:67:in `move_to_start'", "ee/app/services/epic_issues/create_service.rb:15:in `relate_issuables'", "app/services/issuable_links/create_service.rb:56:in `block in link_issuables'", "app/services/issuable_links/create_service.rb:55:in `each'", "app/services/issuable_links/create_service.rb:55:in `link_issuables'", "ee/app/services/ee/issuable_links/create_service.rb:17:in `link_issuables'", "app/services/issuable_links/create_service.rb:51:in `create_links'", "app/services/issuable_links/create_service.rb:28:in `execute'", "ee/app/services/ee/issues/base_service.rb:27:in `assign_epic'", "ee/app/services/ee/issues/base_service.rb:104:in `handle_epic_change'", "ee/app/services/ee/issues/base_service.rb:90:in `handle_changes'", "app/services/issues/create_service.rb:54:in `handle_changes'", "app/services/issuable_base_service.rb:219:in `create'", "ee/app/services/ee/issues/create_service.rb:12:in `create'", "app/services/issues/create_service.rb:25:in `execute'", "app/services/concerns/rate_limited_service.rb:91:in `execute'", "ee/app/services/ee/issues/create_service.rb:24:in `execute'", "lib/api/issues.rb:278:in `block (2 levels) in '", "ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", "ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'", "lib/api/api_guard.rb:213:in `call'", "lib/gitlab/middleware/speedscope.rb:13:in `call'", "lib/gitlab/request_profiler/middleware.rb:17:in `call'", "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'", "lib/gitlab/jira/middleware.rb:19:in `call'", "lib/gitlab/middleware/go.rb:20:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/multipart.rb:173:in `call'", "lib/gitlab/middleware/read_only/controller.rb:50:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:21:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'", "lib/gitlab/middleware/release_env.rb:12:in `call'"
Workaround
-
Start a SQL console in the database
sudo gitlab-psql
-
Create the missing column
ALTER TABLE epics ADD COLUMN IF NOT EXISTS relative_position INTEGER; \q
-
Backfill relative_position for existing epics
UPDATE epics SET relative_position=id*500;
Steps to reproduce
Remove the relative_position
column from the public.epics table.
Example Project
What is the current bug behavior?
Customers that have migrated from CE at some point might not have the epics.relative_position
column, and then cannot create epics from %14.4
What is the expected correct behavior?
The column is present.
Not sure what the sequence events would need to be for instances to be missing this column.
Relevant logs and/or screenshots
See twisties in the description:
-
column "relative_position" does not exist
in PostgreSQL log -
Failed to implement Epic.relativePosition
from Rails -
PG::UndefinedColumn: ERROR: column \"relative_position\" does not exist\nLINE 1: ...(relative_position)
from ActiveRecord
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)