missing column epic.relative_position (instances that migrated from CE)

Summary

We've had a couple of tickets from EE customers that are unable to create epics. The cause is that their epics table is missing the relative_position column, but this only appears to have come to light in %14.4.

The column was added in in 11.9 MR: !9283 (merged), d7417e42

  • ee/db/migrate/20190128104236_add_relative_position_to_epics.rb
  • ee/db/migrate/20190218031401_set_default_position_for_child_epics.rb

Then in 12.0, the backport_enterprise_schema migration did: create_table_if_not_exists "epics" do

It might be the case that these instances had been 'EE' at some point in the past, and switched to 'CE' - and so already had epics tables ... but if they weren't 'EE' between 11.9 and 12.0, they'd not have the relative_position column created.

Ticket links are for GitLab team members to find out more.

ticket 250642

  • customer migrated from community edition at some point.
  • issue reported at GitLab 14.4.2
  • error 'something went wrong when fetching child epics' or 'failed to set epic on this issue, please try again'
  • egrep '20190128104236|20190218031401' on their migration list did not show them as present
500 error 
RuntimeError (          Failed to implement Epic.relativePosition, tried:
          - `Types::EpicType#relative_position`, which did not exist
          - `EpicPresenter#relative_position`, which did not exist
          - Looking up hash key `:relative_position` or `"relative_position"` 
on `#`, but it wasn't a Hash
          To implement this field, define one of the methods above (and check for typos)
):
  lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'
  lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'
  lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'
  lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'
  lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'
  app/graphql/gitlab_schema.rb:40:in `multiplex'
  app/controllers/graphql_controller.rb:118:in `execute_multiplex'
  app/controllers/graphql_controller.rb:44:in `execute'
  ee/lib/gitlab/ip_address_state.rb:10:in `with'
  ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
  app/controllers/application_controller.rb:503:in `set_current_admin'
  lib/gitlab/session.rb:11:in `with_session'
  app/controllers/application_controller.rb:494:in `set_session_storage'
  lib/gitlab/i18n.rb:105:in `with_locale'
  lib/gitlab/i18n.rb:111:in `with_user_locale'
  app/controllers/application_controller.rb:488:in `set_locale'
  app/controllers/application_controller.rb:482:in `set_current_context'
  lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
  lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
  lib/gitlab/middleware/speedscope.rb:13:in `call'
  lib/gitlab/request_profiler/middleware.rb:17:in `call'
  lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
  lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
  lib/gitlab/metrics/web_transaction.rb:21:in `run'
  lib/gitlab/metrics/rack_middleware.rb:16:in `call'
  lib/gitlab/jira/middleware.rb:19:in `call'
  lib/gitlab/middleware/go.rb:20:in `call'
  lib/gitlab/etag_caching/middleware.rb:21:in `call'
  lib/gitlab/middleware/multipart.rb:173:in `call'
  lib/gitlab/middleware/read_only/controller.rb:50:in `call'
  lib/gitlab/middleware/read_only.rb:18:in `call'
  lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  lib/gitlab/middleware/request_context.rb:21:in `call'
  config/initializers/fix_local_cache_middleware.rb:11:in `call'
  lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
  lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
  lib/gitlab/middleware/release_env.rb:12:in `call'

Completed 500 Internal Server Error in 57ms (Views: 0.1ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 28142)

ticket 255884

  • GitLab 14.4.2
  • unable to use the epics featue
    • attach an issue to an epic: 500 error on graphql endpoint
    • viewing / editing epics - error
postgresql error 
==> /var/log/gitlab/postgresql/current <==
ERROR: column "relative_position" does not exist at character 214
STATEMENT: /*application:web,correlation_id:xx,endpoint_id:Projects::IssuesController#create,
db_config_name:main*/ SELECT "epic_issues"."epic_id", MIN(relative_position) AS position FROM 
((SELECT id, relative_position, parent_id, parent_id as epic_id, 'epic' as object_type
FROM "epics" WHERE "epics"."parent_id" = 13)
exceptions log - creating issue from epic 
  "exception.message": "          Failed to implement Epic.relativePosition, tried:\n\n   
       - `Types::EpicType#relative_position`, which did not exist\n          - 
`EpicPresenter#relative_position`, which did not exist\n          - Looking up hash key 
`:relative_position` or `\"relative_position\"` on `#`, but 
it wasn't a Hash\n\n          To implement this field, define one of the methods above 
(and check for typos)\n",
  "exception.backtrace": [
    "lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'",
    "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'",
    "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'",
    "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'",
    "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'",
    "app/graphql/gitlab_schema.rb:40:in `multiplex'",
    "app/controllers/graphql_controller.rb:125:in `execute_query'",
    "app/controllers/graphql_controller.rb:44:in `execute'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'",
    "app/controllers/application_controller.rb:503:in `set_current_admin'",
    "lib/gitlab/session.rb:11:in `with_session'",
    "app/controllers/application_controller.rb:494:in `set_session_storage'",
    "lib/gitlab/i18n.rb:105:in `with_locale'",
    "lib/gitlab/i18n.rb:111:in `with_user_locale'",
    "app/controllers/application_controller.rb:488:in `set_locale'",
    "app/controllers/application_controller.rb:482:in `set_current_context'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/request_profiler/middleware.rb:17:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/jira/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/go.rb:20:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/multipart.rb:173:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:21:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'"
exceptions log - active record error 
  "exception.class": "ActiveRecord::StatementInvalid",
  "exception.message": "PG::UndefinedColumn: ERROR:  column \"relative_position\" does not exist\nLINE 1: 
...(relative_position) AS position FROM ((SELECT id, relative_p...\n                                                             ^\n",
  "exception.backtrace": [
    "lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `block in read_using_load_balancer'",
    "lib/gitlab/database/load_balancing/load_balancer.rb:110:in `block in read_write'",
    "lib/gitlab/database/load_balancing/load_balancer.rb:170:in `retry_with_backoff'",
    "lib/gitlab/database/load_balancing/load_balancer.rb:108:in `read_write'",
    "lib/gitlab/database/load_balancing/connection_proxy.rb:98:in `read_using_load_balancer'",
    "lib/gitlab/database/load_balancing/connection_proxy.rb:47:in `select_all'",
    "lib/gitlab/relative_positioning/item_context.rb:108:in `calculate_relative_position'",
    "lib/gitlab/relative_positioning/item_context.rb:30:in `block in min_relative_position'",
    "lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'",
    "lib/gitlab/relative_positioning/item_context.rb:30:in `min_relative_position'",
    "lib/gitlab/relative_positioning/mover.rb:22:in `move_to_start'",
    "app/models/concerns/relative_positioning.rb:165:in `move_to_start'",
    "ee/app/models/concerns/epic_tree_sorting.rb:67:in `move_to_start'",
    "ee/app/services/epic_issues/create_service.rb:15:in `relate_issuables'",
    "app/services/issuable_links/create_service.rb:56:in `block in link_issuables'",
    "app/services/issuable_links/create_service.rb:55:in `each'",
    "app/services/issuable_links/create_service.rb:55:in `link_issuables'",
    "ee/app/services/ee/issuable_links/create_service.rb:17:in `link_issuables'",
    "app/services/issuable_links/create_service.rb:51:in `create_links'",
    "app/services/issuable_links/create_service.rb:28:in `execute'",
    "ee/app/services/ee/issues/base_service.rb:27:in `assign_epic'",
    "ee/app/services/ee/issues/base_service.rb:104:in `handle_epic_change'",
    "ee/app/services/ee/issues/base_service.rb:90:in `handle_changes'",
    "app/services/issues/create_service.rb:54:in `handle_changes'",
    "app/services/issuable_base_service.rb:219:in `create'",
    "ee/app/services/ee/issues/create_service.rb:12:in `create'",
    "app/services/issues/create_service.rb:25:in `execute'",
    "app/services/concerns/rate_limited_service.rb:91:in `execute'",
    "ee/app/services/ee/issues/create_service.rb:24:in `execute'",
    "lib/api/issues.rb:278:in `block (2 levels) in '",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
    "lib/api/api_guard.rb:213:in `call'",
    "lib/gitlab/middleware/speedscope.rb:13:in `call'",
    "lib/gitlab/request_profiler/middleware.rb:17:in `call'",
    "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
    "lib/gitlab/jira/middleware.rb:19:in `call'",
    "lib/gitlab/middleware/go.rb:20:in `call'",
    "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
    "lib/gitlab/middleware/multipart.rb:173:in `call'",
    "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
    "lib/gitlab/middleware/read_only.rb:18:in `call'",
    "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
    "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
    "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
    "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
    "lib/gitlab/middleware/request_context.rb:21:in `call'",
    "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
    "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
    "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
    "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'",
    "lib/gitlab/middleware/release_env.rb:12:in `call'"

Workaround

  1. Start a SQL console in the database

    sudo gitlab-psql

  2. Create the missing column

    ALTER TABLE epics ADD COLUMN IF NOT EXISTS relative_position  INTEGER;
\q

  3. Backfill relative_position for existing epics

UPDATE epics SET relative_position=id*500;

Steps to reproduce

Remove the relative_position column from the public.epics table.

Example Project

What is the current bug behavior?

Customers that have migrated from CE at some point might not have the epics.relative_position column, and then cannot create epics from %14.4

What is the expected correct behavior?

The column is present.

Not sure what the sequence events would need to be for instances to be missing this column.

Relevant logs and/or screenshots

image

image

See twisties in the description:

  • column "relative_position" does not exist in PostgreSQL log
  • Failed to implement Epic.relativePosition from Rails
  • PG::UndefinedColumn: ERROR: column \"relative_position\" does not exist\nLINE 1: ...(relative_position) from ActiveRecord

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Jan Provaznik