13.9.3-ee db migration issue (zero downtime)

Hi,

We upgraded to version 13.9.3-ee after upgrading to 13.0.14 . It's only been in the last few weeks, users had noticed that they were occasionally getting 500 errors after logging in, though not all the time.

After looking at health check , it seems the DB migrations were not all successful , so advises to run rails db:migrate RAILS_ENV=production

This results in the below:

== 20200518114540 ScheduleFixRubyObjectInAuditEvents: migrating ===============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:audit_events, :id, {:where=>"details ~~ '%ruby/object%'", :name=>"index_audit_events_on_ruby_object_in_details", :algorithm=>:concurrently})
   -> 0.0041s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:audit_events, :id, {:where=>"details ~~ '%ruby/object%'", :name=>"index_audit_events_on_ruby_object_in_details", :algorithm=>:concurrently})
-- execute("RESET ALL")
   -> 0.0002s
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::FeatureNotSupported: ERROR:  cannot create index on partitioned table "audit_events" concurrently
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:150:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:336:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:149:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200518114540_schedule_fix_ruby_object_in_audit_events.rb:24:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::FeatureNotSupported: ERROR:  cannot create index on partitioned table "audit_events" concurrently
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:150:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:336:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:149:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200518114540_schedule_fix_ruby_object_in_audit_events.rb:24:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::FeatureNotSupported: ERROR:  cannot create index on partitioned table "audit_events" concurrently
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:150:in `block in add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:336:in `disable_statement_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:149:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20200518114540_schedule_fix_ruby_object_in_audit_events.rb:24:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

The postgres version is 11.10 (We have not yet upgraded to 12 because we use the Geo feature which as I understand it's not supported in PG12)

We have tried runniing gitlab-ctl reconfigure which does not resolve the issue.

Not sure if this is PG version related from the FeatureNotSupported errors .

PG permissions -

gitlabhq_production=# \l
                                             List of databases
        Name         |    Owner    | Encoding |   Collate   |    Ctype    |        Access privileges        
---------------------+-------------+----------+-------------+-------------+---------------------------------
 gitlabhq_production | gitlab      | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 postgres            | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0           | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/"gitlab-psql"               +
                     |             |          |             |             | "gitlab-psql"=CTc/"gitlab-psql"
 template1           | gitlab-psql | UTF8     | en_US.UTF-8 | en_US.UTF-8 | "gitlab-psql"=CTc/"gitlab-psql"+
                     |             |          |             |             | =c/"gitlab-psql"

gitlabhq_production=#  \du+ gitlab
                  List of roles
 Role name | Attributes | Member of | Description 
-----------+------------+-----------+-------------
 gitlab    |            | {}        | 

I don't know if there is a way to clear this and try and do the migration again ? Otherwise our gitlab instance seems to be running well with no other known issues. However - we would like to resolve this issue so we can plan future upgrade paths.

Many thanks,