Migration to 14.2 fails on push_event_payloads foreign-key problem

Summary

On migration from 14.1.x to 14.2.x, this ForeignKeyViolation error is encountered if the instance has event history longer than 3 calendar years:

PG::ForeignKeyViolation: ERROR:  insert or update on table "push_event_payloads" violates foreign key constraint "fk_36c74129da_tmp"
DETAIL:  Key (event_id)=(389) is not present in table "events".

Steps to reproduce

  • Have an instance running version 14.1.7
  • Examine the two tables events and push_event_payloads for missing records in the events table:
SELECT * FROM push_event_payloads WHERE event_id NOT IN (SELECT id FROM events);
  • If you've been running longer than 3 years (1095 days), the query will return multiple records; if setting up a test to reproduce this scenario, add entries to push_event_payloads with invalid event_id values pointing to events that have already been pruned (apparently there is a background process that limits retention of records in the events table)
  • Restart the instance with version 14.2.1 or later

Example Project

N/A

What is the current bug behavior?

Database migration halts with foreign-key violation.

What is the expected correct behavior?

Migration process should proceed, either by repairing the key violations itself, or by a fix in the 14.1.x maintainence release that eliminates this data-integrity issue.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes