Setting up periodic checks for production schema against what checked into code.

This is a follow up to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5147#note_109028825

And channeling our guidelines on how we prioritize tech debt from gitlab-com/www-gitlab-com!13596 (diffs)

How big and how fast this problem is going to be over time?

We should be looking at reducing the "interest rate" or stablizing it.

@Finotto Can we set this up and dump the results periodically somewhere? Not just pg_dump -s, it'll be nice that we could also provide the result from running rake db:schema:dump. We'll need to run this at the root of GitLab, so perhaps on a node which has GitLab installed and run sudo gitlab-rake db:schema:dump. We'll need to produced db/schema.rb.

Given the extent of the differences found in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5147#note_112776595 Thanks to @godfat and @abrandl. We should ensure that production stays in sync with migrations checked into CE and EE.

A continous check makes sense so the clean up at this scale does not happen again.

Things to consider.

  • A job to compare the checked in version of the schema against production
  • Leverage a snapshot insance instead of real production.
  • Have this run for every release then make it more frequent.
  • Define process on how to reconsile changes, if a diff is detected an MR should be created to resolve the difference.

/cc @Finotto