Skip to content

DISCUSS: A database integrity check that prevents bugs in DB migration scripts

This issue came up from the GCP migration meeting

Nick: Job artifacts registry migration bug: https://gitlab.com/gitlab-org/gitlab-ee/issues/5646 -> gitlab-com/migration#308 (closed)

  • We lost state about which local artifacts (+traces) we have. Need to fix the migration (or revert) in time for 10.7.0 so our customers don’t suffer gprd needs a data fixup, owned by Nick

It's great that we are reviewing migration scripts vigorously but a human can naturally miss things that an automated check can suppliment consistently.

We need a way to check database integrity reliably in CI. This is a guideline on the basic approach I have done before. We should look into implementing this as part of the QA workflow. See https://www.testingexcellence.com/what-is-data-and-database-integrity-testing/

  • Build a comprehensive known gitlab test dataset (baseline), we can start with basic functionality and expand from there.
  • Multiple orgs with multiple projects
  • MRs Labels Epics and milestone configured
  • Users spanning the different permission roles
  • The dataset is setup on a feature environment database every time a MR to migration is open
  • Run the migration
  • Check to validate that data has not been mutated against the baseline, the recommended way is to use backend APIs for this. Make API calls on features that affect the users (FE). Assert that meaningful data is not deleted (if the FE calls and API we expect all the data to be present)

This is relatively simple to implement without involving the selenium testing layer.

This is also a follow back on the question I asked in #qa a few weeks ago :) https://gitlab.slack.com/archives/C3JJET4Q6/p1522346548000588?thread_ts=1522283404.000395&cid=C3JJET4Q6

Ideally this datamodel should reside in the seed database on new GDK instance, that way this test can run locally and also in CI. This should also reside in staging where the main regresssions check on data integrity nightly incase something slips through. Data integrity is important we should also be comfortable blocking merges into master if we catch this issue on staging. This way its easy to fix before letting new changes pile on top.

/cc @rymai @rspeicher @godfat @markglenfletcher @stanhu @nick.thomas @grzesiek

Edited by Martin Wortschack