Skip to content

Gitlab housekeeper initial implementation with finalize BBM

Dylan Griffith requested to merge gitlab-housekeeper into master

What does this MR do and why?

This is an initial implementation of the problem and solutions described in this blueprint !134487 (merged) and it was extracted from a POC implementation in !134485 (closed) which contained other work we don't want to include in the first iteration.

This is a gem which can be run locally or in CI to do static and dynamic analysis of the codebase and using a list of predefined "keeps" it will automatically create merge requests for things that developers would have otherwise needed to remember to do themselves.

It is analogous to a mix of rubocop -a and GitLab Dependency Bot.

The word "keep" is used to describe a specific rule to apply to the code to match a required change and actually edit the code. The word "keep" was chosen as it sounds like "cop" and is very similar to implementing a rubocop rule as well as code to autocorrect the rule.

Today we have a complete implementation of the OverdueFinalizeBackgroundMigration keep which finds all batched background migrations added before the last required stop upgrade point and then checks in Postgres.ai to confirm that this migration is complete on GitLab.com and then generates a migration to finalize the batched background migration. It has generated many migration MRs already.

How the code is organized

The code is organized in a very similar way to RuboCop in that we have a overall gem called gitlab-housekeeper that contains the generic logic of looping over all keeps (analogous to Cops) which are rules for how to detect changes that can be made to the code and then actually how to correct them.

This MR includes the following high level pieces:

  1. gems/gitlab-housekeeper which is the generic gem. It should be responsible for running keeps which will return a Change object. The housekeeper then uses this change object to create a git branch, push the branch, create a merge request
    1. The gitlab-housekeeper command which just calls the Runner
    2. The Runner which contains the high level operation of the gitlab-housekeeper gem. It looks at the options provided and runs the correct keeps. It then takes the Change results from the keeps and creates merge requests
    3. The Git client is just a wrapper around shelling out to git
    4. The GitlabClient is just a thin wrapper around HTTParty to make HTTP requests to create merge requests
    5. The Shell class is just a wrapper around shelling out. Mainly this is useful for stubbing in specs for all the other code.
  2. The keeps/ directory which is analogous to our rubocop directory. It contains implementations of specific keeps and also a helpers directory that we expect to be used in multiple keeps
    1. keeps/helpers/postgres_ai.rb is a wrapper around the PG client that we can use to encapsulate code that looks up things from the current state of our production database. For now it is used to check if a batched background migration is finished in production
    2. keeps/overdue_finalize_background_migration.rb : This is our first keep we've implemented. It has been used successfully to generate at least 15 merged migrations already. This is a pretty complicated piece of code but at a high level it is looking for db/docs/batched_background_migrations files and trying to find ones that were added before the last required stop and have already finished on GitLab.com . This means it's safe to finalize them. It then generates a migration with the change necessary to finalize the BBM.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Dylan Griffith

Merge request reports