Use a fully default gitlab.yml for running GitLab tests

Overview

Currently, running GitLab tests are relying on the values of config/gitlab.yml, which can be configured with GDK, and everyone can be using a different config, even on CI it's not using the same gitlab.yml.

This makes running tests not fully consistent and can behave much differently depending on an intentionally ignored config file in the repository, making things like "It works on my computer" more common than needed.

  • It can work on CI but fail locally, for me, or other
    • We might merge something that can break for others, or me
  • It can work locally, for me, or other, but fail on CI
    • It might be difficult to debug, and would possibly need to grab the config from CI to reproduce failures locally

This was discovered at: gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#543 (comment 2865598509)

Long story short, for the cell config:

  • We disable it on CI
  • We enable it on GDK by default

We merged gitlab!210472 (merged) and it works on CI but can break a lot of tests when cell is enabled locally. We reverted it: gitlab!211332 (merged)

To mitigate this for now, we try to make sure cell is always disabled in tests: gitlab!211389 (merged)

Of course we also disable the feature flag in tests globally when we're reapplying that: gitlab!211568 (merged)

It turns out that we couldn't fully rely on RSpec mocks to stub configs though: gitlab!211389 (comment 2868399178)

We're using let_it_be extensively, which runs in before(:all) which can't see stubs inside before. This means we can't rely on mocking configs to ensure that the tests are running with a consistent config.

It'll be much easier if we just stick with a fixed gitlab.yml:

  • More consistent config across the whole tests
  • Easier to spot difference between other developers. We can just compare with the default generated one
  • Separate the concern between actually running an instance and running tests

Impacted categories

The following categories relate to this issue:

Steps to replicate (optional)

  • Find a test which relies on a specific value in gitlab.yml, or a config that has global impacts
  • Change the config from default and run the tests
  • See how it acts differently without actually changing any code

Proposal (optional)

Use a fixed gitlab.yml that we use for running tests, on everywhere.

This does not apply to end-to-end tests, which by definition will need to be configured with different gitlab.yml, so it's out of the scope of this. We cannot use stubs in end-to-end tests either.

Environment (optional)

  • Operating system name: <!-- output of `uname -a` command -->
  • Architecture: <!-- output of `arch` or `uname -m` command -->
  • The contents of your gdk.yml (if any)
  • Ruby version: <!-- output of `ruby --version` command -->
  • GDK version: <!-- output of `git rev-parse --short HEAD` command -->
Edited by Lin Jen-Shin