Move component test scripts to development

Robert Marshallrequested to merge
atomic/refactor-dev-directory into master

What does this MR do?

Moves the developer files for the compose environment into development/ and adds make targets for each image variant.

  • Moves the component test scripts to development/tests/ with their test plans and ignore rules.
  • Removes the Cells dev environment.
  • Moves the compose files, .env, and mounted fixtures to development/compose/.
  • Adds a Makefile with up, down, and clean targets for the debian, ubi, fips, and nix stacks. Each target accepts an image tag. The targets add docker-compose.override.yml as the last compose file when it exists. The nix stack keeps gitaly-init-cgroups on the tag without -nix because that image has no nix build.

Rationale

Currently compose and test files live in the same directory path. This easily confuses new contributors and makes it difficult to understand what to maintain and where to maintain it.

Additionally, we learned in Discovery: Cells Docker Compose Maintenance Req... (#2395) • Unassigned that the Cells team will not be keeping our highly scripted Cells test updated due to migraiton to Caproni.

There are also a good number of commands required to test the various CNG variants, remove old images, and generally maintain hygiene on the developer system.

The fourth commit of the stack adds a Makefile with well documented targets to stand up, tear down, and clean up images for each of the current variants of Cloud Native GitLab.

This should reduce friction when we test Cloud Native GitLab containers.

Commit 1 of 4:

Move component test scripts to development

- Places the component test scripts under `development/tests/` with
  the test plans and ignore rules that reference them.

Closes https://gitlab.com/gitlab-org/build/CNG/-/work_items/2398

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

Commit 2 of 4:

Remove the Cells dev environment

- Removes the Cells setup script and its README.
- Removes the HTTP Router build context.
- Removes the Topology Service test scripts.
- Removes the README section and ignore rules related to Cells.

Related https://gitlab.com/gitlab-org/build/CNG/-/work_items/2395

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

Commit 3 of 4:

Move compose files and fixtures to development

- Places the compose files and the environment file under
  `development/compose/`.
- Places the mounted fixtures under `development/compose/mnt/shared/`.
- Updates the compose mounts, ignore rules, pre-commit excludes, and
  README instructions that name the new paths.

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

Commit 4 of 4:

Add make targets for the dev compose stacks

- Adds a Makefile that starts, stops, and cleans the debian, ubi, fips,
  and nix compose stacks with an optional image tag.
- Loads `docker-compose.override.yml` as the last compose file when it
  exists.
- Keeps the `gitaly-init-cgroups` image of the nix stack on the tag
  without the nix suffix.
- Documents the targets in the README.

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

Test plan

  • From development/compose, docker compose config -q exits 0 with no unset-variable warning for docker-compose.yml, docker-compose.fips.yml, and docker-compose.yml plus docker-compose.nix.yml. Every bind source in the output exists on disk.
  • docker compose config names the project compose. With that name normalized it renders the same services as master.
  • git ls-files dev prints nothing. git grep -iE 'dev/cells|dev/http-router|dev/topology-service|docker-compose\.cells|GitLab Cells' finds only a comment in gitlab-rails/scripts/lib/checks/topology_service.rb.
  • make compose up nix my-branch starts the stack with -nix tags on every CNG image except gitaly-init-cgroups:my-branch. A docker-compose.override.yml in development/compose joins the command as the last file.
  • make compose clean nix my-branch runs down -v and then removes the nine my-branch-nix images. make compose up nix bad/tag exits 2 with invalid image tag.

Related issues

Related Discovery: Cells Docker Compose Maintenance Req... (#2395) • Unassigned

Closes Spike: Organize developer tests and harnesses (#2398 - closed) • Robert Marshall

Closes Discovery: Scope automation requirements and in... (#2399 - closed) • Unassigned

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
  • The impact any change in container size has should be evaluated
  • New dependencies are managed with GitLab forked renovatebot
Edited by Robert Marshall

Merge request reports