Create orchestrator for setting up cng environment
As our MVP for test-on-cng
shows promising results regarding the stability and reliability of the test pipeline, we are now concentrating on building an orchestrator tool for local execution.
Current MVP for test-on-cng pipeline uses simple bash script to deploy kind cluster and deploy cng
on it.
This works fine for an MVP but is not scalable for adding different configuration types and doesn't support local execution.
Orchestrator similar to gitlab-qa
should be created but addressing all the architectural and design flaws of gitlab-qa
like:
- inevitable coupling with tests but having externally located code and separate release cycle
- inconsistent handling of positional cli arguments
- lack of built in documentation (lack of more advanced library usage for parsing cli arguments)
- fairly complex code design that creates a lot of nested block passing
- app deployment coupling to internal docker network which makes it incredibly hard to use for local development
- lack of built in log tailing
Requirements
- Located alongside
e2e
test code ingitlab
repository so it is always easy to develop cross dependent functionalities, like adding new configuration and tests as a single merge request - CLI has to be self documented and ideally not requiring extensive external documentation,
help
command or--help
flag has to cover most if not all documentation on how to use the tool - Orchestrator has to handle environment deployment only to not bloat the scope of the tool.
- Handle
kind
cluster setup for local and ci usecase - Handle
helm
commands needed for environment installation on k8s cluster - Additional setup via raw
kubectl
commands if necessary
- Handle
- Reuse as many tools as possible already present in
qa/Gemfile.lock
to reduce amount of gems being installed in ci jobs
Implementation
CLI argument parsing
There are several options to use as a CLI framework.
-
Build in rubyOptionParser
Not really an option for anything but simplest script. Requires a lot of boilerplate and lacks functionality of proper framework for building CLI apps
-
Very popular in ruby world, but has few downsides.
- The main downside is architecture where single class does not correspond to a single command but rather every method in a class corresponds to a command. This makes it less convenient to have OOP structure and split code where every command is encapsulated in it's own class.
- Lack of automatic handling of `-h`, `--help` class
-
gem maintained by shopify which has a good track record of creating open source ruby tools. Seems to have better support for splitting code in to commands and even allows adding type systems. Main downsides:
- Lacking documentation
- Doesn't seem to support subcommands
-
Has a good and easily usable api. Downsides:
- Last release 2 years ago, doesn't seem to be maintained that well though some activity in issue tracker is still present
Decision
Use Thor gem as main cli argument parser and command creator because it seems to be the most maintained and supports mostly all options that could be required for cli creation
Cluster setup and deployment
-
kubectl
- kubeclient seems to be good candidate of handling low level kubectl commands. It comes with support to stream logs and other operations around kubernetes which otherwise would require a more complex custom wrapper development. -
helm
,kind
- it seems that there aren't well maintained wrappers for driving `helm` and `kind` commands. Custom wrapper around shell invocation of these commands will have to be created.
Decision
-
Use kubeclient gem for interaction withOpted to just create a thin wrapper aroundkubeclient
given this will require most complex invocations including custom object creation and tailing logskubectl
executable. It seamed likekubeclient
might not be as useful because we don't really need more advanced capabilities - Create thin wrappers for
kind
andhelm
commands using standardopen3
library
Code location and release cycle
To better support development in tandem with E2E tests, like adding new scenario types and running tests against them, code should be located in the same repository functional E2E tests are. This does have a few downsides:
- This comes with a small downside that release cycle is coupled with `gitlab` itself. Overall, this should not be a big problem because usage will not depend on having to release the gem. For occasional external usage, versioning together with main `gitlab` app has added benefit of implicitly showing what version of `gitlab` app is supported by orchestrator.
- It is also a bit harder to automate version update if `cng` is also included in main `qa` Gemfile. When version is updated, it requires to update `Gemfile.lock`.
Decision
Orchestrator is implemented within gems
directory of gitlab repository. For now release is not planned given use case is predominantly for development of gitlab rails app so it can be used from within the repository folder
Commands
Initial implementation should provide several commands for the cli.
-
doctor
- perform sanity check for all the dependent tools to be available in the environment -
create
-
cluster
- setup kubernetes cluster usingkind
with namespace and dependent objects-
--ci
- force CI specific cluster setup
-
-
deployment
- create CNG deployment using helm-
--type
- pass a predefined type of deployment to support various configurations -
--values
- allow to pass custom values to helm deployment -
--ci
- force CI specific deployment type
-
-
-
destroy
-
cluster
- remove kubernetes cluster -
deployment
- uninstall CNG deployment
-
-
log
-
cluster
- print cluster logs -
events
- print all deployment object events -
app
- print application logs
-