Consolidation of automation tools within GitLab

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem statement

There are currently multiple automation tools being used across GitLab:

Automation Tool	Owner	Actively being developed	In use	Tech Stack	In scope for consolidation
Orchestrator	groupdistribution	❌	❌	Go CLI, Terraform, Ansible	✅
gitlab-provisioner	groupdistribution	❌	❌	Terraform, Ansible	✅
ansible-gitlab-geo	groupgeo	❌	❌	Ansible	✅
GitLab Environment Toolkit	Quality	✅	✅	Terraform, Ansible	✅
Proliferate	customer success	✅	✅	Typically Terraform/Ansible, however dependent on customer request (also scripts for Puppet, CloudFormation)	✅
Demo Systems	@jeffersonmartin	✅	✅	Terraform, Ansible	❌
CloudFormation templates & QuickStart	@DarwinJS	:question:		CloudFormation (CF is Required to Target AWS Quick Starts, AWS Service Catalog, AWS Control Tower, AWS SaaS Factory)	❓

While each of these tools has its own unique niche, there is a potential for increased efficiency if we could collaborate and extend the same set of tooling. In an ideal world, we would end up with a more fully featured set of automation, which would address a broader set of use cases across the company.

Further, our customers have the same problem. If you want to deploy any reference architecture above 1K, the complexity goes up significantly and essentially requires automation to maintain. Without a set of automation tools, customers are left to build this themselves significantly slowing down time to value, increasing the effort they must invest, and also being at risk for errors as the processes are complex. One customer's runbook for upgrades is 26 pages long.

Consolidation plan

We have decided to consolidate on GitLab Environment Toolkit, previously known as Performance Environment Builder. This effort will be led by Quality, as they work extensively with automation.

We plan to iteratively consolidate on tooling, replacing the existing tools one by one. This approach allows us to retire the use of tools earlier, resulting in earlier efficiency gains.

Iteration plan:

• Consolidate Quality, Geo, Distribution tools
  • Replace ansible-gitlab-geo
  • Replace gitlab-provisioner
  • Replace Orchestrator
• Consolidate Proliferate for PS
  • Note: we only plan to consolidate the Terraform/Ansible pieces. Customers may still request Professional Services for other automation types, like CloudFormation.