Simplify xenial migration in gitter infrastructure repository

How to make xenial and trusty infrastructure coexist?

We won't be able to migrate all 48 EC2 instances to xenial in the near (12 months) future. There are a number of tasks that are run on the whole ansible inventory (e.g. updating SSH keys for us, some basic instance provisioning).

The current state

We've created a new xenial branch with all changes related to provisioning Ubuntu 16.04 instances. This has become a long lived branch that contains code to provision

  • camoproxy
  • webapp (beta)
  • prod webapp (soon)

On top of that we have got ci-runner that runs on bionic branch

Keeping multiple long-lived branches brings the cost in form of complexity. On top of that there is a chance that the common code in these branches will get out of sync.

Solution: Use when statements to have both trusty and xenial config in master branch

You can see the decision process in details at the end of this description.

Process

The idea is to backport the changes from xenial in 3 batches:

  1. Chages that are OS version independent https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/-/merge_requests/200
  2. Camoproxy changes (potentially with the when: ansible_lsb.codename == "xenial"
  3. webapp changes

These are going to be applied by taking the code changes from xenial without history (for simplicity).

git checkout master
git checkout -b 9318-backport-non-xenial
git merge xenial
git reset 9318-backport-non-xenial

Remove all changes that are not supposed to be part of the given MR and commit.

Solution options

Criteria for choosing an option

  • transparency of the solution (how easy it is to find the piece of config I'm looking for)
  • risk of breakages in production
  • how much maintenance the solution requires
  • How easy it is to test a new setup

Option 1: Tags

  • we would tag the config and AMI would use one ansible config, never updating packages, configs or anything else.
  • good for stateless, maybe not feasible for stateful services
criteria note
transparency ✅ we could assign tag to each instance
risk of breakages in production ✅ tagging would prevent unwanted changes
maintenance cost ❌ fixes/changes affecting current infrastructure would have to be branched of an old tag, making this solution effectively the same as current long-lived branches
ease of testing new config ✅✅ tag could be replaced seamlessly by a new branch that contains work in progress

Option 2: Do nothing (keep separate branches)

This would mean keep xenial branch for the new infrastructure and master for the old infrastructure. These branches would get increasingly out of sync.

criteria note
transparency ✅ each instance would have clearly defined branch, all terradata would live on master
risk of breakages in production ✅ same as tags
maintenance cost ❌ all common fixes would have to be backported to the other branch
ease of testing new config ✅✅ tag could be replaced seamlessly by a new branch that contains work in progress

Option 3: Put ifs around all ansible statements

That would mean using the when: ansible_lsb.codename == "xenial" and when: ansible_lsb.codename == "trusty" conditions around every task that is not the same for each distribution.

Example:

- name: Install deploy-tools
  git:
    repo: git@gitlab.com:gitlab-org/gitter/deploy-tools.git
    dest: /opt/deploy-tools
    depth: 1
    version: "master"
  tags:
    - update-deploy-tools
  when: ansible_lsb.codename == "trusty"

- name: Install deploy-tools
  git:
    repo: git@gitlab.com:gitlab-org/gitter/deploy-tools.git
    dest: /opt/deploy-tools
    depth: 1
    version: "xenial"
  when: ansible_lsb.codename == "xenial"
criteria note
transparency ✅ one branch
risk of breakages in production ✅ lower thanks to having all code in one branch
maintenance cost ✅ the changes are only made in one place
ease of testing new config ❌ there needs to be some kind of "test" tag invented or there will be still a need for branches

Option 4: Copy-paste modules so we can easily delete trusty code later

For example, if we wanted the new instances to use more up to date fluentd we could copy the whole ansible/roles/fluentd into a new folder ansible/roles/fluentd-xenial and let the old instances still use the old config. The advantage of this is not mixing up old and new config in one file (compared to Option 3).

criteria note
transparency ✅ one branch
risk of breakages in production ✅ lower thanks to having all code in one branch
maintenance cost ⚠️ changes to the copied folder have to be potentially backported
ease of testing new config ❌ there needs to be some kind of "test" tag invented or there will be still a need for branches
Edited by Tomas Vik (OOO back on 2026-04-07)