Investigate the possibilities of Krake's E2E_tests performance improvement

The Krake E2E test pipeline stage takes a long time (the last master branch run takes ~102min) which affects (slows down) the development process.

See the last (parsed) ansible profiling report, that summarizes ansible task durations in minutes and also summarizes playbook duration also in minutes.

Report: 
0_PLAY [Launch Krake networks]:
  0_TASK [Create Heat stack for network]: 0.4
  1_TASK [Add router UUID to inventory]: 0.0
  TOTAL_DURATION_[min]: 0.4
1_PLAY [Launch network gateway instances]:
  2_TASK [Create Heat stack for gateway]: 0.43
  3_TASK [Update host variables of gateway]: 0.0
  TOTAL_DURATION_[min]: 0.43
2_PLAY [Setup network gateway instances]:
  '4_TASK [authorized-keys : Add authorized keys]': 0.0
  5_TASK [Wait for gateway to show up]: 0.29
  6_TASK [Gathering facts]: 0.07
  '7_TASK [update_system : Update apt repository and cache]': 0.18
  '8_TASK [update_system : Upgrade all packages]': 2.07
  '9_TASK [update_system : Check if a reboot is needed]': 0.05
  '10_TASK [update_system : Reboot the machine if the kernel has been updated]': 0.64
  11_TASK [Install apt packages]: 0.33
  12_TASK [Check if wireguard private key was generated]: 0.04
  13_TASK [Generate wireguard private key on the gateway]: 0.04
  14_TASK [Read the private key]: 0.04
  15_TASK [set_fact]: 0.0
  16_TASK [Check if public key was generated]: 0.04
  17_TASK [Generate wireguard public key on the gateway]: 0.04
  18_TASK [Read the public key]: 0.04
  19_TASK [set_fact]: 0.0
  20_TASK [Create wireguard configuration on the gateway]: 0.09
  21_TASK [Create the wireguard for each peer in host file]: 0.0
  22_TASK [Start and enable the wireguard service if the configuration changed]: 0.06
  23_TASK [Enable IPv4 forwarding]: 0.04
  24_TASK [Create ferm (iptables) configuration]: 0.09
  25_TASK [Restart ferm service]: 0.05
  TOTAL_DURATION_[min]: 4.21
3_PLAY [Launch Minikube VMs]:
  26_TASK [Create Heat stack for Minikube VM]: 0.55
  27_TASK [Update host variables of Minikube VM]: 0.0
  TOTAL_DURATION_[min]: 0.56
4_PLAY [Setup Minikube VMs]:
  '28_TASK [authorized-keys : Add authorized keys]': 0.0
  29_TASK [Wait for Minikube VM to show up]: 0.21
  30_TASK [Gathering facts]: 0.07
  '31_TASK [update_system : Update apt repository and cache]': 0.29
  '32_TASK [update_system : Upgrade all packages]': 2.99
  '33_TASK [update_system : Check if a reboot is needed]': 0.06
  '34_TASK [update_system : Reboot the machine if the kernel has been updated]': 0.7
  35_TASK [Install system packages]: 0.98
  36_TASK [Configure Docker daemon MTU]: 0.11
  37_TASK [Restart docker to reload the configuration]: 0.06
  38_TASK [Add hostname to /etc/hosts]: 0.05
  39_TASK [Set the machine's hostname]: 0.11
  40_TASK [Create 'wheel' group]: 0.05
  41_TASK [Allow members of 'wheel' group to have passwordless sudo]: 0.05
  42_TASK [Add wheel and docker group to the ubuntu user]: 0.06
  43_TASK [Create a hushlogin for user ubuntu]: 0.11
  44_TASK [Get kubectl latest version]: 0.0
  45_TASK [Get minikube and kubectl installation binaries]: 0.18
  46_TASK [Change binaries ownership, group and permissions]: 0.11
  TOTAL_DURATION_[min]: 6.2
5_PLAY [Start Minikube clusters]:
  47_TASK [Gathering Facts]: 0.07
  48_TASK [Creates minikube and kube directories]: 0.1
  49_TASK [Touch a kube config file]: 0.05
  50_TASK [Stop Minikube]: 0.05
  51_TASK [Get Minikube status]: 0.05
  52_TASK [Start Minikube]: 0.86
  53_TASK [Change ownership of a minikube and kube directories]: 0.11
  54_TASK [Get Minikube status]: 0.06
  55_TASK [Ensure Minikube cluster is running]: 0.0
  56_TASK [Ensure kubectl is configured correctly]: 0.0
  TOTAL_DURATION_[min]: 1.36
6_PLAY [Setup Minikube clusters]:
  57_TASK [Gathering Facts]: 0.07
  TOTAL_DURATION_[min]: 0.07
7_PLAY [Download cluster certificates]:
  58_TASK [Gathering Facts]: 0.07
  59_TASK [Download cluster certificates]: 0.22
  60_TASK [Generate kube config file]: 0.02
  TOTAL_DURATION_[min]: 0.31
8_PLAY [Launch Krake VMs]:
  61_TASK [Create Heat stack for Krake VM]: 0.42
  62_TASK [Update host variables]: 0.0
  TOTAL_DURATION_[min]: 0.42
9_PLAY [Setup Krake VMs]:
  63_TASK [Wait for Krake VM to show up]: 0.32
  64_TASK [Gathering facts]: 0.07
  '65_TASK [update_system : Update apt repository and cache]': 0.19
  '66_TASK [update_system : Upgrade all packages]': 2.12
  '67_TASK [update_system : Check if a reboot is needed]': 0.06
  '68_TASK [update_system : Reboot the machine if the kernel has been updated]': 0.7
  69_TASK [Install system packages]: 1.01
  70_TASK [Upgrade pip to latest version]: 0.1
  71_TASK [Add GPG key for kubectl]: 0.07
  72_TASK [Add APT repository for kubectl]: 0.2
  73_TASK [Install kubectl]: 0.19
  74_TASK [Create 'wheel' group]: 0.05
  75_TASK [Allow members of 'wheel' group to have passwordless sudo]: 0.05
  76_TASK [Create user 'krake']: 0.05
  77_TASK [Create a hushlogin for user krake]: 0.11
  78_TASK [Configure Docker daemon MTU]: 0.1
  79_TASK [Restart docker to reload the configuration]: 0.06
  80_TASK [Add hostname to /etc/hosts]: 0.05
  81_TASK [Set the machine's hostname]: 0.11
  82_TASK [Create .ssh directory]: 0.05
  83_TASK [Copy 'authorized_keys' from user 'ubuntu' to user 'krake']: 0.05
  TOTAL_DURATION_[min]: 5.71
10_PLAY [Manage Krake VM]:
  84_TASK [Gathering Facts]: 0.06
  '85_TASK [authorized-keys : Add authorized keys]': 0.0
  86_TASK [Create git sub-directory]: 0.05
  87_TASK [Copy files with rsync]: 0.59
  88_TASK [Set correct permissions]: 0.05
  89_TASK [Install Krake docker dependencies]: 0.12
  90_TASK [Generate the docker configuration file]: 0.04
  91_TASK [Configure the docker infrastructure]: 0.06
  92_TASK [Remove Krake infrastructure]: 0.06
  93_TASK [Remove Krake image]: 0.06
  94_TASK [Create configuration directory]: 0.05
  95_TASK [Create configuration certificates sub-directory]: 0.05
  96_TASK [Generate certificates for all the Krake components]: 0.37
  97_TASK [Generate all signing certificates]: 0.1
  98_TASK [Get a list of all Krake components's configuration files]: 0.05
  99_TASK [Generate the configuration files]: 0.27
  100_TASK [Change permissions of the configuration directory]: 0.05
  101_TASK [Build Krake image]: 0.92
  102_TASK [Start Krake infrastructure]: 0.19
  103_TASK [Bootstrap etcd database]: 0.08
  104_TASK [Create configuration certificates sub-directory]: 0.05
  105_TASK [Copy the rok certificates]: 0.15
  106_TASK [Generate certificates for rok]: 0.05
  107_TASK [Create configuration directory for rok]: 0.05
  108_TASK [Install Krake rok]: 0.08
  109_TASK [Create Krake rok configuration file]: 0.05
  110_TASK [Create rok configuration directory]: 0.01
  111_TASK [Download rok certificates]: 0.15
  112_TASK [Create Krake rok configuration file]: 0.0
  TOTAL_DURATION_[min]: 3.85
11_PLAY [Generate cluster config]:
  113_TASK [Gathering Facts]: 0.06
  114_TASK [Create directory to host clusters infos]: 0.05
  115_TASK [Upload certificates directory]: 1.09
  116_TASK [Generate kube config file]: 0.21
  TOTAL_DURATION_[min]: 1.42
12_PLAY [Running Krake smoke tests]:
  117_TASK [Gathering Facts]: 0.07
  118_TASK [Install Tests dependencies]: 0.1
  119_TASK [Run smoke test for krake client]: 0.07
  TOTAL_DURATION_[min]: 0.24
13_PLAY [Running Minikube smoke tests]:
  120_TASK [Gathering Facts]: 0.07
  121_TASK [Ensure rak directory is present]: 2.99
  122_TASK [Install Tests dependencies]: 0.13
  123_TASK [Run smoke test for Minikube]: 0.07
  TOTAL_DURATION_[min]: 3.26
14_PLAY [Prepare Krake infrastructure for integration tests]:
  124_TASK [Gathering Facts]: 0.07
  '125_TASK [authorized-keys : Add authorized keys]': 0.0
  126_TASK [Configure the docker infrastructure for prometheus]: 0.06
  127_TASK [Remove Krake infrastructure]: 0.09
  128_TASK [Start Krake infrastructure with prometheus]: 0.43
  129_TASK [Setup Minio docker container]: 0.19
  130_TASK [Bootstrap etcd database with the test-specific resources]: 0.14
  TOTAL_DURATION_[min]: 0.98
15_PLAY [Running Krake integration tests]:
  131_TASK [Gathering Facts]: 0.07
  132_TASK [Install Tests dependencies]: 0.08
  133_TASK [Run integration tests for the core resources]: 0.7
  134_TASK [Check if two Minikube clusters have been spawned]: 0.0
  135_TASK [Run integration tests for the scheduling algorithm]: 15.21
  136_TASK [Run integration tests for the migration of kubernetes applications]: 11.22
  137_TASK [Run integration tests for the Krake k8s resources update]: 4.76
  138_TASK [Run integration tests for the Kubernetes Observer]: 3.34
  139_TASK [Run integration tests for the hooks]: 2.04
  140_TASK [Run integration tests for krake scripts]: 0.16
  141_TASK [Run application example tests for krake]: 4.38
  142_TASK [debug]: 0.0
  143_TASK [Save Krake containers logs]: 0.37
  144_TASK [Create a tar.gz archive from the log files]: 0.05
  145_TASK [Download the Krake logs]: 0.06
  TOTAL_DURATION_[min]: 42.43
16_PLAY [Unprovision infrastructure]:
  146_TASK [Delete Heat stack]: 0.42
  147_TASK [Clean host variables]: 0.0
  TOTAL_DURATION_[min]: 0.42
TOTAL_DURATION_[min]: 72.26

Based on the above report consider and investigate the following:

  • #442 (closed) Consider leveraging "Mitogen for Ansible" for performance improvement
  • #515 (closed) Consider replacing dedicated Minikube VMs with lightweight alternatives (e.g. KinD containers)
    • Ansible plays (Launch Minikube VMs, Setup Minikube VMs, Start Minikube clusters, Setup Minikube clusters, Download cluster certificates) take ~8.5min
    • Instead of dedicated VMs for Minikube instances we could consider the KinD containers as alternatives. KinD instances could be spawned in the Krake VM next to Krake containers, but in the separate docker network
  • #514 (closed) Consider upgrading the OS image version for VMs from Ubuntu 20.04 LTS x64 to e.g. Ubuntu 22.04 LTS x64
    • Currently, all VMs use Ubuntu 20.04 LTS x64. Packages there are upgraded and then the VM is rebooted because the kernel has been updated too. Related tasks (update_system : Upgrade all packages and update_system : Reboot the machine if the kernel has been updated) take ~2.8 min for each VM
  • Consider some other well know practices of how to speed up ansible playbooks, e.g. read this or this
  • #519 (closed) Consider splitting up the e2e test into provisioning of the OS infrastructure and the e2e tests itself
    • Move the Ansible plays for setting up the test environment to the 'unittest' stage ( see .gitlab-ci.yml)
Edited by Patrick Thiem