Automated GUI Testing
Problem Outline
When releasing new versions of pmOS or different UIs, it would be nice if the gui could be automatically tested. Notably, many users are worried about daily driving pmOS unless they can rely on telephony support and showing unit tests for most core features would be reassuring for enough users to make the jump. Furthermore, it would speed up the inclusion of new well-tested software into pmOS. As pmOS supports more devices and UIs, automated testing becomes more essential.
OpenQA
I found a tool to automate GUI testing called openqa http://open.qa/ :
Pros
- Has support for testing bare metal and qemu images
- Tests the images how a user would test it. Ie. Does not touch the software directly, controls keyboard and mouse, uses opencv to read the screen out to check if the system is passing tests, and can also reads plain text from serial output
- The tests are relatively easy to write: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/x11/chromium.pm
Cons
- Has components written in perl
- I could not install it on my laptop running alpine linux with podman (it failed with a cannot get a git repo. I cannot find what they mean from the docs). It also seems that Arch tried porting it to their distro and didnt get around to it: https://libreddit.silkky.cloud/r/archlinux/comments/7nak0v/have_arch_devs_considered_using_suses_openqa/ Im not sure why more distros have not adopted openqa. Is it just hard to run and use and requires the corporate backing of redhat or opensuse (fedora and opensuse are the only two distros using it)?
I cannot get it to install but my laptop also doesnt have kvm support. Bhushan Shah told me not install the entire openqa suite but only the command line utility that manages the running of tests (called OS-autoinst https://github.com/os-autoinst/os-autoinst). They shared the scripts they use to setup openqa here: https://invent.kde.org/sysadmin/neon-openqa https://invent.kde.org/sysadmin/neon-openqa/-/blob/master/bin/install.rb
In the matrix chat, I was told @MartijnBraam and @calebccff were involved with this. So Im just posting my findings here so far so they can build off it. I think openqa is a good solution and I really want to integrate sxmo with it but I dont have the hardware to use it.