Fix --compatibility-chart functionality

There is a feature called --compatibility-chart that should look in a given folder for the Molecule results of a role and generate a compatibility chart that represents which operating systems a role supports. You can find an example test result here: https://gitlab.com/megabyte-labs/ansible-roles/sdkman/-/blob/master/molecule/.results/logs/2022-01-06-default.txt

The chart that this command generates is in the format of:

[
  ['OS Family', 'OS Version', 'Status', 'Idempotent', 'Tested On'],
  ['Debian', '10 (Buster)', ':heavy_check_mark: ', ':x:', 'August 5th, 2021'],
  ['CentOS', 'Stream 8', ':x:', ':x:', 'August 7th, 2021'],
  ...
  ...
]

With the above data, our build scripts will embed the chart into the README like you see here in the "Supported Operating Systems" section:

https://gitlab.com/megabyte-labs/ansible-roles/androidstudio/-/tree/239813c7f647bb178d27420184d79e61f3271b8f#supported-operating-systems

And you can run the command locally by running the following code:

git clone https://gitlab.com/megabyte-labs/ansible-roles/sdkman.git
cd sdkman
bash .config/scripts/start.sh
poetry run ansibler --generate-compatibility-chart --molecule-results-dir 'molecule/.results/logs' --json-file .cache/compatibility-chart.json

The feature was working at some point but stopped working when timings were added to the log format. The error that is currently returned looks something like this:

Error while parsing molecule test file /home/hawkwood/gas-station/roles/languages/sdkman/molecule/.results/logs/2022-01-06-default.txt: Could not parse molecule test file

The text file that is linked to above might look confusing at first, but the requirements are simpler. The Molecule results have two pieces of information we are looking to extract. The first piece of information is whether the role was able to run without error. And the second piece of information is whether the role is idempotent. An idempotent role would be a role that made changes the first time the role was run but not the second time (the Molecule test includes logic to install the role twice in a row on a given machine).

To fix this --compatibility-chart feature, the code should be adjusted to accomodate the current format of the above-linked text file. You should also make sure the following logic is used:

  1. Scan file for first occurrence of "PLAY [Converge] ************************************" (This is where the role first starts installing the first time around)
  2. Go to next occurrence of "PLAY RECAP ***************************************************"
  3. Extract data from the following section -- using the above text as an example, the section should look something like this:
ArchLinux-Latest           : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
CentOS-7                   : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
CentOS-8                   : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
CentOS-Stream-8            : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Debian-10-Buster           : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Debian-11-Bullseye         : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Debian-9-Stretch           : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Fedora-33                  : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Fedora-34                  : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Fedora-35                  : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Ubuntu-18.04-Bionic-Beaver : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Ubuntu-20.04-Focal-Fossa   : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Ubuntu-21.04-Hirsute-Hippo : ok=12   changed=5    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Ubuntu-21.10-Impish-Indri  : ok=12   changed=4    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
Windows-10                 : ok=1    changed=0    unreachable=0    failed=0    skipped=7    rescued=0    ignored=0

Since all the operating systems have nothing in the failed or unreachable column, all the operating systems would be marked as success (checkmark).

  1. For idempotency column, continue in text until you see "PLAY [Converge] ************************************" again (which should be right below the previous section if you're looking at it visually)
  2. Go to next occurrence of "PLAY RECAP ***************************************************"
  3. Scan through the results and mark anything that has any changed counts as a failure for idempotency. Since there was a change, the role is idempotent and should be marked as such.

With the logic mentioned above, the code should be enough to generate the JSON for a chart. However, it needs to be taken a step further. In some cases, the role might have historical result data and tests for only Windows or only Linux etc. This feature has to scan through all the text files in the directory specified with --molecule-results-dir. The code should create a matrix and start with the text files with the oldest dates first and then loop through all of them until it gets to the newest text file. This way, the newest results will take precedence. The test's date is in file's name.

Most of the code required for this should already be in place. This is just a long issue because the feature it adds is somewhat complex.

Other requirements:

  • If test data only has one [Converge] *** section, then do not calculate the idempotency indicator
Edited by Brian Zalewski