Proper handling of tests vs tasks
Right now, we have some mess with the tests vs tasks we have to run. Part of this mess is discussed in redhat/centos-stream/tests/kernel/kpet-db#97 (closed) and redhat/centos-stream/tests/kernel/kpet-db#98 The other part is, that we need information about setup tasks so we can debug them if they start failing (or even, we need to know they started failing as it renders our testing useless a lot of times!).
The standup discussion lead to the following proposal:
-
Add the variable for identifying tests into kpet-db redhat/centos-stream/tests/kernel/kpet-db#97 (closed) -
Migrate all our tooling that relies on CKI_ID
orCKI_NAME
to mark a test, e.g. https://gitlab.com/cki-project/cki-tools/-/blob/main/cki/kcidb/beaker_to_kcidb.py#L29, to use the variable from step 1 instead -
Have DataWarehouse use the variable from step 1 to identify a test and count it in result computation. Right now, everything submitted to DW is considered a test -
Add DW API and API lib interface to query executed tasks that are not considered tests. Let's keep the current tests interfaces as they are because 1. naming them as "tests" is correct and 2. we can keep all tools using the interfaces as they are -
On DW test view (e.g. https://datawarehouse.cki-project.org/kcidb/builds/313480), have the identified tests. But, create a dropdown table below it for "setup tasks" so the setup task maintainers can view the status and debug any issues / check logs if needed, in an accessible way -
Add CKI_ID
to every task. Ideally in kpet-db so all our tools can use the same IDs, but if kpet-db maintainers are truly against this, upt will need to create its own indexing on first walk of the XML and add it to kcidb, handle it with retries etc. -
Simplify our tooling for test plan creation to save all tasks into the test plan, and upt save results to kcidb for all tasks. Here is where we'd really appreciate the CKI_ID
to be done in kpet-db -- so the test plan creation and upt can use the same ids instead of guessing which is which.