Implement CLI tests
At the moment we have a few of command-line tools implemented in Kwai:
-
kwai-split- split an input dataset into train/validate/test datasets -
kwai-train- train a model on a dataset -
kwai-test- test a model on a dataset -
kwai-triage- triage test output files -
kwai-dw-triage- triage testing results retrieved from a DataWarehouse
We have tests for some of the higher-level functions these use, but none for the tools themselves. So, we're forced to test them manually.
It would be great to have some (even basic) tests for these. A python implementation using pytest would be best, but Bash would be alright too, given the tests are integrated with the CI pipeline.
Here are some testing ideas for each of these:
-
kwai-split- I can create a small CSV file, accompanied by a ZIP file, constituting a dataset together. The test could verify that the tool produces the correct output files (with byte-for-byte comparison). -
kwai-train- Using the training dataset produced above we can verify that the execution produces expected output on stdout as well as a trained models file. However, there's some non-determinism in the training process and we won't be able to compare the models file exactly. -
kwai-test- We can run the tool on the train/validate/test datasets above and check the output. -
kwai-triage- the tool accepts local files as well, IIRC, so we can take a file from the above dataset and see that the tool works -
kwai-dw-triage- not much can be done here, I suppose, except running the tool against our DW with a fake URL and checking that it throws the correct error. This would still be better than nothing. Unless, that is, DW has a few objects for testing with known URLs which will always stay there. It doesn't, right, @tales-aparecida?
Jira: CKI-6968
Edited by Tales da Aparecida