Add fuzz testing for Remote Development
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
To get a little philosophical because I am bored of writing dry issues
In terms of testing, it would be something like
- Things you know you know - Tests that we normally write. Given an input, the expected output should be this.
- Things you don't know you know - Same as above but these tests catch an unexpected problem down the road which you did not intend to while writing the test.
- Things you know that you don't know - It is difficult to write tests for these or we don't know how to test it. So we skip it.
- Things that you don't know that you don't know - Edge cases that you run into production and only then realize that we should have a test for this.
One such edge we ran into in production is Workspace stuck in limbo if namespace directly ... (#411205 - closed) . To cover the entire spectrum of tests, we should randomize our tests. Not just the unit tests but the integration tests as well as the E2E tests. Our entire reconciliation logic for Workspaces is based on interaction between Rails and Agentk. While we have mapped different state transitions in our docs, we always run into the risk of not knowing what we do not know.
Given that there is a lot to do in terms of rigorous testing for Workspaces and also the fact that our reconciliation logic relies on a sequence of events, it would be really beneficial for us to randomize this sequence i.e. Fuzz Test at all levels.
We have some material about Fuzz Testing at GitLab.
I'd like us to invest time in this to do Fuzz Testing at all levels to improve the confidence in our system.