Investigate Package code area logs to determine customer usage to generate test fixtures
Motivation
Recent production incidents have in some part stemmed from a lack of awareness of how our customers might be using our features. In at least one case, this led to an outage that was discovered by the team and easily rectified but not in time to avoid impacting our customers.
Conjecture
By doing a distribution analysis of our logs, we could determine what our most used end points are and common requests to allow us to more accurately and easily validate common usage.
By programatically analysing the usage and its distribution, we could generate fixtures (statically or dynamically) that would form the basis of some automated test cases.
We can't necessarily predict all possible patterns for our APIs, but by using logs in this way, we could quickly determine how it was being used and adjust our testing quickly and automatically.
This analysis would be in addition to existing test harnesses.
Questions to answer
-
Are we logging requests to our endpoints? -
Is there enough detail to use these logs to form the basis of tests? -
Are we able to programmatically interact with the logs? -
Can we determine distribution of requests? -
Can we programatically generate fixtures from this data? -
Is there a preferred, alternate approach to automate testing of our code areas?