Define a process and document adjusting request duration thresholds
After we've made it possible to set thresholds in #1223 (closed). Most endpoints endpoints would have the default threshold set to them. Some endpoints might warrant changing that.
In #1237 (closed) we will describe how to set a custom threshold in the code. For this issue, we need to describe the process that accompanies making this change.
For example:
- how to determine what is acceptable for the new value
- who can make the decision about what is a reasonable value
Things to take into account:
For decreasing a duration threshold we need to take the SLO set on the service that handles the endpoint is still met. I suspect the easiest way to do that would be to use the logs to check if the p<SLO>
is below the threshold being proposed. The target is specified as the apdexScore
in the service catalog.
For increasing a duration threshold there should be a solid reasoning as to why a duration should be higher than 1s. Depending on how busy the endpoint is, we might want to prevent that. I don't know how busy an endpoint would be to busy and should require looking into increasing performace rather than the threshold.