Backend: Add better logging to config_compose and config_mapper_process
Summary
As a part of investigating our error budget, we uncovered a high amount of slow requests for GraphqlController#execute: getCiConfigData and getPipeline
.
As a result, it has created an unhealthy error budget in the last 7 days:
Proposal
In order for the error budget to return to a healthier state, before we can fix the slow requests, we need to add better logging in config_compose
and config_mapper_process
to better understand what is contributing to the slower behavior which will then help with knowing how to improve performance.
Based on the noted provided here:
-
yaml_process_duration_s
=config_compose_duration_s
+config_expand_duration_s
+ other processes -
config_expand_duration_s
=config_external_process_duration_s
+ other processes
- Most of the time is spent on
config_compose
andconfig_expand
. - We don't know much about internal durations of
config_compose
. - Most of the time in
config_expand
is spent inconfig_external_process
. - Most of the time in
config_external_process
is spent inconfig_mapper_process
.
Prioritization Table
Title | Issue Link |
---|---|
Backend: Add better logging to config_compose and config_mapper_process |
|
Backend: Puma Apdex: slow requests for GraphqlController#execute: getCiConfigData and getPipeline | #349968 (closed) |
Links/References
Edited by Mark Nuzzo