Backend performance tracing using Sentry
Description
Sentry has introduced a "performance tracing" feature. This allows for tracing the time cost in a granular matter, with more detail than our InfluxDB based metrics currently allow for.
First of all, we can't do this yet:
-
We're still using Sentry-Symfony 3.x. Need to upgrade to 4.1 at least -
Documentation is relatively sparse for the Symfony SDK, all the examples and text is about PHP in general, even if you're looking at the Symfony guide. This is because the Symfony guide "inherits" everything from PHP, and can override things. But it wasn't maintained as well as the base PHP SDK.
Once those roadblocks are out of the way, and the documentation is clearer, we will have to decide on sampling rates. As tracing is not zero-cost (tracking how long every step in a request takes creates some overhead), we will be able to set a percentage of requests that will be served with tracing enabled. I suggest we pull the sampling rate from the environment configuration.
- Beta should get a higher tracing rate, because the user base is much smaller -> we need more data to get meaningful statistics
- Production should have a conservative sampling rate. Let's start with a small value, and work our way up with a maximum of 20% (arbitrary, but I think it's a good bar to set for now). We can delegate this to the admin team, they have a much shorter feedback loop for adjusting it (direct access to both sentry and configuration)
Links / references
- Tracing is available since sentry-symfony 4.1: https://github.com/getsentry/sentry-symfony/issues/442
- Documentation:
- Plain PHP: https://docs.sentry.io/platforms/php/performance/
- Symfony: https://docs.sentry.io/platforms/php/guides/symfony/performance/
- (they're basically the same, and all configuration examples are PHP code, where sentry-symfony is configured through YAML)
Edited by Alex