Verify and rollout `optimized_issuable_label_filter` feature flag
What
This is a placeholder issue to test and verify the performance of the label based search queries. The optimization is currently behind a global feature flag (optimized_issuable_label_filter) which was introduced in !34503 (merged) (merged)
Owners
- Team: group::optimize
- Most appropriate slack channel to reach out to:
#g_manage_optimize - Best individual to reach out to: ahegyi
Expectations
What are we expecting to happen?
With the optimized_issuable_label_filter feature flag, we expect that the response time improves when searching merge requests or issues by label(s).
The feature flag was enabled several times for shorter periods (SREs were involved in the first trials). It's enabled on GL.com since 5th of January 2021.
Perf measurements:
| url | avg(before → after) | min(before → after) | max(before → after) |
|---|---|---|---|
| 1. | 406ms → 313ms(~30%↓) | 256ms → 244ms(~5%↓) | 616ms → 470ms(~31%↓) |
| 2. | 179ms → 185ms(~3%↑) | 157ms → 158ms(even) | 209ms → 268ms(~22%↑) |
| 3. | 885ms → 690ms(~28%↓) | 769ms → 653ms(~18%↓) | 992ms → 776ms(~28%↓) |
| 4. | 4372ms → 2629ms(~66%↓) | 3056ms → 1901ms(~61%↓) | 6829ms → 6445ms(~6%↓) |
| 5. | 3210ms → 2024ms(~58%↓) | 2944ms → 1792ms(~64%↓) | 3533ms → 2484ms(~42%↓) |
| 6. | 1881ms → 358ms(~425%↓) | 1722ms → 319ms(~439%↓) | 2026ms → (~410%↓) |
| 7. | 422ms → 419ms(even) | 374ms → 379ms(even) | 488ms → 459ms(~6%↓) |
| 8. | 3127ms → 483ms(~547%↓) | 2828ms → 435ms(~524%↓) | 3399ms → (~494%↓) |
| 9. | 3099ms → 374ms(~728%↓) | 2988ms → 330ms(~805%↓) | 3368ms → 568ms(~492%↓) |
What might happen if this goes wrong?
- Performance problem: search with
label_nameparameter might be slower than expected. Based on the PRD data, this is unlikely. - Search inconsistency: search with
label_namemight not return all the relevant items.
What can we monitor to detect problems with this?
Roll Out Steps
-
Enable on staging ( /chatops run feature set feature_name true --staging) -
Test on staging -
Ensure that documentation has been updated -
Enable on GitLab.com for individual groups/projects listed above and verify behaviour ( /chatops run feature set --project=gitlab-org/gitlab feature_name true) -
Coordinate a time to enable the flag with the SRE oncall and release managers - In
#productionmention@sre-oncalland@release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
- In
-
Announce on the issue an estimated time this will be enabled on GitLab.com -
Enable on GitLab.com by running chatops command in #production(/chatops run feature set feature_name true) -
Cross post chatops Slack command to #support_gitlab-com(more guidance when this is necessary in the dev docs) and in your team channel -
Announce on the issue that the flag has been enabled -
Remove feature flag and add changelog entry -
After the flag removal is deployed, clean up the feature flag by running chatops command in #productionchannel
Rollback Steps
-
This feature can be disabled by running the following Chatops command:
/chatops run feature set optimized_issuable_label_filter falseEdited by Adam Hegyi