Large Performance Issues Proposal

Why is this change being made?

We are now getting performance issues from the Quality team based on the GPT results.

These GPT results point out the average, P90 and maximum response times of some endpoints that we are responsible for. The results of these tests are included in the performance issues. The P90 helps in determining the severity of the issue.

As we make changes to resolve the performance issues totally or to reduce its severity, the performance issues gets harder to fix and takes time to investigate and devise a solution based on my experience. There can be multiple problems that is causing high response times and possibly need multiple solutions to resolve.

Proposal

We are investigating performance issues as part of our planning process. But since investigating them can take time and we are aiming to make the investigation a little bit more lightweight, I feel that we should tinker our process for investigating and resolving performance issues.

The proposed process is the content of this MR.

Risks

Slower velocity on fixing performance issues

Since we will work on a performance issue as a Deliverable but we're not really fixing it yet, the actual fix will likely be done in the next milestone.

It is possible that the fixed weight given can actually be greater than actual effort. When that happens, engineers will have more spare time left in a milestone to start working on it and potentially resolve it in the same release.

Less team capacity

The fixed weight given is a bit high and will eat up half of an individual engineer's capacity. If we have a lot of performance issues to work on a single milestone, multiple engineers might get dedicated on investigating them.

But since we shouldn't get that many performance issues as we fix them, this shouldn't be a problem as long as we're not introducing more of them. We also seem to balance performance issues and other stuff well when scheduling.

Benefits

Reduce chances of slippage

We will be working on smaller issues as much as possible. It'll be more focused. Resolving a smaller issue doesn't require us to reduce the response times to meet a certain target for it to be closed.

Decrease pressure on engineer

Based on my experience, while working on a performance issue, I felt a self-inflicted pressure of being able to investigate it thoroughly and fix it in the same milestone.

With this proposal, that should be alleviated since more ample time is given for investigation and working on actual solution.

Work can be split into multiple engineers

It is possible that a performance issue will be split into smaller issues. If we need to take care of the issues as soon as possible to reduce impact of performance issue, multiple engineers can be assigned to work on those issues separately and in parallel.

Author Checklist

• Provided a concise title for the MR
• Added a description to this MR explaining the reasons for the proposed change, per say-why-not-just-what
• Assign this change to the correct DRI
  • If the DRI for the page/s being updated isn’t immediately clear, then assign it to your manager.
  • If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies.
  • If the changes relate to any part of the project other than updates to content and/or data files please make sure to ping @gl-static-site-editor in a comment for a review and merge. For example changes to .gitlab-ci.yml, JavaScript/CSS/Ruby code or the layout files.