Consider alternative Gitaly configurations for Review Apps
As part of gitlab-com/www-gitlab-com#11568 (closed) I planned to investigate if Gitaly-related E2E test failures could be avoided by using a hybrid architecture (with Gitaly on a VM and all else as the standard review app in k8s).
My first step was to set up a reference hybrid architecture, just to be sure that the problems didn't exist. I was not able to reproduce the test failures.
The next step is to investigate alternative Gitaly configurations that don't require changing the entire review app architecture.
Proposal options
These options are listed in order of increasing performance but also increasing complexity and cost.
- Use the existing Gitaly chart but with higher performance storage to try to satisfy its IOPs needs.
- Configure the cluster to attach local SSDs to the nodes
- Use Gitaly as an external service.
Option 1 should be the smallest possible change, so we could try that first.
Option 3 is most flexible but would require more work to implement (but it's an option that some customers have had to take)
Tasks
-
Test the alternatives to verify that QA tests pass consistently: !64747 (merged) -
Make the change permanent in engineering-productivity-infrastructure | https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/merge_requests/9 -
Resolve the issue of Gitaly being intermittently unresponsive: #335354 (closed)