Rapid Action: Identify root cause of performance bottleneck due to Gitaly Transactions
## Problem statement
The transaction layer in Gitaly allows it to serialize writes and produce a WAL (write-ahead log) which is a building block for other features like WAL-enabled backup/restore as well as RAFT replication.
However, the two times we turned it on in production on our cny node, we had to turn it off due to CPU contention.
We need to do a deep dive into what exactly is the cause of the performance bottlenecks, and propose a list of potential solutions to address these.
DRI: @samihiltunen
## Timeframe
This investigation will have an end date of 2025-04-28, which is two weeks after it was turned on in cny. The result of the investigation is a decision to either roll out transactions more widely on other nodes in the Gitaly fleet, or issues are created to address transaction performance.
## Exit criteria
1. Results of an investigation detailing the cause of any performance bottlenecks on CNY due to Gitaly Transactions is added to this epic.
2. A decision is made to either roll out transactions more widely, or generate a list of proposals on how to address said performance bottlenecks.
## Workstreams
| Workstream | DRI | Participants | Tracking |
|------------|-----|--------------|----------|
| Analyze CNY memory/cpu profile | @msmiley | @samihiltunen, @justintobler | https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/128 |
## Communication
1. Slack: [#p_gitaly_transactions](https://gitlab.enterprise.slack.com/archives/C08MKU3QY2Y)
## Links
<!-- STATUS NOTE START -->
<!-- STATUS NOTE END -->
epic