Benchmark the effect of decrypting secrets on Rails instead of Agent
MR: Pending
<!--
The first line of the MR must be one of the following:
1. `MR: Pending`
2. `MR: <MR link with trailing +>`,
and the first description line of the MR should be `Issue: <Issue link with trailing +>`
3. `MR: No MR`
For more context, see:
https://about.gitlab.com/handbook/engineering/development/dev/create/ide/index.html#1-to-1-relationship-of-issues-to-mrs
-->
<!--
The following sections should be filled out as part of the refinement process before the issue is prioritized.
For more context, see:
https://about.gitlab.com/handbook/engineering/development/dev/create/ide/#2-pre-iteration-planning-meeting
-->
## Description
In https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/1011#note_1504356228 , we decided to keep the decryption of secrets to be injected into the workspace at Rails side. We need to benchmark if there are any effects of this on the response time of the API especially for full reconciliation of the agent at scale(10 agents with 500 workspaces each with 100 secrets per workspace).
Currently, there is no lower bound to the full reconciliation interval. What are the negative impacts of that in this scenario.
## Acceptance Criteria
- [ ] Benchmark results of the performance of the API at without decryption at various scales of workspaces.
- [ ] If possible isolate how much of the request time is taken by db access, and by invoking the `devfile` CLI via the `devfile` gem.
- [ ] Benchmark again with decryption at various scales of workspaces X secrets.
- [ ] Based on the benchmark results, take a call if decryption can stay Rails or needs to be moved to Agent. Maybe reimagining(some sort of pagination) full reconciliation might help?
<!-- Replace with other type, e.g. bug or maintenance, if appropriate -->
<!-- Replace with other subtype if appropriate -->
<!-- By default, all issues start in the unprioritized status. See https://about.gitlab.com/handbook/engineering/development/dev/create/ide/#-remote-development-planning-process -->
<!-- For simplicity and to avoid triage bot warnings about missing workflow labels, we will default to issues starting at the refinement phase -->
issue