Memoize Project & Pipeline predefined_variables

What does this MR do and why?

Describe in detail what your merge request does and why.

This memoizes some method on Project and Pipeline that are repeatedly called when evaluating Pipeline variables, as part of the Gitlab::Ci::Pipeline::Chain::Seed sequence in the Ci::CreatePipelineService.

This sequence is currently one of the slowest parts of creating a Pipeline.

📊 Performance stats on gitlab.com

From: https://dashboards.gitlab.net/d/stage-groups-pipeline_execution/stage-groups-group-dashboard-verify-pipeline-execution?orgId=1&from=1631744700000&to=1631766359999&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=All&var-action=All&var-runner_type=All&viewPanel=1173128221

These method appear to generate the same result each time they are called, so can safely be memoized.

We needed to update some usages in specs to reload the object, since we were re-using instances in between spec runs. In Production though this won't be an issue as the instance is only memoized for a single request.

📊 Performance improvements, locally in GDK when creating a gitlab-org/gitlab(imported) pipeline:

Before

TIMING: Gitlab::Ci::Pipeline::Chain::Build took 0.0620 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Build::Associations took 0.0061 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::Abilities took 0.0205 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::Repository took 0.0003 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::SecurityOrchestrationPolicy took 0.0025 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Skip took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Config::Content took 0.0126 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Config::Process took 4.4289 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::AfterConfig took 0.0114 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::RemoveUnwantedChatJobs took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::SeedBlock took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::EvaluateWorkflowRules took 0.0991 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Seed took 25.3295 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Size took 0.0044 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Deployments took 0.0715 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::External took 0.0003 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Populate took 0.3228 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::StopDryRun took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Create took 6.0971 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Activity took 0.0039 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::JobActivity took 0.0036 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::CancelPendingPipelines took 0.0147 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Metrics took 0.0001 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::TemplateUsage took 0.0418 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Pipeline::Process took 0.0073 seconds.

After

TIMING: Gitlab::Ci::Pipeline::Chain::Build took 0.0239 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Build::Associations took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::Abilities took 0.0186 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::Repository took 0.0004 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::SecurityOrchestrationPolicy took 0.0021 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Skip took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Config::Content took 0.0054 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Config::Process took 3.3466 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::AfterConfig took 0.0077 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::RemoveUnwantedChatJobs took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::SeedBlock took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::EvaluateWorkflowRules took 0.0430 seconds.
**TIMING: Gitlab::Ci::Pipeline::Chain::Seed took 6.4841 seconds.**
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Size took 0.0017 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Deployments took 0.0260 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Validate::External took 0.0003 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Populate took 0.3370 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::StopDryRun took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Create took 6.6892 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::Activity took 0.0026 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Limit::JobActivity took 0.0020 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::CancelPendingPipelines took 0.0114 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Metrics took 0.0000 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::TemplateUsage took 0.0066 seconds.
TIMING: Gitlab::Ci::Pipeline::Chain::Pipeline::Process took 0.0063 seconds.

We see a significant performance improvement. A 75% decrease in the time Gitlab::Ci::Pipeline::Chain::Seed takes in this example.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

1. Import https://gitlab.com/gitlab-org/gitlab into your GDK
2. Add timing log statements into Gitlab::Ci::Pipeline::Chain#observe_step_duration - (Rails.logger.info("TIMING: #{step_class} took #{"%0.4f" % duration} seconds.")
3. Run the pipeline on master.
4. Change to this branch, and compare timings.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.