Fix bug when secrets provider URL given in dotenv file

Jayakrishnan Mallisseryrequested to merge
jay-fix-dotenv-provided-secrets-provider-url into master

Why this MR ?

  • Originally, we had a bug where the JWT tokens for the build had incorrect value for expiry ("exp" key in JSON). It did not consider the job timeout. This was because the JWT tokens were computed and memoized at a stage where the job's timeout information was NOT set. This happened as a side effect of the way we were validating the presence of secrets provider URL. Please go through the description in the bug fix MR

  • This bug was fixed in this bug fix MR. The fix involved moving the validation of the secrets provider URL to the pipeline creation phase. However this fix introduced a regression where the pipeline creation would fail if the secrets provider URL was provided using dotenv files from the output of a different job. This was because at the time of pipeline creation, we DO NOT have access to these dotenv files generated only at job run time.

  • So we need to revert the bug fix MR which caused the regression and solve the Original bug in a different way that DO NOT cause the regression case for dotenv files. This MR does that

What does this MR do ?

  • It has 3 main changes. 2 Changes to revert the MR that caused the regression. 3rd change to fix the original bug.

  • We revert the bug fix MR that caused the regression. This is done by moving back the secrets provider validation check from pipeline creation stage to pre_assign_runner_checks as it existed originally.

    1. disable secret provider validation check on pipeline creation
    2. enable secret provider validation check on pre_assign_runner_checks

  • Change 3 is to solve the original bug of incorrect expiry for JWTs, we DO NOT compute and memoize the JWT tokens when we DO NOT have timeout information. We do this by checking if the secrets provider URL is present ONLY in the subset of variables where it can be present. Only these subset of variables are computed and memoized at the time of Secrets Provider validation. As a result, the JWT Token variables are only computed and memoized later when its needed which is after we have set the timeout information correctly.

  • All 3 changes are behind a derisk feature flag

  • Feature Flag Rollout issue

  • Feature Flag Cleanup issue

References

Issue

Screenshots or screen recordings

VAULT URL in Dotenv file Before this MR VAULT URL in Dotenv file After this MR
before after
Incorrect 5 minute expiry when all variables passed to Secrets::Integration class Correct 45 minute expiry when only scoped and job variables passed to Secrets::Integration class
all_variables_memoized expiry_45_mins

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jayakrishnan Mallissery

Merge request reports