Skip to content

Support DotEnv Variables through report type artifact

Shinya Maeda requested to merge dotenv-report-artifact into master

What does this MR do?

This MR allows users to persist user-defined variables into ci_build_dotenv_variables table by using dotenv report artifact. The persisted variables are used when the system expands an environment URL. In the future, we're going to extend the usage to Inherit environment variables from dependent jobs

Related #17066 (closed)

Process Flow

Job Start, Parse artifact and Persist the result

  1. Runner requests a job. POST api/v4/jobs/:id/request. Rails presents the job as it requires dotenv file as a report-type artifact.
  2. Runner uploads the dotenv file to object storage and notifies to Rails. (POST api/v4/jobs/:id/artifacts)
  3. Rails creates a new record to ci_job_artifacts table via CreateJobArtifactsService.
  4. Rails parses the uploaded artifact and persist the parsed data into a dedicated table.

Persisted Data Consumption

  1. Runner updates the ci_builds.status from running to success at POST api/v4/jobs/:id
  2. Ci::Build.state_machine hooks the event to transitions a deployment.status from running to success.
  3. Deployment.state_machine hooks the event and invokes Deployments::SuccessWorker.
  4. In the Deployments::SuccessWorker, we sets environments.external_url, which is expanded from build.options[:environment][:url] with its variables. For example, if the original URL is https://$DYNAMIC_URL and there is a key/value pair in dotenv file that DYNAMIC_URL=awesome.com, the environments.external_url will be https://awesome.com.

Feature Flag

This feature is implemented behind ci_dotenv_variables feature flag and disabled by default. Once we've confirmed the feature is deemed stable, we remove the feature flag in order to publish the feature as GA.

TODO

  • Local QA
  • app/models/ci/bridge.rb
  • app/models/ci/build.rb
  • app/models/ci/builds/dotenv_variable.rb
  • app/models/ci/job_artifact.rb
  • app/models/commit_status_enums.rb
  • app/models/concerns/ci/contextable.rb
  • app/presenters/commit_status_presenter.rb
  • lib/gitlab/ci/config/entry/reports.rb
  • lib/gitlab/ci/parsers.rb
  • lib/gitlab/ci/status/build/failed.rb
  • lib/gitlab/ci/parsers/dotenv_variable.rb
  • app/services/ci/parse_dotenv_artifact_service.rb
  • lib/api/runner.rb

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda

Merge request reports