Set dynamic environment URLs after a job finished

Problem to solve

Our current Review apps implementation, requires a static URL that is used as the CI/CD variable CI_ENVIRONMENT_SLUG. Many use cases the environment variable is not static but dynamic, for example when using AWS , a user will probably want to use the environment name based on the stage stage: qa-lambda-47378475 and it can be replaced. In order to run Review apps in such an environment , we would need to support dynamic URLs.

Proposal

We introduce a new report type of artifact - dotenv.

review:
  script:
    - DYNAMIC_ENVIRONMENT_URL=$(deploy-script)       # In script, you get an environment URL.
    - echo "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" >> deploy.env    # Add the value to a dotenv file.
  artifacts:
    reports:
      dotenv: deploy.env                        # Report back dotenv file to rails.
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: $DYNAMIC_ENVIRONMENT_URL                    # and set the variable produced in script to `environment:url`
    on_stop: stop_review

stop_review:
  script:
    - ./teardown-environment
  when: manual
  environment:
    name: review/$CI_COMMIT_REF_SLUG            # NOTE: Don't set `environment:url` as `dotenv` file is not generated in this job.
    action: stop

This opens an awesome possibility for later passing variables between different builds, by using dotenv syntax. The .env is a simple KEY=VALUE text file with env per line.

Use cases

AWS Lambda and the Serverless framework which give a dynamic URL based on the stage:

Serverless: Stack update finished...
Service Information
service: lambda-api
stage: qa-lambda-47378475
region: us-east-1
stack: lambda-api-qa-lambda-47378475
api keys:
  None
endpoints:
  POST - https://sg01n2q8ph.execute-api.us-east-1.amazonaws.com/qa-lambda-47378475/gql
functions:
  graphqlServer: lambda-api-qa-lambda-47378475-graphqlServer
layers:
  None

We have the same requirement to set CI_ENVIRONMENT_URL dynamically (based on data available in the job), but want to allow it to be overridden from .gitlab-ci.yml when it is manually set there.

Technical Proposal

Runner returns artifacts:report:dotenv: #{dot_env_file_path} as a raw artifact.
In Ci::BuildFinalizeWorker/Service, Rails parses dotenv file and persists it into ci_job_variables table.
In ci_builds.status == finalizing, we processes the build.

TODO

Persist DotEnv Variables From an artifact => !26247 (merged)
Publish Feature (documentation and feature flag removal)

Old Proposal for safe keeping:

If you did support a script-created variable for URLs, I could do something like this :heart::

Current Implementation

create_env:
  stage: review
  when: manual
  script: echo "Create platform environment and create a dynamic variable called FANCY_SLUG"
  environment:
    name: env/$CI_BUILD_REF_SLUG
    url: https://$CI_BUILD_REF_SLUG-$FANCY_SLUG-$APPS_DOMAIN
    on_stop: delete_env

$APPS_DOMAIN is a YAML-defined variable (not job-level)
$FANCY_SLUG is a variable that, ideally, could be set during the script: portion of the job

Limitation

Environment urls in .gitlab-ci.yml definitions only expand CI-* variables

Example

Using a popular hosting platform (https://platform.sh/) I can create environments pretty easily by installing their CLI and using authentication tokens. But the entire integration requires that I have some dynamic variable ability in setting the environment URL. Their URL has a token that is unique per branch, something that requires the script step of the review stage.

Functional Work Around

Let me upfront tell you that I'm probably the worst dev ops to tell you how or what you should do. The below works for me and I see it as a stop-gap measure and provide no warranty whatsoever.

<job>:
  stage: review
  when: manual
  script:
    - export ENVURL='http://replacethiswithanAPIgeneratedURL.com/'
    - export PRIVATE_TOKEN='I recommend you set a project variable called PRIVATE_TOKEN and drop your private token in there.'
    - export GITLAB_URL='https://gitlab.websites-r-us.com'

    # Using the API to bypass the limitation 
    # @see https://docs.gitlab.com/ce/api/enviroments.html#environments
    # I clearly don't know how to script in this environment
    # See this issue for something better hopefully: 
    - export ENVIRONMENT_REQ_HEADER="PRIVATE-TOKEN:"" "$PRIVATE_TOKEN
    - export ENV_ID_REQ_URL=$GITLAB_URL"/api/v3/projects/$CI_PROJECT_ID/environments"
    - export JSON=$(curl --header "$ENVIRONMENT_REQ_HEADER" "$ENV_ID_REQ_URL")
    - export PARSE_JSON='$r = json_decode(fgets(STDIN)); foreach ($r as $e) if ($e->name == "'$CI_ENVIRONMENT_NAME'") { echo $e->id; break;}'
    - export CI_ENVIRONMENT_ID=$(echo $JSON | php -r "$PARSE_JSON")
    - export ENV_SET_URL=$GITLAB_URL"/api/v3/projects/$CI_PROJECT_ID/environments/$CI_ENVIRONMENT_ID"
    - curl --request PUT --data "external_url=$ENVURL" --header "$ENVIRONMENT_REQ_HEADER" "$ENV_SET_URL"
  environment:
    name: $CI_BUILD_REF_SLUG
    on_stop: delete_env
  only:
    - branches
  except:
    - master

Caveat: This will not work if your CI_ENVIRONMENT_NAME is not unique. Though, I can't imagine how that would happen
OMG! You used something that is stupid. ... I would really like the above snippet to either get better or for us to not have to revert to using the API to make a URL dynamic.
I've not tested on a new environment yet, but I expect it to not work.

Edited Mar 05, 2020 by Shinya Maeda