Set dynamic environment URLs after a job finished
Problem to solve
Our current Review apps implementation, requires a static URL that is used as the CI/CD variable CI_ENVIRONMENT_SLUG
. Many use cases the environment variable is not static but dynamic, for example when using AWS , a user will probably want to use the environment name based on the stage stage: qa-lambda-47378475
and it can be replaced. In order to run Review apps in such an environment , we would need to support dynamic URLs.
Proposal
We introduce a new report type of artifact - dotenv
.
review:
script:
- DYNAMIC_ENVIRONMENT_URL=$(deploy-script) # In script, you get an environment URL.
- echo "DYNAMIC_ENVIRONMENT_URL=$DYNAMIC_ENVIRONMENT_URL" >> deploy.env # Add the value to a dotenv file.
artifacts:
reports:
dotenv: deploy.env # Report back dotenv file to rails.
environment:
name: review/$CI_COMMIT_REF_SLUG
url: $DYNAMIC_ENVIRONMENT_URL # and set the variable produced in script to `environment:url`
on_stop: stop_review
stop_review:
script:
- ./teardown-environment
when: manual
environment:
name: review/$CI_COMMIT_REF_SLUG # NOTE: Don't set `environment:url` as `dotenv` file is not generated in this job.
action: stop
This opens an awesome possibility for later passing variables between different builds, by using dotenv
syntax. The .env
is a simple KEY=VALUE
text file with env per line.
Use cases
AWS Lambda and the Serverless framework which give a dynamic URL based on the stage:
Serverless: Stack update finished...
Service Information
service: lambda-api
stage: qa-lambda-47378475
region: us-east-1
stack: lambda-api-qa-lambda-47378475
api keys:
None
endpoints:
POST - https://sg01n2q8ph.execute-api.us-east-1.amazonaws.com/qa-lambda-47378475/gql
functions:
graphqlServer: lambda-api-qa-lambda-47378475-graphqlServer
layers:
None
We have the same requirement to set CI_ENVIRONMENT_URL
dynamically (based on data available in the job), but want to allow it to be overridden from .gitlab-ci.yml
when it is manually set there.
Technical Proposal
- Runner returns
artifacts:report:dotenv: #{dot_env_file_path}
as a raw artifact. - In
Ci::BuildFinalizeWorker/Service
, Rails parsesdotenv
file and persists it intoci_job_variables
table. - In
ci_builds.status == finalizing
, we processes the build.
TODO
-
Persist DotEnv Variables From an artifact => !26247 (merged) -
Publish Feature (documentation and feature flag removal)
Old Proposal for safe keeping:
Old Proposal for safe keeping:
If you did support a script-created variable for URLs, I could do something like this :heart::
Current Implementation
create_env:
stage: review
when: manual
script: echo "Create platform environment and create a dynamic variable called FANCY_SLUG"
environment:
name: env/$CI_BUILD_REF_SLUG
url: https://$CI_BUILD_REF_SLUG-$FANCY_SLUG-$APPS_DOMAIN
on_stop: delete_env
-
$APPS_DOMAIN
is a YAML-defined variable (not job-level) -
$FANCY_SLUG
is a variable that, ideally, could be set during thescript:
portion of the job
Limitation
- Environment urls in
.gitlab-ci.yml
definitions only expand CI-* variables
Using a popular hosting platform (https://platform.sh/) I can create environments pretty easily by installing their CLI and using authentication tokens. But the entire integration requires that I have some dynamic variable ability in setting the environment URL. Their URL has a token that is unique per branch, something that requires the script step of the review stage.
Functional Work Around
Let me upfront tell you that I'm probably the worst dev ops to tell you how or what you should do. The below works for me and I see it as a stop-gap measure and provide no warranty whatsoever.
<job>:
stage: review
when: manual
script:
- export ENVURL='http://replacethiswithanAPIgeneratedURL.com/'
- export PRIVATE_TOKEN='I recommend you set a project variable called PRIVATE_TOKEN and drop your private token in there.'
- export GITLAB_URL='https://gitlab.websites-r-us.com'
# Using the API to bypass the limitation
# @see https://docs.gitlab.com/ce/api/enviroments.html#environments
# I clearly don't know how to script in this environment
# See this issue for something better hopefully:
- export ENVIRONMENT_REQ_HEADER="PRIVATE-TOKEN:"" "$PRIVATE_TOKEN
- export ENV_ID_REQ_URL=$GITLAB_URL"/api/v3/projects/$CI_PROJECT_ID/environments"
- export JSON=$(curl --header "$ENVIRONMENT_REQ_HEADER" "$ENV_ID_REQ_URL")
- export PARSE_JSON='$r = json_decode(fgets(STDIN)); foreach ($r as $e) if ($e->name == "'$CI_ENVIRONMENT_NAME'") { echo $e->id; break;}'
- export CI_ENVIRONMENT_ID=$(echo $JSON | php -r "$PARSE_JSON")
- export ENV_SET_URL=$GITLAB_URL"/api/v3/projects/$CI_PROJECT_ID/environments/$CI_ENVIRONMENT_ID"
- curl --request PUT --data "external_url=$ENVURL" --header "$ENVIRONMENT_REQ_HEADER" "$ENV_SET_URL"
environment:
name: $CI_BUILD_REF_SLUG
on_stop: delete_env
only:
- branches
except:
- master
- Caveat: This will not work if your CI_ENVIRONMENT_NAME is not unique. Though, I can't imagine how that would happen
- OMG! You used
something
that is stupid. ... I would really like the above snippet to either get better or for us to not have to revert to using the API to make a URL dynamic. - I've not tested on a new environment yet, but I expect it to not work.