Concept of `metadata` in Gitlab CI/CD
Problem to solve
Please find the following issues which I struggle with:
- The
environment
config is loose concept which has no effect on thegitlab-ci.yml
pipelines. From pipeline perspective it is just meta information, an extra information which enables other Gitlab features like theEnvironments
view. The documentation is also describing theenvironment
presence in the pipeline definition as meta.
Note: The environment keyword is just a hint for GitLab that this job actually deploys to the name environment. It can also have a url that is exposed in various places within GitLab. Each time a job that has an environment specified succeeds, a deployment is recorded, storing the Git SHA and environment name.
-
Environments
view lacks the possibility to display the deployed version on each environment. Many will quickly say that this issue is easy to solve by adding an extra property toenvironment
(i.e.environment.version
). But that will be bad for complex pipeline stages which deploy in blue-green or canary fashion. -
The current version of the pipeline webhook event doesn't propagate the pipeline
variables
at all. Thevariables
field presence in the payload is misleading for anyone reading the documentation.
https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#pipeline-events
The environment variables usage can be quite complex (group-level, project-level, pipeline, scheduled, etc.) and adding them to the webhook event payload may lead to security issues, especially when the webhook endpoint is public and not secured (https). Having in mind the above it may be a good idea to just deprecated and remove them from any webhook event payload in the future versions of Gitlab.
How is that related to themetadata
?
In most cases engineers want to have some meta information in the webhook payload such asversion(s)
, etc. There is no way to achieve that with the current webhook event api. Introducing themetadata
array in thepipeline
andjobs
webhook event payload will be safe and natural fit. It will be clearly separated from thevariables
and there won't be any security concerns of leaking sensitive information. -
Enrich the
pipelines
andjobs
webhook events with additional information viametadata
array. By doing that you will reduce notably the amount of extra calls from Gitlab to the 3rd party webhook event services. Current events are limiting and favour therequest-response
model. The event content cannot be customized/enriched with extra information which leads to more complex communication flows with Gitlab. i.e.:
current: withoutmetadata
- request-response (pull based):graph TD; A[Gitlab] -- pipeline webhook call -->B[WH Service]; B -- consume and fetch to get extra meta info -->A;
ideal: with
metadata
- event driven (push based):graph TD; C[Gitlab] -- pipeline webhook call -->D[WH Service];
Intended users
- Delaney (Development Team Lead)
- Sasha (Software Developer)
- Devon (DevOps Engineer)
- Sidney (Systems Administrator)
- Sam (Security Analyst)
- Dana (Data Analyst)
Further details
Proposal
1. Minimalistic:
This approach will not break any existing code/configuration.
.gitlab-ci.yml
:
deploy_prod:
stage: deploy
script:
- echo "Deploy to production server"
metadata:
version: $VERSION
region: $REGION
strategy: $DEPLOYMENT_STRATEGY
Webhook Pipeline event:
{
"object_kind": "pipeline",
"object_attributes": {
...
"metadata": [
{
"key": "maintainer",
"value": "team-a"
}
]
},
"builds":[
{
...
"metadata": [
{
"key": "version",
"value": "0.5.0.RELEASE"
},
{
"key": "region",
"value": "some-region"
}
]
}
]
}
Webhook Job Event:
{
"object_kind": "build",
...
"metadata": [
{
"key": "version",
"value": "0.5.0.RELEASE"
},
{
"key": "region",
"value": "some-region"
}
]
}
2. Nice to have:
This approach will have breaking changes in the .gitlab-ci.yml
and Environments and Deployment
view.
The idea here is to consider environment
part a metadata
information.
i.e
current:
deploy_prod:
stage: deploy
script:
- echo "Deploy to production server"
environment:
name: production
url: some-nice-prod.example.com
metadata:
version: $VERSION
region: $REGION
strategy: $DEPLOYMENT_STRATEGY
after the change:
deploy_prod:
stage: deploy
script:
- echo "Deploy to production server"
metadata:
environment_name: production
environment_url: some-nice-prod.example.com
version: $VERSION
region: $REGION
strategy: $DEPLOYMENT_STRATEGY
Webhook Pipeline & Jobs events payload is the same as in the Minimalistic point.
Permissions and Security
Documentation
Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
Environments and Deployments
Environment Variables
WebHooks: Pipeline Events
WebHooks: Job Events