Commit 06952aaf authored by Grzegorz Bizon's avatar Grzegorz Bizon 🔴

Merge branch 'feature/ci-job-env-variables' into 'master'

Add environment variables on a job level in CI

Make it possible to define environment variables on a job level.

Closes #14716

See merge request !3612
parents 2b8fc138 1339fda1
Pipeline #1946513 passed with stage
......@@ -3,6 +3,7 @@ Please view this file on the master branch, on stable branches it's out of date.
v 8.7.0 (unreleased)
- Method instrumentation now uses Module#prepend instead of aliasing methods
- Repository.clean_old_archives is now instrumented
- Add support for environment variables on a job level in CI configuration file
- The Projects::HousekeepingService class has extra instrumentation
- All service classes (those residing in app/services) are now instrumented
- Developers can now add custom tags to transactions
......
......@@ -365,11 +365,23 @@ module Ci
self.update(erased_by: user, erased_at: Time.now)
end
private
def yaml_variables
global_yaml_variables + job_yaml_variables
end
def global_yaml_variables
if commit.config_processor
commit.config_processor.global_variables.map do |key, value|
{ key: key, value: value, public: true }
end
else
[]
end
end
def job_yaml_variables
if commit.config_processor
commit.config_processor.variables.map do |key, value|
commit.config_processor.job_variables(name).map do |key, value|
{ key: key, value: value, public: true }
end
else
......
## Variables
When receiving a build from GitLab CI, the runner prepares the build environment.
It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables**
The variables can be overwritten. They take precedence over each other in this order:
1. Trigger variables
1. Secure variables
1. YAML-defined variables
1. YAML-defined job-level variables
1. YAML-defined global variables
1. Predefined variables
For example, if you define:
1. API_TOKEN=SECURE as Secure Variable
1. API_TOKEN=YAML as YAML-defined variable
1. `API_TOKEN=SECURE` as Secure Variable
1. `API_TOKEN=YAML` as YAML-defined variable
The API_TOKEN will take the Secure Variable value: `SECURE`.
The `API_TOKEN` will take the Secure Variable value: `SECURE`.
### Predefined variables (Environment Variables)
......@@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts.
The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
Variables can be defined at a global level, but also at a job level.
More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md).
### User-defined variables (Secure Variables)
**This feature requires GitLab Runner 0.4.0 or higher**
GitLab CI allows you to define per-project **Secure Variables** that are set in build environment.
GitLab CI allows you to define per-project **Secure Variables** that are set in
the build environment.
The secure variables are stored out of the repository (the `.gitlab-ci.yml`).
The variables are securely passed to GitLab Runner and are available in build environment.
It's desired method to use them for storing passwords, secret keys or whatever you want.
The variables are securely passed to GitLab Runner and are available in the
build environment.
It's desired method to use them for storing passwords, secret keys or whatever
you want.
**The value of the variable can be shown in build log if explicitly asked to do so.**
If your project is public or internal you can make the builds private.
......
......@@ -23,6 +23,7 @@ If you want a quick introduction to GitLab CI, follow our
- [Jobs](#jobs)
- [script](#script)
- [stage](#stage)
- [job variables](#job-variables)
- [only and except](#only-and-except)
- [tags](#tags)
- [when](#when)
......@@ -174,6 +175,8 @@ These variables can be later used in all executed commands and scripts.
The YAML-defined variables are also set to all created service containers,
thus allowing to fine tune them.
Variables can be also defined on [job level](#job-variables).
### cache
>**Note:**
......@@ -324,6 +327,7 @@ job_name:
| services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) |
| stage | no | Defines a build stage (default: `test`) |
| type | no | Alias for `stage` |
| variables | no | Define build variables on a job level |
| only | no | Defines a list of git refs for which build is created |
| except | no | Defines a list of git refs for which build is not created |
| tags | no | Defines a list of tags which are used to select Runner |
......@@ -414,6 +418,18 @@ job:
The above example will run `job` for all branches on `gitlab-org/gitlab-ce`,
except master.
### job variables
It is possible to define build variables using a `variables` keyword on a job
level. It works basically the same way as its global-level equivalent but
allows you to define job-specific build variables.
When the `variables` keyword is used on a job level, it overrides global YAML
build variables and predefined variables.
Build variables priority is defined in
[variables documentation](../variables/README.md).
### tags
`tags` is used to select specific Runners from the list of all Runners that are
......
......@@ -7,9 +7,9 @@ module Ci
ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services,
:allow_failure, :type, :stage, :when, :artifacts, :cache,
:dependencies]
:dependencies, :variables]
attr_reader :before_script, :image, :services, :variables, :path, :cache
attr_reader :before_script, :image, :services, :path, :cache
def initialize(config, path = nil)
@config = YAML.safe_load(config, [Symbol], [], true)
......@@ -40,6 +40,17 @@ module Ci
@stages || DEFAULT_STAGES
end
def global_variables
@variables
end
def job_variables(name)
job = @jobs[name.to_sym]
return [] unless job
job.fetch(:variables, [])
end
private
def initial_parsing
......@@ -115,7 +126,7 @@ module Ci
end
unless @variables.nil? || validate_variables(@variables)
raise ValidationError, "variables should be a map of key-valued strings"
raise ValidationError, "variables should be a map of key-value strings"
end
if @cache
......@@ -145,6 +156,7 @@ module Ci
validate_job_types!(name, job)
validate_job_stage!(name, job) if job[:stage]
validate_job_variables!(name, job) if job[:variables]
validate_job_cache!(name, job) if job[:cache]
validate_job_artifacts!(name, job) if job[:artifacts]
validate_job_dependencies!(name, job) if job[:dependencies]
......@@ -206,6 +218,13 @@ module Ci
end
end
def validate_job_variables!(name, job)
unless validate_variables(job[:variables])
raise ValidationError,
"#{name} job: variables should be a map of key-value strings"
end
end
def validate_job_cache!(name, job)
if job[:cache][:key] && !validate_string(job[:cache][:key])
raise ValidationError, "#{name} job: cache:key parameter should be a string"
......
......@@ -345,20 +345,76 @@ module Ci
end
end
describe "Variables" do
it "returns variables when defined" do
variables = {
var1: "value1",
var2: "value2",
}
config = YAML.dump({
variables: variables,
before_script: ["pwd"],
rspec: { script: "rspec" }
})
describe 'Variables' do
context 'when global variables are defined' do
it 'returns global variables' do
variables = {
VAR1: 'value1',
VAR2: 'value2',
}
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.variables).to eq(variables)
config = YAML.dump({
variables: variables,
before_script: ['pwd'],
rspec: { script: 'rspec' }
})
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.global_variables).to eq(variables)
end
end
context 'when job variables are defined' do
context 'when syntax is correct' do
it 'returns job variables' do
variables = {
KEY1: 'value1',
SOME_KEY_2: 'value2'
}
config = YAML.dump(
{ before_script: ['pwd'],
rspec: {
variables: variables,
script: 'rspec' }
})
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.job_variables(:rspec)).to eq variables
end
end
context 'when syntax is incorrect' do
it 'raises error' do
variables = [:KEY1, 'value1', :KEY2, 'value2']
config = YAML.dump(
{ before_script: ['pwd'],
rspec: {
variables: variables,
script: 'rspec' }
})
expect { GitlabCiYamlProcessor.new(config, path) }
.to raise_error(GitlabCiYamlProcessor::ValidationError,
/job: variables should be a map/)
end
end
end
context 'when job variables are not defined' do
it 'returns empty array' do
config = YAML.dump({
before_script: ['pwd'],
rspec: { script: 'rspec' }
})
config_processor = GitlabCiYamlProcessor.new(config, path)
expect(config_processor.job_variables(:rspec)).to eq []
end
end
end
......@@ -730,14 +786,14 @@ EOT
config = YAML.dump({ variables: "test", rspec: { script: "test" } })
expect do
GitlabCiYamlProcessor.new(config, path)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings")
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
end
it "returns errors if variables is not a map of key-valued strings" do
it "returns errors if variables is not a map of key-value strings" do
config = YAML.dump({ variables: { test: false }, rspec: { script: "test" } })
expect do
GitlabCiYamlProcessor.new(config, path)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-valued strings")
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "variables should be a map of key-value strings")
end
it "returns errors if job when is not on_success, on_failure or always" do
......
......@@ -238,6 +238,22 @@ describe Ci::Build, models: true do
it { is_expected.to eq(predefined_variables + predefined_trigger_variable + yaml_variables + secure_variables + trigger_variables) }
end
context 'when job variables are defined' do
##
# Job-level variables are defined in gitlab_ci.yml fixture
#
context 'when job variables are unique' do
let(:build) { create(:ci_build, name: 'staging') }
it 'includes job variables' do
expect(subject).to include(
{ key: :KEY1, value: 'value1', public: true },
{ key: :KEY2, value: 'value2', public: true }
)
end
end
end
end
end
end
......
......@@ -4,7 +4,7 @@ services:
before_script:
- gem install bundler
- bundle install
- bundle install
- bundle exec rake db:create
variables:
......@@ -17,7 +17,7 @@ types:
rspec:
script: "rake spec"
tags:
tags:
- ruby
- postgres
only:
......@@ -26,27 +26,32 @@ rspec:
spinach:
script: "rake spinach"
allow_failure: true
tags:
tags:
- ruby
- mysql
except:
- tags
staging:
variables:
KEY1: value1
KEY2: value2
script: "cap deploy stating"
type: deploy
tags:
tags:
- ruby
- mysql
except:
- stable
production:
variables:
DB_NAME: mysql
type: deploy
script:
script:
- cap deploy production
- cap notify
tags:
tags:
- ruby
- mysql
only:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment