Local runner execution MVC
In 10.0 (#2710 (closed)) we deprecated (but did not remove) the documented
exec functionality of gitlab-runner, since it was incomplete and hard to maintain. It was later un-deprecated (and the limitations section was added at the document link), but at the moment there is no more complete replacement for it and the fundamental issues with it were not resolved.
The exec command allows you to run jobs locally, trying to replicate the CI environment as much as possible. It doesn't need to connect to GitLab, instead it reads the local .gitlab-ci.yml and creates a new build environment in which all the build steps in a single job are executed. For example, the following command will execute the job named tests locally using a shell executor:
gitlab-runner exec shell tests
- For pipeline or job editors,
execcan be used as a fast way of validating that a pipeline or job performs in the expected way, without having to generate a lot of commits/wait for remote builds to trigger and run. This is primarily a use case for when iterating on changes to the way a job works.
- For developers/testers, it can also be used to execute part of the pipeline or job on a local copy of your code, for example: a local test run on local files, validating that tests pass (in the way they will run in the runner) prior to committing.
- Related to above, for people who use an offline workflow with Git (for example, while traveling), this gives them the ability to run pipelines or jobs locally.
- This offers the only safe way to run a pipeline or job and print all environment variables (for testing purposes) without leaking them in a stored log somewhere
- Stored manual tasks which are made available to local
execusers, i.e. in https://gitlab.com/fdroid/fdroid-website/blob/master/.gitlab-ci.yml:
# This is a manual task for building in preparation to deploy to # https://f-droid.org. The intention is for it to be run locally using # `gitlab-runner` each time a tag is found that is signed by a key in # the whitelist keyring. Invoke like so: # # gitlab-runner exec docker f-droid.org --pre-build-script ./prepare-for-deploy.py \ # --docker-volumes "/root/deploy-whitelist-keyring.gpg:/root/.gnupg/pubring.gpg:ro" \ # --docker-volumes `pwd`/_site:/builds/output # # And when it is finished, you should have a directory in _site/build/ # which includes the entire static site ready to be deployed to # https://f-droid.org.
See 'future improvements' section below for additional use cases that can be expanded upon later.
We are looking for a better way to provide this functionality in a way that is as close as possible to the "proper" run, but also simple to maintain. The idea for this at the moment is to implement a way for a local runner to not only run a single job, but to run an entire pipeline if wanted. This would work in a maintainable way by:
- Local runner receives a request to run a pipeline or job
- Runner passes the request off to the remote GitLab instance for evaluation
- GitLab evaluates the job and return needed values (for which the user has permission)
- Local runner executes the pipeline or job locally
We want to keep this as simple and straightforward as possible without requiring a lot of configuration to setup.
Note that this does not, for now, address the use case of the completely disconnected person who wants to use Git/GitLab CI without internet access. It will be potentially possible in the future to add an offline mode where all values are provided by the local user (#3610).
In order to actually build out this capability, we will:
gitlab-runner lintwhich will fetch current
.gitlab-ci.ymland send it to GitLab, GitLab will validate and return syntax failures. We will have to add a corresponding
GitLab Railsendpoint to achieve that. The difficulty here is that we will try to detect GitLab URL from
git remote origin. It is easy for
https://slightly more difficult for SSH.
- Based on work of
gitlab-runner lintwe will extend
gitlab-runner execto use the given API endpoint. This will work in a context of the global instance without any access to project specific things.
gitlab-runner execto be able to authenticate as a user and evaluate
gitlab-ci.ymlin context of given project (read from
git remote origin). We will allow accessing variables for masters of the project and when a
--access-variableswill be passed to
gitlab-runner exec. This will allow testing the job as-is executed by GitLab Runner remotely. This means that since we will be evaluating that in context of the project and existing pipelines, this will also support artifacts as they are accessed at GitLab. This will require some auditing feature to store that this action was executed on GitLab, otherwise, we will not have a trace of that happening. The difficulty here is to access/store user-credentials.
The end story after implementing 3., it will allow us to support all
.gitlab-ci.yml features that we support today, and allow for easy tests configuration and run locally if needed. Accessing variables will be done through the authentication token provided, ensuring that no variables that the person running exec does not have access to are provided.
The following technical discussions need to take place to confirm this issue is ready:
- Are artifacts truly part of the MVC? For a pipeline using intra-pipeline artifacts that would be necessary, but where will they be stored for local runs?
- Technical assessment needed to ensure both jobs & pipelines is possible in MVC; if not, how can we best sequence
- More specifics needed on how authentication will work to get variables/evaluation
- How does exec do linting today, if at all?
- Could the linter be published as a module that would allow offline use, for if we later offer this as an offline feature?
- Alternate idea for variables - unlocks offline mode? #2797 (comment 104158578)
Because this feature was previously marked as being limited, upon implementing this issue we should mark it as no longer limited in that way and update associate documentation with the new way of working.
- Another potential goody would be to prepare a build container (including
pre_scripts) but instead of starting the
scriptsection, drop into a shell (or an arbitray command), e.g.
gitlab-runner job_debug docker JOBNAME /bin/bash(#3609)
- Advanced use case of using GitLab.com runners (or other instance runners) on your local code, instead of just running on local machine (https://gitlab.com/gitlab-org/gitlab-ee/issues/4176)
- Completely offline mode which does not attempt to contact the GitLab remote and instead requires the user to provide all environment variables/any other server provided information. (#3610)
- Add parameter to drop into shell on failure, for troubleshooting (#1650 (closed))
At the time of deprecation, the
exec feature had the following capabilities:
TODO: @jlenny - add column indicating which "no" answers will become "yes" with this feature.
Compatibility table - features based on
|GitLab CI feature||Available with
|image||yes||extended configuration (
|services||yes||extended configuration (
|before_script||yes||supports both global and job-level
|variables||yes||Supports default (partially), global and job-level variables; default variables are pre-set as can be seen in https://gitlab.com/gitlab-org/gitlab-runner/blob/master/helpers/gitlab_ci_yaml_parser/parser.go#L147|
|cache||partially||Regarding the specific configuration it may or may not work as expected|
|hidden keys||no||If explicitly asked to run,
|YAML features||yes||Anchors (
|pages||partially||Job's script will be executed if explicitly asked, but it doesn't affect pages state, which is managed by GitLab|
Compatibility table - features based on variables
|GitLab CI feature||Available with
|ARTIFACT_DOWNLOAD_ATTEMPTS||no||artifacts are not supported|
Compatibility table - other features
|GitLab CI feature||Available with
|job timeout||no||hardcoded to 1 hour|
- Unable to exec any docker job locally from Windows 10 host (with Docker Toolbox)
- Unable to use
- Some YAML anchor not parsed