Initial version of Configure-Container for viewing; this only sets up an SSH_PRIVATE_DEPLOY_KEY.

What does this MR do?

This adds a new template.

This is a template of a different kind. It doesn't run any particular task. Instead, it configures a Docker container container to run another job.

The central idea is to allow any Docker container to pick up and run any job for any GitLab instance.

But this first demonstration doesn't actually do that. You can see a bit of the ultimate idea at https://gitlab.com/shell-bootstrap-scripts/shell-bootstrap-scripts/-/blob/master/.gitlab-ci.yml.

include:
  - project: 'shell-bootstrap-scripts/shell-bootstrap-scripts'
    file: 'before_script.yaml'

test_centos:
  image: centos
  script:
  - dnf upgrade-minimal --assumeyes
  - dnf install --assumeyes --setopt=install_weak_deps=False wget curl
  - dnf install --assumeyes --setopt=install_weak_deps=False epel-release
  - dnf clean all
  - curl http://www.google.com
  - curl https://www.google.com

test_python:
  script:
  - wget http://www.google.com
  - rm index.html
  - git clone https://github.com/cookiecutter/cookiecutter.git
  - rm -r -f cookiecutter/
  - pip install git+https://github.com/cookiecutter/cookiecutter.git

These commands might be running, for example, behind a proxy that MITMs all traffic. And yet we can take the standard centos Docker image (for example) and access the Internet.

We could bake the network settings into the Docker image, but we'd then have to store that Docker image in the local GitLab instance's Docker registry, we'd have to do that for every Docker image we might ever want to use, which is obviously infeasible.

So instead, the before_script configures the Docker image.

The script can then be standard across all networks and all GitLab instances, including, for example, GitLab.com.

In particular, this allows using standard job templates of the ordinary kind (provided they do not use before_script themselves).

But as mentioned, this MR in its current state doesn't actually do everything. You can see in https://gitlab.com/shell-bootstrap-scripts/shell-bootstrap-scripts/-/blob/master/before_script.yaml that there's quite a lot to do, and most of it is very hard to read and comprehend.

Before getting into all that, I wanted to start a conversation about higher-level design, how to put together a "mix-in" like this.

The real purpose of this MR is simply to make it clearer what we're talking about, since otherwise it can be difficult to get across what it means to have this kind of "mix-in" that doesn't correspond to any particular job, even though on a technical level it is a job.

https://docs.gitlab.com/ee/development/cicd/templates.html#template-authoring-guidelines has some strong language about using default, so this demonstration version does not use default. Instead, each job must extends: .configure_container individually.

However, I kind of want to push back on that a little, since this seems like a case where you really do want default. The entire point of this mix-in is to apply to any script running on any image, to allow the standard script to "just work".

More importantly, there might be a third option that I haven't thought of, better than either.

One thing I'm definitely wondering about is whether before_script is the best way to do this at all.

Fundamentally, what we want here is a "modifier" to any given job, that inserts a set of commands to run before that job, no matter how that initial job is defined.

As far as I'm aware, there's no way to do multiple inheritance with GitLab-CI jobs, and even if there were we'd have the question of what to do if the job already defined before_script.

So the only ways I know to accomplish this "insert commands before any given job" business is to either have the job extends: .configure_container in its own initial definition, or use default.

Either way, this is useless if the job to be modified already uses before_script. I figure this is still useful applying to some jobs, but if there is a way to automatically insert the commands to run before any job, that would be superior.

As a bit of an afterthought, I should describe the specific bit that's getting inserted in before_script here.

All this does is set up the ability to git clone a private git repository using a provided SSH_PRIVATE_DEPLOY_KEY. The deploy key must, of course, be enabled on the target repository separately. This seemed like an easy place to start since it's a no-op if SSH_PRIVATE_DEPLOY_KEY is not specifically defined.

This shows the overall concept of how these things work. Note that the repository-to-clone is not set as a variable, or anything. The before_script never knows what it is, never acknowledges it. Instead, you git clone git@gitlab.com:shell-bootstrap-scripts/shell-bootstrap-scripts.git (for example) in the script just as you normally would. It's just that the busywork has already been taken care of automagically (if you have a deploy key enabled and set somewhere; we can't automate that part).

Fetching all the network settings to get through a proxy is another can of worms, since those settings have to be stored somewhere. Perhaps perversely, SSH_PRIVATE_DEPLOY_KEY is simpler to talk about precisely because you don't want to make the private key available all across the GitLab instance, so it always requires a manual step. The really important things do not require a manual step every time, because we can make all the necessary settings available across the GitLab instance. You can see above that the

include:
  - project: 'shell-bootstrap-scripts/shell-bootstrap-scripts'
    file: 'before_script.yaml'

is all that is necessary to allow normal scripts to work.

Screenshots or Screencasts (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.
• This change is backwards compatible across updates, or this does not apply.

Availability and Testing

• I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
• I have tested this MR in all supported browsers, or it's not needed.
• I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team