Docs and Provision updates [skip ci]

CHANGELOG.md

 # Changelog
 
+## [1.4.0] - 2024-09-02
+
+### Added
+
+- Documentation on supported "Device Cloud Runner Modes" in [SETUP.md](Device-Cloud-Pool-Mgmt/SETUP.md)
+- Documentation on various settings for hardware compatibility on Docker Executor and Docker itself in [ContainerRunnersForEmbeddedGuide.md](Device-Cloud-Pool-Mgmt/ContainerRunnersForEmbeddedGuide.md)
+- **--additionalrunnerregargs** arg for PROVISIONDEVICEGWRUNNER.sh - arguments passed directly to runner register command.
+- **--installdocker** arg for PROVISIONDEVICEGWRUNNER.sh - used to force install of docker when --runnerexecutor is 'shell' in order to have docker as a sidecar process to a shell executor
+- PROVISIONDEVICEGWRUNNER.sh - added support for --cache-dir and --docker-cache-dir
+
 ## [1.3.0] - 2024-08-20
 
 ### Added

Device-Cloud-Pool-Mgmt/ContainerRunnersForEmbeddedGuide.md

 
 ## Important Runtime Execution Modes
 
-### Gitlab Runner Shell Executor with Docker Sidecar
+The runner registration switches pertains to the Docker Executor configuration of the Device Cloud runners. Other modes are [discussed here](SETUP.md).
 
-Many guides for using Docker for build tools assume that the build script is running *along side* docker. Frequently this is because the examples are for local workstation development environments. Some CI systems run in this arrangement as well.
-
-GitLab CI can support this mode if you register it as a shell runner and install docker on the same machine.
-
-### GitLab Runner Docker Executor
-
-However, when your ci jobs are using the `image:` keyword, the GitLab CI runs the build script **INSIDE** the containers that it uses.  This is a fundamentally different approach for embedded because simply using a container while running on the host means the actual script has hardware access and interfaces - when running inside the container, hardware and permissions must be mapped into the container.
-
-### GitLab Runner Shell Executor
-
-GitLab Runner also supports pure Shell runners and allows build scripts used in local development environments to be used directly in CI jobs.
+In the Docker Sidecar configuration has the runner job scripts running on the real OS, so does not use these switches. If similar configuration is required for Docker Side car, you would simply configure docker on the host normally or use docker CLI switches when starting the docker container or docker compose in a GitLab Job script. 
 
 ## Current Parameters Used By Device Cloud for Docker Executor
 

Device-Cloud-Pool-Mgmt/PRO.ps1

@@ -158,7 +158,7 @@ function Register-HILGatewayRunner {
     }
 
     if ($runnerexecutor -eq "docker") {
-        $EXTRAREGARGS = "--docker-volumes '/certs/client' --docker-image 'bash' --docker-privileged --docker-pull-policy if-not-present"
+        $EXTRAREGARGS = "--docker-volumes '/certs/client' --docker-image 'bash' --docker-privileged --docker-pull-policy if-not-present --docker-devices /dev/bus/usb --docker-devices /dev/ttyACM0"
         if (-not (Get-Command docker -ErrorAction SilentlyContinue)) {
             Write-Host "Docker not present, installing..."
             #????TODO install docker?

Device-Cloud-Pool-Mgmt/PROVISIONDEVICEGWRUNNER.sh

@@ -44,6 +44,8 @@ Defaulted Parameters
 --devicepoolgroupname - if --devicepooldataprojectpath is not provided, this cannot be autopopulated, so must be provided manually.
 --runnerdescription - if --devicepooldataprojectpath is not provided, this cannot be autopopulated, so must be provided manually.
 --runnerexecutor - defaults to 'shell', can be set to 'docker'. If you need to run local host commands like mounting media, the docker executor will not be helpful until GitLab runner can pass through pid options as note in [Missing pid settings for runners docker configuration](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/36847) 
+--additionalrunnerregargs - arguments passed directly to runner register command.
+--installdocker - used to force install of docker when --runnerexecutor is 'shell' in order to have docker as a sidecar process to a shell executor
 --hostnamematchrunnertag - force hostname to match runnertag, defaults to false. Hostname has to remove dash and underscore. It is visible in runner logs so can help with 
 --devicepooluniqueid - defaults to using random identifier, can be set to hostname or to custom value with --devicepooluniqueid=hostname or --devicepooluniqueid={customvaluethatisuniqueamongallinthepool}
 --verbose - verbose update
@@ -110,11 +112,7 @@ function registerhilgatewayrunner () {
     echo "gitlab-runner ALL=(ALL) NOPASSWD: ALL" | sudo tee -a /etc/sudoers.d/gitlab-runner
   fi
   
-  if [[ "${runnerexecutor,,}" = "docker" ]]; then
-    # --docker-privileged and --docker-devices /dev/bus/usb --docker-devices /dev/ttyACM0 enable hardware access to usb connected hardware
-    # there are finer grained permissions possible https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
-    # 'seccomp' and 'pid=host' are not available - but may not be necessary with other configuration parameters that are available
-    export EXTRAREGARGS="--docker-volumes "/certs/client" --docker-image "bash" --docker-privileged --docker-pull-policy if-not-present"  --docker-devices /dev/bus/usb --docker-devices /dev/ttyACM0
+  if [[ "${runnerexecutor,,}" = "docker" ]] || [[ "$installdocker" ]]; then
     if [[ -z "$(command -v docker)" ]] ; then
       echo "Docker not present, installing..."
       sudo apt-get install -y docker.io
@@ -124,6 +122,13 @@ function registerhilgatewayrunner () {
     fi
   fi
 
+  if [[ "${runnerexecutor,,}" = "docker" ]]; then
+    # --docker-privileged and --docker-devices /dev/bus/usb --docker-devices /dev/ttyACM0 enable hardware access to usb connected hardware
+    # there are finer grained permissions possible https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
+    # 'seccomp' and 'pid=host' are not available - but may not be necessary with other configuration parameters that are available
+    export DOCKEREXECUTORARGS="--docker-volumes "/certs/client" --docker-image "bash" --docker-cache-dir "/cache" --docker-privileged --docker-pull-policy if-not-present  --docker-devices /dev/bus/usb --docker-devices /dev/ttyACM0"
+  fi
+
   sudo sed -i "s/^\s*concurrent.*/concurrent = 1/g" /etc/gitlab-runner/config.toml #1 is the default, but it is also extremely important for an Embedded Gateway Runner, so reasserted
 
   if [[ -z "${RunnerRegTokenStored}" ]]; then
@@ -133,7 +138,8 @@ function registerhilgatewayrunner () {
     --url "${gitlaburl}" \
     --token "${RunnerRegToken}" \
     --executor "${runnerexecutor,,}" \
-    ${EXTRAREGARGS}
+    --cache-dir "/cache" \
+    ${DOCKEREXECUTORARGS} ${additionalrunnerregargs}
   fi
 
   if [[ $(grep "pre_build_script =" /etc/gitlab-runner/config.toml -c) -lt 1 ]]; then  
@@ -211,6 +217,7 @@ needs_arg() { if [ -z "${OPTARG}" ]; then die "No arg for --$OPT option"; fi; }
 # Defaults for required parameters that can be overwritten by param specification
 devicepooluniqueid="random"
 runnerexecutor="shell"
+installdocker=false
 
 while getopts ab:c-: OPT; do
   if [ "$OPT" = "-" ]; then   # long option: reformulate OPT and OPTARG
@@ -227,6 +234,8 @@ while getopts ab:c-: OPT; do
 
     devicepooluniqueid )              needs_arg; devicepooluniqueid="${OPTARG}" ;;
     runnerexecutor )                  needs_arg; runnerexecutor="${OPTARG}" ;;
+    installdocker )                   installdocker=true ;;
+    additionalrunnerregargs )         needs_arg; additionalrunnerregargs="${OPTARG}" ;;
     runnerdescription )               needs_arg; runnerdescription="${OPTARG}" ;;
     devicepoolgroupname )             needs_arg; devicepoolgroupname="${OPTARG}" ;;
     

Device-Cloud-Pool-Mgmt/SETUP.md

 
-## HIL Pool Manager for GitLab Lightweight Orchestration of Hardware In the Loop for CI
+## GitLab Device Cloud Setup
+
+### Supported Runner Modes
+
+#### Device Cloud Using GitLab Runner Shell Executor
+
+GitLab Runner and the Device Cloud support pure Shell runners and allows build scripts used in local development environments to be used directly in CI jobs.
+
+| Benefits                                                     | Challenges                                                   |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| Most familiar if you have been using local build scripts directly against hardware.<br />Runner build cache is local, fast and in-place. | All the historic problems with overloaded and brittle local runtimes of using a "machine" as the job execution context that containers can help solve for.<br /><br />GitLab components and example CI that use the `image:` tag are not usable without being adjusted.Runner build cache is not automatically cleared, potentially resulting in unclean builds. |
+
+![devicecloudshell](../images/devicecloudshellexecutor.png)
+
+To configure this mode with PROVISIONDEVICEGWRUNNER.sh, simply use --runnerexecutor="shell" (shell is also the default if left blank)
+
+#### Device Cloud Using Gitlab Runner Shell Executor with Docker Sidecar
+
+Many guides for using Docker for build tools assume that the build script is running *along side* a copy of docker. Frequently this is because the examples are for local workstation development environments. Some CI systems run in this arrangement as well. This enables configuration of docker containers during launch because build scripts will have docker or docker compose commands right in them.
+
+| Benefits                                                     | Challenges                                                   |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| Adds containerized tooling benefits in the same way as using build or IDE based containers does at a developer workstation.<br />Full control over docker container launch options.<br />For containers that do not touch hardware, no docker hardware interfacing configuration is required.<br />Runner build cache is local, fast and in-place. | Requires docker commands in CI jobs.<br />GitLab components and example CI that use the `image:` tag are not usable without being adjusted.<br />Runner build cache is not automatically cleared, potentially resulting in unclean builds. |
+
+![](../images/deviceclouddockersidecar.png)
+
+To configure this mode with PROVISIONDEVICEGWRUNNER.sh, simply use --runnerexecutor="shell" --installdocker 
+
+#### Device Cloud Using GitLab Runner Docker Executor
+
+However, when your ci jobs are using the `image:` keyword, the GitLab CI runs the build script **INSIDE** the containers that it uses.  This is a fundamentally different approach for embedded because simply using a container while running on the host means the actual script has hardware access and interfaces - when running inside the container, hardware and permissions must be mapped into the container.
+
+| Benefits                                                     | Challenges                                                   |
+| ------------------------------------------------------------ | ------------------------------------------------------------ |
+| Enables the most prevalent mode of GitLab Runner container support - those that use the `image:` tag.<br />Very clean builds as containers are an emphmeral environment that is disposed at the end of every build. | Some hardware actions from within containers may require special docker executor configuration.<br />Not all docker launch options are supported by GitLab Runner config.toml or GitLab CI YAML.<br />Runner artifact and build cache are autosaved out of and propagated into each container job. |
+
+![devicecloudshellexecutor](../images/devicecloudshellexecutor.png)
+
+To configure this mode with PROVISIONDEVICEGWRUNNER.sh, simply use --runnerexecutor="docker" (a docker install is triggered by this automatically)
+
+## Using Existing Hardware Setup
+
+While it's always nice to start fresh, you can bring an existing hardware setup into the CI loop by simply provisioning the existing hardware into the Device Pool. This works even if it is attached to a developer workstation. The challenge is that you must remember to offline the runner from the orchestration if and when you want to locally attach and do work.
 
 ### Deploying This Solution