Commit 83817464 authored by Brian Lee's avatar Brian Lee

Merge branch 'version-4.1' into 'master'

Version 4.1

Closes #12 and #9

See merge request !23
parents e8e010d7 d4b5c77f
# Cangelog
## 4.1.0
* [env-setup/common] update install_tools function
* [setup/upgrade] Add a migration tool from version 2 and 3 to 4 and upper
* [ci-toolbox] Add OS_RELEASE_INFO hashmap
* [ci-toolbox] Fix in docker script part to be more portable on distribution like Fedora
* [wrapper] Move wrapper directory outside of tools/ci-toolbox
* [ci-toolbox] Increase portability by using /usr/bin/env in shebang
* [README] README files update
## 4.0.0
* [README] Add French version
* [LICENSE] Remove instruction section at the end
......
......@@ -3,15 +3,15 @@
Cette boîte à outils peut être utilisée pour
* exécuter une tâche Gitlab en dehors d'un exécuteur de tâche Gitlab (outil `gitlab-runner`) sur votre poste de travail (fonction de base)
* écrire la partie script en utilisant une sctructure modulaire (fonction avancée)
* écrire la partie script en utilisant une structure modulaire (fonction avancée)
## Table des matières
* [Le projet](#le-projet)
* [Installation des outils ci-toolbox](#installation-des-outils-ci-toolbox)
* [Mise à jours des outils ci-toolbox](#mise-à-jours-des-outils-ci-toolbox)
* [Installation de l'outil ci-toolbox](#installation-de-loutil-ci-toolbox)
* [Mise à jours de l'outil ci-toolbox](#mise-à-jours-de-loutil-ci-toolbox)
* [Intégration dans une arborescence projet](#intégration-dans-une-arborescence-projet)
* [Mise à jour des outils ci-toolbox du projet](#mise-à-jour-des-outils-ci-toolbox-du-projet)
* [Mise à jour de l'outil ci-toolbox du projet](#mise-à-jour-de-loutil-ci-toolbox-du-projet)
* [Cas d'utilisation : Tâche de pipeline Gitlab standard](#cas-dutilisation--tâche-de-pipeline-gitlab-standard)
* [Exécuter une tâche spécifique localement](#exécuter-une-tâche-spécifique-localement)
* [Première solution : utiliser l'outil gitlab-runner](#première-solution--utiliser-loutil-gitlab-runner)
......@@ -29,14 +29,14 @@ Cette boîte à outils peut être utilisée pour
## Le projet
Les outils ci-toolbox penvent être installé dans le système et/ou intégré dans l'arborescence de votre project (ce qui est utile pour forcer l'équipe du projet à utiliser une version spécigique et ainsi maîtriser les mises à jours)
L'outil ci-toolbox peut être installé dans le système et/ou intégré dans l'arborescence de votre projet (ce qui est utile pour forcer l'équipe du projet à utiliser une version spécifique et ainsi maîtriser les mises à jours)
La commande ci-toolbox (`/usr/local/bin/ci-toolbox`) est un outil intermédiaire qui va appeler `tools/ci-toolbox/ci-toolbox.sh` depuis le projet si il est trouvé dedans, autrement, il va exécuter celui installé dans le système (`/usr/local/lib/ci-toolbox/ci-toolbox.sh`)
### Installation des outils ci-toolbox
### Installation de l'outil ci-toolbox
L'installation peut se faire :
* Manuellement en clonant le porget et en lançant le script `./tools/ci-toolbox/ci-toolbox.sh setup`
* Manuellement en clonant le projet et en lançant le script `./tools/ci-toolbox/ci-toolbox.sh setup`
* En copiant ce qui suit dans une invite de commande (en remplaçant si besoin 'master' par le numéro de version choisi dans l'URL) :
```bash
......@@ -49,13 +49,13 @@ mkdir -p /tmp/ci-toolbox \
&& rm -rf /tmp/ci-toolbox
```
### Mise à jours des outils ci-toolbox
### Mise à jours de l'outil ci-toolbox
Il suffit de taper la commande `ci-toolbox setup --component ci-tools` ou `ci-toolbox setup --component ci-tools <custom_version>`
### Intégration dans une arborescence projet
Cas d'intégration des outils ci-toolbox dans une arborescence projet :
Cas d'intégration de l'outil ci-toolbox dans une arborescence projet :
```
├── .gitlab-ci.yml
├── tools
......@@ -91,9 +91,9 @@ Liste des éléments pour une utilisation avancée :
* `jobs` : dossier contenant les fonctions spécifiques aux différentes tâches
* `modules` : dossier contenant les modules
### Mise à jour des outils ci-toolbox du projet
### Mise à jour de l'outil ci-toolbox du projet
La commande `ci-toolbox update` ou `ci-toolbox update <custom_version>` permet de mettre à jour les outils ci-toolbox de votre projet
La commande `ci-toolbox update` ou `ci-toolbox update <custom_version>` permet de mettre à jour de l'outil ci-toolbox de votre projet
## Cas d'utilisation : Tâche de pipeline Gitlab standard
......
......@@ -3,15 +3,15 @@
This toolbox can be used to
* run Gitlab-CI jobs outside a runner on your workstation (classical use case)
* write the script part using a modularized format (advanced use case)
* write the script part using a modular format (advanced use case)
## Table of contents
* [The project](#the-project)
* [System side ci-toolbox tools setup](#system-side-ci-toolbox-tools-setup)
* [System side ci-toolbox tools update](#system-side-ci-toolbox-tools-update)
* [ci-toolbox tools integration in a project](#ci-toolbox-tools-integration-in-a-project)
* [ci-toolbox tools upgrade from a project](#ci-toolbox-tools-upgrade-from-a-project)
* [System side ci-toolbox tool setup](#system-side-ci-toolbox-tool-setup)
* [System side ci-toolbox tool update](#system-side-ci-toolbox-tool-update)
* [ci-toolbox tool integration in a project](#ci-toolbox-tool-integration-in-a-project)
* [ci-toolbox tool upgrade from a project](#ci-toolbox-tool-upgrade-from-a-project)
* [Use case: Standard Gitlab-CI pipeline job](#use-case-standard-gitlab-ci-pipeline-job)
* [Run a specific job locally](#run-a-specific-job-locally)
* [First solution: using gitlab-runner tool](#first-solution-using-gitlab-runner-tool)
......@@ -29,11 +29,11 @@ This toolbox can be used to
## The project
The ci-toolbox tools can be installed into the system and/or also integrated into a project tree (useful to force the project team to use a specific version)
The ci-toolbox tool can be installed into the system and/or also integrated into a project tree (useful to force the project team to use a specific version)
The ci-toolbox command (`/usr/local/bin/ci-toolbox`) is a wrapper which will execute `tools/ci-toolbox/ci-toolbox.sh` from the project if found here, otherwise, the ci-toolbox wrapper will use this stored into system (`/usr/local/lib/ci-toolbox/ci-toolbox.sh`)
### System side ci-toolbox tools setup
### System side ci-toolbox tool setup
You can perform this setup (where `master` can be changed by a specific version in the URL):
* Manually by cloning this project and execute `tools/ci-toolbox/ci-toolbox.sh setup`
......@@ -49,13 +49,13 @@ mkdir -p /tmp/ci-toolbox \
&& rm -rf /tmp/ci-toolbox
```
### System side ci-toolbox tools update
### System side ci-toolbox tool update
You can update simply with the command `ci-toolbox setup --component ci-tools` or `ci-toolbox setup --component ci-tools <custom_version>`
### ci-toolbox tools integration in a project
### ci-toolbox tool integration in a project
Case of integration of ci-toolbox tools into your project:
Case of integration of ci-toolbox tool into your project:
```
├── .gitlab-ci.yml
├── tools
......@@ -91,9 +91,9 @@ The advanced additional element list:
* `jobs`: directory containing job specific functions
* `modules`: directory containing the modules
### ci-toolbox tools upgrade from a project
### ci-toolbox tool upgrade from a project
You can update embedded ci-toolbox tools simply by run `ci-toolbox update` ou `ci-toolbox update <custom_version>`
You can update embedded ci-toolbox tool simply by run `ci-toolbox update` ou `ci-toolbox update <custom_version>`
## Use case: Standard Gitlab-CI pipeline job
......
......@@ -29,9 +29,9 @@ case "$CITBX_SETUP_TYPE" in
_sudo chmod +x /usr/local/lib/ci-toolbox/ci-toolbox.sh
_sudo cp -v $CITBX_SRC_DIR/tools/ci-toolbox/env-setup/* /usr/local/lib/ci-toolbox/env-setup/
_sudo cp -v $CITBX_SRC_DIR/tools/ci-toolbox/3rdparty/bashopts.sh /usr/local/lib/ci-toolbox/3rdparty/
_sudo cp -v $CITBX_SRC_DIR/tools/ci-toolbox/wrapper/ci-toolbox /usr/local/bin/
_sudo cp -v $CITBX_SRC_DIR/wrapper/ci-toolbox /usr/local/bin/
_sudo mkdir -p /etc/bash_completion.d
_sudo cp -v $CITBX_SRC_DIR/tools/ci-toolbox/wrapper/bashcomp /etc/bash_completion.d/ci-toolbox
_sudo cp -v $CITBX_SRC_DIR/wrapper/bashcomp /etc/bash_completion.d/ci-toolbox
if [ -f /usr/local/lib/ci-toolbox/ci-toolbox.properties ]; then
_sudo cp -v $CITBX_SRC_DIR/tools/ci-toolbox/ci-toolbox.properties /usr/local/lib/ci-toolbox/ci-toolbox.properties.default
else
......
set -e
# Print an error message and exit with error status 1
print_critical() {
>&2 printf "\033[91m[CRIT] %s\033[0m\n" "$@"
exit 1
}
# Print a note message
print_note() {
printf "[NOTE] %s\n" "$@"
}
# Pring an info message
print_info() {
printf "\033[92m[INFO] %s\033[0m\n" "$@"
}
CITBX_TMPDIR=$(mktemp -d)
install_finish() {
rm -rf $CITBX_TMPDIR
}
trap install_finish EXIT SIGHUP SIGINT SIGQUIT SIGABRT SIGKILL SIGALRM SIGTERM
CI_PROJECT_DIR="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [ -z "$CI_PROJECT_DIR" ]; then
print_critical "Unable to find the project root directory"
fi
CITBX_CUR_VERSION_MAJOR=$(find "$CI_PROJECT_DIR" -type f -regex \
"$CI_PROJECT_DIR"'/tools/\(ci-toolbox\|gitlab-ci\)/\(run\|gitlab-ci\)\.sh' \
-exec sed -E 's/^CITBX_VERSION=([0-9]+).*$/\1/;t;d' '{}' \;)
if [ "$CITBX_CUR_VERSION_MAJOR" != "2" ] && [ "$CITBX_CUR_VERSION_MAJOR" != "3" ]; then
print_critical "This tool can't be used with version other than 2 and 3"
fi
# Move ci-toolbox directory
if [ -d "$CI_PROJECT_DIR/tools/gitlab-ci" ]; then
mv -v $CI_PROJECT_DIR/tools/{gitlab-ci,ci-toolbox}
fi
# Move job directory
if [ -d "$CI_PROJECT_DIR/tools/ci-toolbox/run.d" ]; then
mv -v $CI_PROJECT_DIR/tools/ci-toolbox/{run.d,jobs}
fi
# Move ci-toolbox tool
if [ -f "$CI_PROJECT_DIR/tools/ci-toolbox/run.sh" ]; then
mv -v $CI_PROJECT_DIR/tools/ci-toolbox/{run,ci-toolbox}.sh
fi
# Move ci-toolbox properties
if [ -f "$CI_PROJECT_DIR/tools/ci-toolbox/citbx.properties" ]; then
mv -v $CI_PROJECT_DIR/tools/ci-toolbox/{citbx,ci-toolbox}.properties
fi
# Remove link
if [ -L "$CI_PROJECT_DIR/tools/ci-toolbox/run" ]; then
rm -v "$CI_PROJECT_DIR/tools/ci-toolbox/run"
fi
# Install the last version
CITBX_ABS_DIR="$CI_PROJECT_DIR/tools/ci-toolbox"
print_note "Downloading archive from gitlab.com..."
curl -fSsL https://gitlab.com/ercom/citbx4gitlab/repository/master/archive.tar.bz2 | tar -C $CITBX_TMPDIR -xj
# Execute setup process from the remote archive
CITBX_SRC_DIR="$(readlink -f $CITBX_TMPDIR/citbx4gitlab-*)"
CITBX_SETUP_TYPE="project"
BASHOPTS_FILE_PATH="$(find $PWD -name bashopts.sh)"
source $CITBX_SRC_DIR/setup/install.sh
print_info "Migration done!"
#!/bin/bash -e
#!/usr/bin/env bash
# citbx4gitlab: CI toolbox for Gitlab
# Copyright (C) 2017-2018 ERCOM - Emeric Verschuur <emeric@mbedsys.org>
#
......@@ -15,13 +15,15 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
CITBX_VERSION=4.0.0
CITBX_VERSION=4.1.0
############################################################
# ### COMMON PART ###
# All environments: Gitlab-CI runner & local workstation
############################################################
set -e
# display a message
print_log() {
local level=$1;
......@@ -238,6 +240,10 @@ if [ "$CITBX" == "true" ]; then
fi
export CITBX="true"
# Collect OS release information
declare -A OS_RELEASE_INFO
eval "$(sed -E 's/^([^=]+)=(.*)$/OS_RELEASE_INFO[\1]=\2/g' /etc/os-release)"
# YAML to JSON convertion
yaml2json() {
cat "$@" | python -c 'import sys, yaml, json; json.dump(yaml.load(sys.stdin), sys.stdout)'
......@@ -492,13 +498,7 @@ $(for j in "${CITBX_JOB_LIST[@]}"; do echo " $j"; done | sort -u)"
-x "(${CITBX_SETUP_COMPONENT_DEFAULT_LIST[*]})"
bashopts_declare -n CITBX_SETUP_OS_ID -l os-id -i \
-t string -d "Operating system ID" \
-v "$(
if which lsb_release > /dev/null 2>&1; then
lsb_release --id --short
elif [ -f /etc/os-release ]; then
eval "$(sed 's/^NAME=/echo /;tx;d;:x' /etc/os-release)"
fi
)"
-v "${OS_RELEASE_INFO[NAME]}"
check_tool_name() {
if [[ "$1" =~ ^[a-zA-Z0-9_-]+$ ]]; then
echo $1
......@@ -819,6 +819,22 @@ for hook in $citbx_job_stage_setup; do
$citbx_after_script
done
CITBX_SCRIPT_COMMON='
_which() {
for p in $(echo "$PATH" | tr : \ ); do
if [ -x "$p/$1" ]; then
echo "$p/$1"
return 0
fi
done
return 1
}
if ! SHELL="$(_which bash || _which sh)"; then
echo shell not found
exit 1
fi
'
case "$CITBX_JOB_EXECUTOR" in
shell)
print_info "Running the job \"$CI_JOB_NAME\" into the shell..."
......@@ -832,23 +848,7 @@ case "$CITBX_JOB_EXECUTOR" in
eval "export $e=${CITBX_SHELL_ENV[$e]}"
done
# Shell selector (like the gitlab-runner one) will try to find bash, otherwise sh
eval "if [ -x /usr/local/bin/bash ]; then
SHELL=/usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
SHELL=/usr/bin/bash
elif [ -x /bin/bash ]; then
SHELL=/bin/bash
elif [ -x /usr/local/bin/sh ]; then
SHELL=/usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
SHELL=/usr/bin/sh
elif [ -x /bin/sh ]; then
SHELL=/bin/sh
else
echo shell not found
exit 1
fi
\$SHELL -c $CITBX_JOB_SCRIPT"
eval "$CITBX_SCRIPT_COMMON"'$SHELL -c '"$CITBX_JOB_SCRIPT"
)
;;
docker)
......@@ -861,6 +861,7 @@ case "$CITBX_JOB_EXECUTOR" in
if [ -f "$HOME/.docker/config.json" ]; then
CITBX_JOB_DOCKER_RUN_ARGS+=(-v $HOME/.docker/config.json:/root/.docker/config.json:ro)
fi
CITBX_DOCKER_SCRIPT_PARTS+=("$CITBX_SCRIPT_COMMON")
if [ "$CITBX_UID" -ne 0 ]; then
CITBX_USER=$USER
if [ -f "$HOME/.docker/config.json" ]; then
......@@ -868,20 +869,45 @@ case "$CITBX_JOB_EXECUTOR" in
fi
if [ "$CITBX_DOCKER_USER" == "root" ]; then
CITBX_DOCKER_SCRIPT_PARTS+=('
if which useradd > /dev/null 2>&1; then
useradd -o -u '"$CITBX_UID"' -s /bin/sh -d '"$HOME"' -M '"$CITBX_USER"'
elif readlink -f "$(which adduser)" | grep -q /busybox$ > /dev/null 2>&1; then
busybox adduser -u '"$CITBX_UID"' -s /bin/sh -h '"$HOME"' -H -D '"$CITBX_USER"'
# _adduser <name> <uid> <home>
if _which useradd > /dev/null 2>&1; then
_adduser() {
useradd -o -u $2 -s /bin/sh -d $3 -M $1
}
elif _which busybox > /dev/null 2>&1; then
_adduser() {
busybox adduser -u $2 -s /bin/sh -h $3 -H -D $1
}
else
echo "[!!] No usual tool found to add an user"
exit 1
fi
# _adduser2group <user> <group>
if _which usermod > /dev/null 2>&1; then
_adduser2group() {
usermod -a -G $2 $1
}
elif _which busybox > /dev/null 2>&1; then
_adduser2group() {
addgroup $1 $2 > /dev/null
}
else
echo "[!!] No usual tool found to add an user to a group"
exit 1
fi
# Add the user
_adduser '"$CITBX_USER"' '"$CITBX_UID"' '"$HOME"'
# Fix rights on HOME directory
chown '"$CITBX_UID"':'"$CITBX_UID"' '"$HOME"'
# Add user to the suitable groups
for group in '"${CITBX_USER_GROUPS[*]}"'; do
if grep -q ^$group /etc/group; then
addgroup '"$CITBX_USER"' $group > /dev/null
_adduser2group '"$CITBX_USER"' $group
fi
done
# Add sudoers suitable rule
if [ -f /etc/sudoers ]; then
sed -i "/^'"$CITBX_USER"' /d" /etc/sudoers
echo "'"$CITBX_USER"' ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
......@@ -892,26 +918,6 @@ case "$CITBX_JOB_EXECUTOR" in
CITBX_USER=root
fi
# Shell selector (like the gitlab-runner one) will try to find bash, otherwise sh
CITBX_DOCKER_SCRIPT_PARTS+=('
echo "ENV_PATH PATH=$PATH" >> /etc/login.defs
echo "ENV_SUPATH PATH=$PATH" >> /etc/login.defs
if [ -x /usr/local/bin/bash ]; then
SHELL=/usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
SHELL=/usr/bin/bash
elif [ -x /bin/bash ]; then
SHELL=/bin/bash
elif [ -x /usr/local/bin/sh ]; then
SHELL=/usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
SHELL=/usr/bin/sh
elif [ -x /bin/sh ]; then
SHELL=/bin/sh
else
echo shell not found
exit 1
fi
')
CITBX_DOCKER_SCRIPT_PARTS+=('su '"$CITBX_USER"' -s "$SHELL"')
if [ "$CITBX_RUN_SHELL" != "true" ]; then
CITBX_DOCKER_SCRIPT_PARTS+=("-c $CITBX_JOB_SCRIPT")
......
......@@ -67,7 +67,7 @@ install_tools() {
mkdir -p $tmpdir
curl -fSsL https://gitlab.com/ercom/citbx4gitlab/repository/$version/archive.tar.bz2 | tar -C $tmpdir -xj
# Execute setup process from the remote archive
CITBX_SRC_DIR="$tmpdir/$(ls -1 $tmpdir)"
CITBX_SRC_DIR="$(readlink -f $tmpdir/citbx4gitlab-$version-*)"
CITBX_SETUP_TYPE=$1
source $CITBX_SRC_DIR/setup/install.sh
print_info "done!"
......
......@@ -45,7 +45,7 @@ if setup_component_enabled base-pkgs; then
fi
if setup_component_enabled git-lfs; then
if [ $(eval "$(sed 's/^VERSION_ID=/echo /;tx;d;:x' /etc/os-release)" | tr -d '.' ) -lt 1710 ]; then
if [ ${OS_RELEASE_INFO[VERSION_ID]/./} -lt 1710 ]; then
if grep -qr 'git-lfs' /etc/apt/; then
print_note "GIT LFS apt repository is already present."
else
......
#!/bin/bash -e
#!/usr/bin/env bash
# citbx4gitlab: CI toolbox for Gitlab
# Copyright (C) 2017-2018 ERCOM - Emeric Verschuur <emeric@mbedsys.org>
#
......
#!/bin/bash -e
#!/usr/bin/env bash
# citbx4gitlab: CI toolbox for Gitlab
# Copyright (C) 2017-2018 ERCOM - Emeric Verschuur <emeric@mbedsys.org>
#
......@@ -15,6 +15,8 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
set -e
# Print an error message and exit with error status 1
print_critical() {
>&2 printf "\e[91m[CRIT] %s\e[0m\n" "$@" \
......@@ -48,19 +50,19 @@ for d in $CITBX_SEARCH_PATH; do
d="$CI_PROJECT_DIR/$d"
fi
if is_bash_file "$d/ci-toolbox.sh"; then
CITBX_WRAPPER_PATH="$d/ci-toolbox.sh"
CITBX_CI_TOOLBOX_PATH="$d/ci-toolbox.sh"
break
fi
# Assure backward compatibility with CITBX_VERSION < 4
if is_bash_file "$d/run.sh"; then
CITBX_WRAPPER_PATH="$d/run.sh"
CITBX_CI_TOOLBOX_PATH="$d/run.sh"
break
fi
add_debug_info "ci-toolbox.sh or run.sh not found in directory $d"
done
IFS=$_IFS
if [ -z "$CITBX_WRAPPER_PATH" ]; then
if [ -z "$CITBX_CI_TOOLBOX_PATH" ]; then
print_critical "Unable to find ci-toolbox.sh script" \
" => You may have to update the CITBX_SEARCH_PATH to add the suitable ci-toolbox.sh relative directory from the project root" \
" e.g.: add export CITBX_SEARCH_PATH=\"$CITBX_SEARCH_PATH:your/custom/dir\" to your environment"
......@@ -74,11 +76,11 @@ if [ -n "$CITBX_BASHCOMP" ]; then
}
NR > 50 {
exit
}' $CITBX_WRAPPER_PATH 2> /dev/null || true)
}' $CITBX_CI_TOOLBOX_PATH 2> /dev/null || true)
# skip bash completion for CITBX_VERSION < 3.x.x
test "0${CITBX_VERSION%%.*}" -ge 3 || exit
fi
export CITBX_TOOL_NAME=ci-toolbox
exec $CITBX_WRAPPER_PATH "$@"
print_critical "Unable to run $CITBX_WRAPPER_PATH tool"
exec $CITBX_CI_TOOLBOX_PATH "$@"
print_critical "Unable to run $CITBX_CI_TOOLBOX_PATH tool"
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment