Commit 14f70632 authored by hestiahacker's avatar hestiahacker
Browse files

Initial commit

parents
Pipeline #398694112 passed with stage
in 4 minutes and 33 seconds
qemu-rpi
# This defines all the CI jobs in this repo. Stages are: build, test, deploy
# Documentation can be found here: https://docs.gitlab.com/ee/ci/yaml/#stages
# Require these tests be run under docker, so dependencies can be installed
stretch:
stage: test
tags: [ docker ]
script:
- ./build_emulator.sh arm1176 stretch
buster:
stage: test
tags: [ docker ]
script:
- ./build_emulator.sh arm1176 buster
Copyright (c) 2020 Hestia Hacker, Script Fiend
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# Overview
This repo is intended to help developers and administrators who want to write
a script to automatically set up a raspberry pi. The real target audience are
those who contribute to open source projects. For example, the first use
of this project was to develop an install script for the
[HestiaPi](https://hestiapi.com/).
The idea is you write a script, or series of scripts, that will SSH into a
stock raspberry pi (according to the project(s) you are building on), install
the packages you want, and set up configuration files to match your deployment
environment. Then, you use build_emulator.sh from this repo to test your script
before you deploy it to hardware, to make sure it doesn't (for example) cut off
your access to a potentially difficult-to-restore system.
# Getting started
First, you need to describe the state you expect the hardware to be in before
your customization script runs. To do this, you configure some environment
variables in a script named BASE_VERSION.sh. This repo already includes
configurations to emulate the official Raspbian lite distros for stretch and
buster.
Next, write your configuration script to SSH into localhost on port 5022
(which maps to port 22/sshd on your emulated pi) and do the actual
configuration. Call it CONFIG.sh.
Put the contents of this repo, your BASE_VERSION.sh if you wrote a new one,
and CONFIG.sh in the same directory. Make sure your new .sh scripts are
executable. From that working directory, run:
```sh
./build_emulator.sh CPU BASE_VERSION CONFIG.sh
```
Note that build_emulator.sh adds the .sh extension to BASE_VERSION, so you
shouldn't include the extension in the command. You can
include as many CONFIG arguments as you want; they will be run in order.
If your CONFIG.sh isn't perfect on the first iteration, simply kill QEMU (e.g.
press crtl-c on the terminal where you launched build_emulator.sh) if it is
still running, and re-run build_emulator.sh. When it asks if you want to
overwrite the .img file, say yes. This will make sure you are starting from a
fresh image.
When your script appears to run perfectly, log into the guest system and shut
it down. Then start it back up from the same working directory:
```sh
./resume.sh CPU BASE_VERSION
```
This will boot up the machine normally. You can log in on the
console to explore the emulated pi and confirm that your script worked.
# Less common tasks
## Creating new base versions
Each BASE_VERSION.sh must specify
- the locations of
- a filesystem image
- a matching kernel a suitable for use by qemu
- a compiled device tree
- login credentials for an account with sudo access on the pi
You configure build_emulator.sh to find these items by exporting environment
variables in BASE_VERSION.sh.
Some good places to look for these items:
- [official Raspberry Pi OS downloads](https://www.raspberrypi.org/downloads/raspberry-pi-os/)
- image downloads for the specific Pi project you want to use, e.g. [HestiaPi
downloads](https://hestiapi.com/downloads/)
- [the QEMU Raspberry Pi project](https://github.com/dhruvvyas90/qemu-rpi-kernel)
## Re-enabling SSH
If your last CONFIG disabled SSH, but you need to log in via ssh for some reason, re-run:
```sh
./build_emulator.sh CPU BASE_VERSION
```
Do not specify a CONFIG (unless you have an additional one that must be applied
after a reboot). When it asks if you want to overwrite the .img file, say no.
Note that if you re-enable SSH and save the VM, it will still have SSH enabled when you resume it later.
# Under the hood
The build_emulator.sh script:
- Runs BASE_VERSION.sh
- Downloads the files described in BASE_VERSION.sh
- Uses turn_on_ssh.ex to actually start running Rasbian in QEMU, automatically
log into the guest O/S, and enable SSH on guest:22/host:5022
- Runs each CONFIG.sh in the order given
There are numerous assumptions in the existing code. Please embrace and extend.
#!/usr/bin/expect -f
#
# Demo: start a process and background it after it reaches a certain stage
#
# From https://stackoverflow.com/questions/17916201/background-spawned-process-in-expect#32545624
#
#!/usr/bin/expect -f
spawn -ignore HUP perl -e "for (\$a='A';; \$a++) {print qq/value is \$a\\n/; sleep 1;}"
#set timeout 600
set timeout -1
# Detailed log so we can debug (uncomment to enable)
# exp_internal -f /tmp/expect.log 0
# wait till the subprocess gets to "G"
#expect "value is G"
#send_user "\n>>> expect: got G\n"
# when we get to G, background the process
expect_background
#send_user ">>> spawned process backgrounding successful\n"
#exit 0
#!/usr/bin/expect -f
#
# start a process and background it after it reaches a certain stage
#
# From https://stackoverflow.com/questions/17916201/background-spawned-process-in-expect#32545624
#
spawn perl -e "\$SIG{HUP} = 'IGNORE'; for (\$a='A';; \$a++) {print qq/value is \$a\\n/; sleep 1;}"
set timeout 600
# Detailed log so we can debug (uncomment to enable)
# exp_internal -f /tmp/expect.log 0
# wait till the subprocess gets to "G"
expect -ex "value is G"
send_user "\n>>> expect: got G\n"
# when we get to G, background the process
expect_background
send_user ">>> spawned process backgrounding successful\n"
exit 0
\ No newline at end of file
#!/bin/sh -e
# I build emulators using scripts you can use to configure actual Pis.
# See README.md for more information.
set -e
#
# Command line arguments
#
if [ "$#" -lt "2" ]; then
echo "Usage: $0 CPU BASE_VERSION [CONFIG ...]"
echo ""
echo " CPU - the CPU to emulate (e.g. arm1176)"
echo " BASE_VERSION - the version of raspbian to put on the emulated pi (e.g. buster)"
echo " CONFIG - the name(ish) of a script that configures the pi (e.g. hestiapi.sh)"
echo ""
exit 1
fi
export RPI_CPU="$1"
shift
export RPI_OS_VERSION="$1"
shift
#
# Environment variables
#
# The default Rasbian image is 1.5GB. If you want something bigger, change
# EXPAND_BY to be the number of MB to add on to the image.
if [ -z "$EXPAND_BY" ]; then
# Default to expanding the image by 2GB
EXPAND_BY=$((2*1024)) # 2GB
fi
#
# Do it
#
identifier=$(lsb_release -i | sed 's/.*:\s*//g' | tr [A-Z] [a-z] || true)
if [ -z "$identifier" ]; then # this happens if lsb_release isn't installed
identifier=$(grep ^ID= /etc/os-release | sed 's/.*=//g')
fi
echo "Distribution is: $identifier"
if [ "$identifier" = "debian" -o "$identifier" = "ubuntu" ]; then
# Debian-based distro; use apt to ensure we have qemu-system-arm & expect
echo "Installing qemu"
# Root does not need to use sudo
if [ "`whoami`" = "root" ]; then
sudo=""
else
sudo="sudo"
fi
export DEBIAN_FRONTEND=noninteractive
$sudo apt-get update
$sudo apt-get install -y qemu-system-arm expect wget unzip
fi
# for Macs
#brew install qemu
export QEMU=$(which qemu-system-arm)
export TMP_DIR=./qemu-rpi
. ./${RPI_OS_VERSION}.sh
mkdir -p $TMP_DIR; cd $TMP_DIR
if [ ! -f ${RPI_KERNEL} ]; then
echo "No kernel in ${RPI_KERNEL}"
echo "Fetching kernel for ${RPI_OS_VERSION}"
wget ${RPI_KERNEL_URL} -O ${RPI_KERNEL}
fi
if [ ! -f ${PTB_FILE} ]; then
echo "Nothing in ${PTB_FILE}"
wget ${PTB_FILE_URL} -O ${PTB_FILE}
fi
if [ ! -f $IMAGE_FILE ]; then
echo "No image in ${IMAGE_FILE}"
wget ${IMAGE_FILE_URL}
fi
unzip $IMAGE_FILE
# Sort by time, newest first, and take the first entry
export RPI_FS=`ls -t *-raspbian-${RPI_OS_VERSION}-lite.img | head -n 1`
if [ ! -z $EXPAND_BY ]; then
dd if=/dev/zero count=$EXPAND_BY bs=$((1024*1024)) >> $RPI_FS
fi
# SSH is disabled by default, need to enable it via the console
../turn_on_ssh.ex
#echo "ssh is up"
sleep 30
while [ $# -gt 0 ]
do
RPI_CONFIG="$1"
echo "${RPI_CONFIG}"
pwd
../${RPI_CONFIG}
shift
done
#!/bin/sh
# I set environment variables so build_emulator.sh can use a buster image.
export RPI_KERNEL=kernel-qemu-4.19.50-buster
export RPI_KERNEL_URL=https://github.com/dhruvvyas90/qemu-rpi-kernel/blob/master/kernel-qemu-4.19.50-buster?raw=true
export PTB_FILE=versatile-pb.dtb
export PTB_FILE_URL=https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/${PTB_FILE}
export IMAGE_FILE=raspbian_lite_latest
export IMAGE_FILE_URL=https://downloads.raspberrypi.org/${IMAGE_FILE}
export SDA2_START=532480
export username="pi"
export password="raspberry"
#!/bin/sh
#./build_emulator.sh arm1176 stretch hestiapi
./build_emulator.sh arm1176 buster hestiapi
#!/bin/sh -e
export host=127.0.0.1
export port=5022
#export username=pi
#export password=raspberry
#export password=hestia
# If you've ever used this script before, you'll have an old entry in your
# known hosts. We need to clean that up.
ssh-keygen -f "$HOME/.ssh/known_hosts" -R "[127.0.0.1]:5022"
echo "Pushing keys over to the VM..."
../push_keys.ex
#Start the guest VM
echo $QEMU -kernel $RPI_KERNEL \
-cpu $RPI_CPU -m 256 -M versatilepb \
-dtb $PTB_FILE -no-reboot \
-device virtio-rng-pci -nographic \
-append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
-drive "file=$RPI_FS,index=0,media=disk,format=raw" \
-net user,hostfwd=tcp::5022-:22 -net nic
$QEMU -kernel $RPI_KERNEL \
-cpu $RPI_CPU -m 256 -M versatilepb \
-dtb $PTB_FILE -no-reboot \
-device virtio-rng-pci -nographic \
-append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
-drive "file=$RPI_FS,index=0,media=disk,format=raw" \
-net user,hostfwd=tcp::5022-:22 -net nic &
echo "Waiting a minute while the Guest VM starts"
sleep 60
echo "Changing password of $username..."
ssh -o "StrictHostKeyChecking no" -i ./id_25519 -p $port $username@$host 'printf "hestia\nhestia\n" | sudo passwd pi'
password=hestia
run="ssh -o 'StrictHostKeyChecking no' -i ./id_25519 -p $port $username@$host"
echo "Password has been changed"
$run "echo 'openhab ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/020_openhab-nopasswd"
$run "sudo apt-get update && sudo apt-get -y upgrade && sudo apt-get install -y apt-transport-https bc dnsmasq hostapd vim python3-flask python3-requests git dirmngr accountsservice build-essential python quilt devscripts python-setuptools python3 libssl-dev cmake libc-ares-dev uuid-dev daemon zip zlibc zlib1g zlib1g-dev python-smbus unclutter matchbox-window-manager xwit xinit openbox lxterminal geoclue-2.0 libjavascriptcoregtk-3.0-0 libwebkitgtk-3.0-0 && sudo apt-get install -y --no-install-recommends xserver-xorg"
$run "touch /home/pi/.xinitrc && chmod 755 /home/pi/.xinitrc && printf '#!/bin/sh\nexec openbox-session\n' | tee --append /home/pi/.xinitrc;"
$run "echo 0 | sudo update-alternatives --config x-window-manager"
$run "sudo update-alternatives --config x-session-manager" # should say there is only one alternative
$run "sudo cp /etc/wpa_supplicant/ifupdown.sh /etc/ifplugd/action.d/ifupdown"
###
# MQTT
###
$run "sudo apt-get remove mosquitto -y && cd && wget https://github.com/warmcat/libwebsockets/archive/v2.4.1.zip && unzip v2.4.1.zip && cd libwebsockets-2.4.1/ && mkdir build && cd build && cmake .. && sudo make install && sudo ldconfig"
$run "cd && wget http://mosquitto.org/files/source/mosquitto-1.6.9.tar.gz && tar zxvf mosquitto-1.6.9.tar.gz && cd mosquitto-1.6.9 && sed -i -e "s/WITH_WEBSOCKETS:=no/WITH_WEBSOCKETS:=yes/g" config.mk && make && sudo make install && sudo cp mosquitto.conf /etc/mosquitto"
$run "printf 'user pi\nport 1883\nlistener 9001\nprotocol websockets\npid_file /var/run/mosquitto.pi\n' | sudo tee --append /etc/mosquitto/mosquitto.conf"
$run "sudo useradd mosquitto"
$run "sudo systemctl stop mosquitto && sudo update-rc.d mosquitto remove && sudo rm /etc/init.d/mosquitto"
$run "printf '[Unit]\nDescription=MQTT v3.1 message broker\nAfter=network.target\nRequires=network.target\n\n[Service]\nType=simple\nExecStart=/usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf\nRestart=always\nUser=pi\n\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/mosquitto.service"
$run "sudo systemctl daemon-reload && sudo systemctl unmask mosquitto && sudo systemctl enable mosquitto && sudo systemctl start mosquitto.service && sudo userdel mosquitto && sudo rm -rf /home/pi/.cmake /home/pi/libwebsockets-2.4.1 /home/pi/mosquitto-1.6.9 /home/pi/mosquitto-1.6.9.tar.gz /home/pi/v2.4.1.zip"
###
# LCD Start
###
#!/usr/bin/expect -f
# This will generate SSH keys and push the public key to the target
# This script expects the following environment variables to be set:
# host = host to push SSH keys to
# port = port SSH should use when connecting to the remote host
# username = username for remote host
# password = password for remote host
set timeout -1
spawn ssh-keygen -t ed25519 -N "" -f ./id_25519
#Start the guest VM
spawn $env(QEMU) -kernel $env(RPI_KERNEL) \
-cpu $env(RPI_CPU) -m 256 -M versatilepb \
-dtb $env(PTB_FILE) -no-reboot \
-device virtio-rng-pci -nographic \
-append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
-drive "file=$env(RPI_FS),index=0,media=disk,format=raw" \
-net user,hostfwd=tcp::5022-:22 -net nic
#Wait for the VM to be booted
expect "login: "
# We automatically accept the host key fingerprint
spawn ssh -o "StrictHostKeyChecking no" -p $env(port) $env(username)@$env(host)
#match_max 100000
expect "*?assword:*"
send "$env(password)\r\r"
expect " $"
send "mkdir -p .ssh\r"
expect " $"
send "chmod 700 .ssh\r"
expect " $"
send "echo '"
send [exec cat ./id_25519.pub]
send "' > .ssh/authorized_keys\r"
expect " $"
send "chmod 600 .ssh/authorized_keys\r"
expect " $"
#send "ls -la\r"
#expect " $"
#send "ls -la .ssh\r"
#expect " $"
#send "exit\r"
# Shutdown gracefully to make sure the changes to the filesystem get flushed
send "sudo shutdown -h now\r"
# Now we need to wait until it's done shutting down or it'll get killed
expect "Reached target Power-Off"
exit 0
#!/bin/sh
# This script will pick up where you left off after shutting down your Rasbian
# VM.
if [ "$#" -ne "2" ]; then
echo "Usage: $0 CPU BASE_VERSION"
echo ""
echo " CPU - the CPU to emulate"
echo " BASE_VERSION - the version of raspbian to put on the emulated pi"
echo ""
exit 1
fi
export RPI_CPU="$1"
RPI_OS_VERSION="$2"
export QEMU=$(which qemu-system-arm)
export TMP_DIR=./qemu-rpi
. ./${RPI_OS_VERSION}.sh
cd $TMP_DIR
export RPI_FS=`ls -t *-raspbian-${RPI_OS_VERSION}-lite.img | head -n 1`
$QEMU -kernel $RPI_KERNEL \
-cpu $RPI_CPU -m 256 -M versatilepb \
-dtb $PTB_FILE -no-reboot \
-device virtio-rng-pci -nographic \
-append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
-drive "file=$RPI_FS,index=0,media=disk,format=raw" \
-net user,hostfwd=tcp::5022-:22 -net nic
#!/bin/sh
# I set environment variables so build_emulator.sh can use a stretch image.
export RPI_KERNEL=kernel-qemu-4.14.79-stretch
export RPI_KERNEL_URL=https://github.com/dhruvvyas90/qemu-rpi-kernel/blob/master/kernel-qemu-4.14.79-stretch?raw=true
export PTB_FILE=versatile-pb.dtb
export PTB_FILE_URL=https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/${PTB_FILE}
export IMAGE_FILE=2018-11-13-raspbian-stretch-lite.zip
export IMAGE_FILE_URL=http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-11-15/${IMAGE_FILE}
export SDA2_START=98304
export username="pi"
export password="raspberry"
#!/usr/bin/expect -f
# Thanks https://stackoverflow.com/questions/3146013/qemu-guest-automation !
# This script expects the following environment variables to be set:
# QEMU - Name of qemu executable for ARM system emulation
# RPI_CPU - CPU of Raspberry Pi being emulated
# RPI_KERNEL - Filename of kernel to boot
# RPI_FS - Filename of disk image (filesystem) to boot
# PTB_FILE - Filename of PTB file (device tree binary)
# username - User to login as (e.g. "pi")
# password - User's password (e.g. "raspberry")
#starts guest vm, run benchmarks, poweroff
set timeout -1
#Assign a variable to the log file
set log [lindex $argv 0]
#Enable hella logging
exp_internal -f /tmp/expect.log 0
#Start the guest VM
spawn $env(QEMU) -kernel $env(RPI_KERNEL) \
-cpu $env(RPI_CPU) -m 256 -M versatilepb \
-dtb $env(PTB_FILE) \
-device virtio-rng-pci -nographic \
-append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
-drive "file=$env(RPI_FS),index=0,media=disk,format=raw" \
-net user,hostfwd=tcp::5022-:22 -net nic
#Login process
expect "login: "
#Enter username
send "$env(username)\r"
#Enter Password
expect "Password: "
send "$env(password)\r"
#Do whatever you want to do with in the guest VM. ( Run a process and write result to log )
send "sudo systemctl enable ssh\r"
expect "$ "
send "sudo systemctl start ssh\r"
expect "$ "
# Lets see wht the disk looks like
send "sudo fdisk -l /dev/sda\r"
expect "$ "
# Expand the second partition to be as large as possible
send "sudo fdisk /dev/sda\r"
expect ": "
send "d\r"
# delete partition
expect ": "
send "2\r"
# partition #2
expect ": "
send "n\r"
# new partition
expect ": "
send "p\r"
# primary partition
expect ": "
send "2\r"
# partition #2
expect ": "
send "$env(SDA2_START)\r"
expect ": "
send "\r"
# default end location
expect ": "
send "n\r"
# Do not overwrite the ext4 signature
expect ": "
send "w\r"
# Write changes
expect "$ "
# Then resize the partition
send "sudo resize2fs /dev/sda2\r"
expect "$ "
send "sudo shutdown -h now\r"
#send "exit\r"
#expect "login: "
# Execute the scripts to do the setup for your project
# ...or...
# Exit now so the calling script knows ssh is up
# Instead of shutting it down, drop to an interactive shell
#interact
#poweroff the Guest VM
#expect "# "
#send "shutdown -h now\r"
# Make sure qemu lives on after we perish
exit 0
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment