Skip to content
Snippets Groups Projects
Commit 8f6ffe56 authored by Scott Cosentino's avatar Scott Cosentino Committed by Ian Ernst
Browse files

Updates to system administration labs

parent bbbcebcf
No related branches found
No related tags found
Loading
Showing with 233 additions and 205 deletions
......@@ -36,9 +36,11 @@ Before you get started, complete the following:
1. Use your assigned IP address and SSH key file to log in to the server that will host your GitLab Omnibus install:
```bash
ssh -i <keyfile_name> ec2-user@<vm_ip_address>
ssh -i <keyfile_name> root@<vm_ip_address>
```
>If you encounter an error like: `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!`, you may need to reset your SSH known hosts. To do this, run the command `ssh-keygen -R <vm_ip_address>`
1. Press <kbd>Enter</kbd>.
1. If your system displays an authentication warning, type `yes` and press <kbd>Enter</kbd>.
......@@ -50,9 +52,13 @@ Before you get started, complete the following:
1. Install Postfix so GitLab can send notification emails by using the command below:
```bash
sudo dnf install -y curl policycoreutils perl postfix
sudo apt-get install -y curl policycoreutils perl postfix
```
1. If you receive a Postfix Configuration Dialog when running this command, first select `Ok` by pressing the right arrow key to highlight the choice, and then pressing **Enter**. On the next page, select `No configuration` using the up and down arrow keys, then press <kbd>Enter</kbd>.
> For a production system, you can use this configuration to be able to configure your email settings with postfix. This will allow you to send emails from GitLab.
1. Start and enable Postfix using the `systemctl enable` and the `systemctl start` commands:
```bash
......@@ -65,13 +71,13 @@ Before you get started, complete the following:
1. Add the GitLab install repository via the `curl` command:
```bash
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | sudo bash
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
```
1. Install the GitLab package using the command below. Use your training system's assigned public IP address in lieu of a fully qualified domain name. Make sure not to include the <> symbols.
```bash
sudo EXTERNAL_URL="http://<your_assigned_public_ip>" dnf install -y gitlab-ee
sudo EXTERNAL_URL="http://<your_assigned_public_ip>" apt-get install -y gitlab-ee
```
> This step may take a few minutes to complete.
......@@ -99,7 +105,7 @@ Before you get started, complete the following:
8. Complete the remaining fields with a new, permanent password of your choosing.
9. Click **Save password** to save the changes. You will need to re-sign in with your new password.
9. Click **Save password** to save the changes. You will be logged out, and will need to sign in again with your new password.
## Lab Guide Complete
......
......@@ -85,6 +85,8 @@ The objective of this lab is to show various gitlab-ctl commands you can use to
1. Change the `Default group visibility` to **Internal** by clicking on the associated radio button.
1. Select **Save changes** at the bottom of the **Visibility and access controls** section to apply these changes.
### Task C. Locate sign-in settings
1. The second setting we will update are Sign-In restrictions. Still under **Settings** > **General**, click **Collapse** next to **Visibility and access controls**.
......@@ -93,8 +95,26 @@ The objective of this lab is to show various gitlab-ctl commands you can use to
1. Under **Two-factor authentication**, click the checkbox next to **Enforce two-factor authentication**.
1. Select **Save changes** at the end of the **Sign-in restrictions** section to apply this change.
1. Click **Collapse** next to **Sign-in restrictions** to close the menu.
1. After applying this change, you will be redirected to a page to setup 2FA on your administrator account. You can either enabled 2FA on your account, or disable the 2FA setting to avoid this notification.
To disable the 2FA setting:
1. Select **Configure it later** on the 2FA page.
1. Select **Admin Area** in the left sidebar.
1. Select **Settings** > **General**.
1. Select **Expand** next to **Sign-in restrictions**.
1. Uncheck **Enforce two-factor authentication**.
1. Select **Save Changes** at the end of the **Sign-in restrictions** section
### Task D. Update the header logo
You can personalize your GitLab instance by uploading a header logo.
......
......@@ -14,7 +14,7 @@ The objective of this lab is to demonstrate how to create and register a GitLab
1. Use your assigned IP address and SSH key file to log into your **GitLab Runner** server (*not* your Omnibus server).
```bash
ssh -i <YOUR_ASSIGNED_SSH_KEYFILE> ec2-user@<YOUR_RUNNER_SERVER_PUBLIC_IP>
ssh -i <YOUR_ASSIGNED_SSH_KEYFILE> root@<YOUR_RUNNER_SERVER_PUBLIC_IP>
```
1. If your system displays an authentication warning, type `yes` and press <kbd>Enter</kbd>
......@@ -22,24 +22,24 @@ The objective of this lab is to demonstrate how to create and register a GitLab
1. Add the GitLab Runner installation repository.
```bash
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
```
1. Install the GitLab Runner agent.
```bash
sudo dnf install -y gitlab-runner
```
```bash
sudo apt-get install -y gitlab-runner
```
1. Once the install completes, check that the service is running.
```bash
sudo gitlab-runner status
```
```bash
sudo gitlab-runner status
```
### Task B. Register a runner with GitLab
1. Sign into your GitLab instance with a web browser and select **Menu > Admin Area**.
1. Sign into your GitLab instance with a web browser and select **Admin Area** from the left sidebar.
1. In the left sidebar, under **CI/CD**, select **Runners**.
......@@ -51,6 +51,8 @@ The objective of this lab is to demonstrate how to create and register a GitLab
1. Leave all other options as default and select **Create runner**.
1. Ensure that **Linux** is selected as the **Operating system** for the runner.
1. Copy the command in **Step 1**, and add `sudo` in front of the command. Run it in your command prompt.
1. The command will first prompt you for your GitLab instance URL. Verify that this URL matches your GitLab instance, then press <kbd>Enter</kbd>
......@@ -63,7 +65,7 @@ The objective of this lab is to demonstrate how to create and register a GitLab
1. Run `sudo gitlab-runner list` to verify the runner after registration.
1. Return to your web browser and select **Go to runners page**. Verify the runner you registered appears in the list and shows as online.
1. Return to your web browser and select **View runners**. Verify the runner you registered appears in the list and shows as online.
### Task C. Test the runner with a CI/CD pipeline
......@@ -85,21 +87,21 @@ The objective of this lab is to demonstrate how to create and register a GitLab
1. Paste the following code into the body of the file.
```yml
stages:
- build
- test
build_app:
stage: build
script:
- echo "The build stage requires at least one job"
test_app:
stage: test
script:
- echo "The test stage requires at least one job"
```
```yml
stages:
- build
- test
build_app:
stage: build
script:
- echo "The build stage requires at least one job"
test_app:
stage: test
script:
- echo "The test stage requires at least one job"
```
1. Select **Commit changes**.
......
......@@ -46,16 +46,16 @@ The objective of this lab is to demonstrate how to back up a Gitlab instance on
1. Take a full backup of the GitLab instance.
```bash
sudo gitlab-backup create
```
```bash
sudo gitlab-backup create
```
1. After the backup completes, go to the backup location and inspect the backup file.
```bash
sudo ls /tmp/backups
sudo tar -tvf /tmp/backups/<backup_filename>
```
```bash
sudo ls /tmp/backups
sudo tar -tvf /tmp/backups/<backup_filename>
```
### Task C. Make some changes to GitLab settings
......@@ -75,29 +75,29 @@ sudo tar -tvf /tmp/backups/<backup_filename>
1. Move your backup file to the location GitLab requires for performing the restore.
```bash
sudo cp /tmp/backups/<backup_filename> /var/opt/gitlab/backups/
```
```bash
sudo cp /tmp/backups/<backup_filename> /var/opt/gitlab/backups/
```
1. Ensure the backup file has correct permissions for performing the restore.
```bash
sudo chown git:git /var/opt/gitlab/backups/<backup_filename>
```
```bash
sudo chown git:git /var/opt/gitlab/backups/<backup_filename>
```
1. Stop the puma and sidekiq services before restoring.
```bash
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status
```
```bash
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status
```
1. Restore from backup. Replace *<backup_timestamp>* with the portion of the backup filename up to and including `-ee`. For example, if the backup file name starts with `1663207732_2022_09_15_15.3.3-ee`, the command will be `sudo gitlab-backup restore BACKUP=1663207732_2022_09_15_15.3.3-ee`.
```bash
sudo gitlab-backup restore BACKUP=<backup_timestamp>
```
```bash
sudo gitlab-backup restore BACKUP=<backup_timestamp>
```
1. Type `yes` when prompted during the restore operation. You may see what looks like error messages. That is normal.
......@@ -105,11 +105,11 @@ sudo gitlab-backup restore BACKUP=<backup_timestamp>
1. Restart sidekiq and puma services.
```bash
sudo gitlab-ctl start sidekiq
sudo gitlab-ctl start puma
sudo gitlab-ctl status
```
```bash
sudo gitlab-ctl start sidekiq
sudo gitlab-ctl start puma
sudo gitlab-ctl status
```
1. Wait up to 5 minutes before refreshing GitLab in your web browser. Verify that the maximum attachment size and the default project limits you changed revert back to the defaults (i.e. when the backup was taken).
......
......@@ -15,25 +15,25 @@ The `gitlab-ctl` command allows you to tail all GitLab log files as well as filt
1. From a shell session on your GitLab instance, run the following command to view all active GitLab logs.
```bash
sudo gitlab-ctl tail
```
```bash
sudo gitlab-ctl tail
```
Amidst all the output, you should notice the command shows the full file path to each log. Most GitLab logs live in `/var/log/gitlab`. (Note: You can type `CTRL-C` to exit the `tail` command.)
Amidst all the output, you should notice the command shows the full file path to each log. Most GitLab logs live in `/var/log/gitlab`. (Note: You can type `CTRL-C` to exit the `tail` command.)
1. You can also view GitLab logs by service. Run the following command to view only NGINX logs (i.e. log files in `/var/log/gitlab/nginx`).
```bash
sudo gitlab-ctl tail nginx
```
```bash
sudo gitlab-ctl tail nginx
```
You should now see the most recent entries of log files specific to the NGINX web server.
You should now see the most recent entries of log files specific to the NGINX web server.
1. Finally, you can drill down to an individual log file.
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
### Task B. Set minimum log levels
......@@ -41,26 +41,26 @@ Admins are able to set minimum log levels for some GitLab services. Note that on
1. Check the current minimum log levels for GitLab services.
```bash
sudo grep -n -E 'log_level|logging_level' /etc/gitlab/gitlab.rb
```
```bash
sudo grep -n -E 'log_level|logging_level' /etc/gitlab/gitlab.rb
```
1. Note the line number for `nginx['error_log_level']`.
1. Change the minimum log level for `nginx`. Replace "1731" with the appropriate line number from the `grep` output in the previous step.
```bash
sudo sed -i '1731s/\"error\"/\"warn\"/' /etc/gitlab/gitlab.rb
sudo sed -i '1731s/# //' /etc/gitlab/gitlab.rb
```
```bash
sudo sed -i '1731s/\"error\"/\"warn\"/' /etc/gitlab/gitlab.rb
sudo sed -i '1731s/# //' /etc/gitlab/gitlab.rb
```
1. Re-run the `grep` command from Step 1 to verify the line was modified as intended.
1. Reconfigure to apply the changes.
```bash
sudo gitlab-ctl reconfigure
```
```bash
sudo gitlab-ctl reconfigure
```
### Task C. Manage log retention
......@@ -68,49 +68,49 @@ GitLab uses **logrotate** to manage retention of all logs except those managed b
1. Examine default logrotate retention settings.
```bash
sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb
```
```bash
sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb
```
1. **Optional**: View the default retention settings for the runit-managed logs.
```bash
sudo grep -n 'svlogd' /etc/gitlab/gitlab.rb
```
```bash
sudo grep -n 'svlogd' /etc/gitlab/gitlab.rb
```
1. It appears logrotate (and svlogd) rotate log files every day, and retain 30 days worth of logs. We can verify this by looking inside the service log directories.
```bash
sudo ls /var/log/gitlab/puma
```
```bash
sudo ls /var/log/gitlab/puma
```
Note the gzipped archive files for Puma's stdout and stderr logs from previous days.
Note the gzipped archive files for Puma's stdout and stderr logs from previous days.
1. Change logrotate's behavior to rotate log files weekly. As before, modify the line `sed` edits accordingly using the line number from the grep output.
```bash
sudo sed -i '1234s/daily/weekly/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb
```
```bash
sudo sed -i '1234s/daily/weekly/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb
```
1. Change logrotate's retention period to 1 year of retained log files. As before, modify the line `sed` edits accordingly using the line number from the grep output.
```bash
sudo sed -i '1234s/30/52/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb
```
```bash
sudo sed -i '1234s/30/52/g' /etc/gitlab/gitlab.rb
sudo sed -i '1234s/# //' /etc/gitlab/gitlab.rb
```
1. Run the following again to ensure your changes are properly written to `gitlab.rb`.
```bash
sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb
```
```bash
sudo grep -n 'logrotate' /etc/gitlab/gitlab.rb
```
1. Reconfigure to apply the changes.
```bash
sudo gitlab-ctl reconfigure
```
```bash
sudo gitlab-ctl reconfigure
```
### Task D. Change log formatting
......
......@@ -14,7 +14,7 @@ The purpose of this lab is to walk you through using Prometheus Service, and how
1. GitLab's Prometheus server can be reached via TCP port 9090. Unfortunately, the training environment currently blocks inbound traffic to that port. As a workaround, you can open an SSH tunnel as follows.
```bash
ssh -L 9090:localhost:9090 -i <SSH_KEY_NAME>.pem ec2-user@<GITLAB_INSTANCE_HOSTNAME>
ssh -L 9090:localhost:9090 -i <SSH_KEY_NAME> root@<GITLAB_INSTANCE_HOSTNAME>
```
1. Navigate to `http://localhost:9090` in a web browser to view the built-in Prometheus server.
......
......@@ -13,52 +13,52 @@ The purpose of this lab is to show how to troubleshoot the GitLab server by usin
1. From a shell session on your GitLab instance, view one of the NGINX active logs.
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
Note the log adds new entries every few seconds. Most of these entries are gitlab-runner checking in with the GitLab instance via HTTP.
Note the log adds new entries every few seconds. Most of these entries are gitlab-runner checking in with the GitLab instance via HTTP.
1. Stop the NGINX services.
```bash
sudo gitlab-ctl stop nginx
```
```bash
sudo gitlab-ctl stop nginx
```
1. Attempt to navigate to `http://<your_gitlab_instance>` using a web browser. Your web browser should display "**This site can't be reached**" or a similar message.
1. Check `nginx/access_log` again.
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
The log should no longer be updating since no clients can make HTTP/HTTPS requests to GitLab after stopping NGINX.
1. Verify web services aren't running or listening anywhere.
```bash
curl -i http://localhost/nginx_status
curl -i http://localhost:80
```
```bash
curl -i http://localhost/nginx_status
curl -i http://localhost:80
```
1. Restart NGINX services.
```bash
sudo gitlab-ctl restart nginx
```
```bash
sudo gitlab-ctl restart nginx
```
1. Verify the clients (e.g. the GitLab Runner) can communicate with GitLab again.
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
```bash
sudo gitlab-ctl tail nginx/gitlab_access.log
```
1. Verify the webserver is running and listening on port 80.
```bash
curl -i http://localhost/nginx_status
```
```bash
curl -i http://localhost/nginx_status
```
### Task B. Troubleshoot Puma
......@@ -66,59 +66,59 @@ curl -i http://localhost/nginx_status
1. From a shell session on the GitLab instance, stop Puma.
```bash
sudo gitlab-ctl stop puma
```
```bash
sudo gitlab-ctl stop puma
```
1. Refresh GitLab in your web browser. You should immediately see an error that reads "**502: GitLab is taking too much time to respond**". NGINX is running, so it can accept HTTP requests. However, when workhorse tries to pass an HTTP request to the Rails application, there is no running service to accept it.
1. View the GitLab Workhorse logs.
```bash
sudo gitlab-ctl tail gitlab-workhorse/current
```
```bash
sudo gitlab-ctl tail gitlab-workhorse/current
```
You will see a variety of **502** and **badgateway** errors in the output.
You will see a variety of **502** and **badgateway** errors in the output.
1. View Puma logs.
```bash
sudo gitlab-ctl tail puma
```
```bash
sudo gitlab-ctl tail puma
```
You should see a message in `puma/puma_stdout.log` about the Puma service shutting down. You may also see errors in `puma/puma_stderr.log`.
You should see a message in `puma/puma_stdout.log` about the Puma service shutting down. You may also see errors in `puma/puma_stderr.log`.
1. Restart Puma.
```bash
sudo gitlab-ctl restart puma
```
```bash
sudo gitlab-ctl restart puma
```
1. View Puma's runit log.
```bash
sudo gitlab-ctl tail puma/current
```
```bash
sudo gitlab-ctl tail puma/current
```
You may see output indicating Puma has restarted.
You may see output indicating Puma has restarted.
1. View `puma/puma_stdout.log`.
```bash
sudo gitlab-ctl tail puma/puma_stdout.log
```
```bash
sudo gitlab-ctl tail puma/puma_stdout.log
```
You should see that Puma is running and consuming resources again.
You should see that Puma is running and consuming resources again.
1. Wait about 2 minutes, then refresh GitLab in your web browser. The application should now be reachable.
1. View the GitLab Workhorse log.
```bash
sudo gitlab-ctl tail gitlab-workhorse/current
```
```bash
sudo gitlab-ctl tail gitlab-workhorse/current
```
Recent entries should indicate successful requests to Puma (i.e. when you reloaded GitLab in your web browser).
Recent entries should indicate successful requests to Puma (i.e. when you reloaded GitLab in your web browser).
### Task C. Troubleshoot Gitaly
......@@ -136,15 +136,15 @@ Recent entries should indicate successful requests to Puma (i.e. when you reload
1. SSH into your **GitLab Runner server**.
```bash
ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_runner_host>
```
```bash
ssh -i <SSH_HOST_KEY> root@<GITLAB_runner_host>
```
1. Download Git if it is not already installed.
```bash
sudo dnf install -y git
```
```bash
sudo apt-get install -y git
```
1. Back on GitLab's **Test project**, select **Clone** on the right side of the page.
......@@ -152,51 +152,51 @@ sudo dnf install -y git
1. From your GitLab Runner server, clone the repository.
```bash
git clone <URL_copied_from_previous_step>
```
```bash
git clone <URL_copied_from_previous_step>
```
1. Verify the project is correctly cloned.
```bash
cd ~/test-project
ls -a
git status
```
```bash
cd ~/test-project
ls -a
git status
```
1. Enter `exit` to exit the SSH session on your GitLab Runner server.
1. Open an SSH session on your **GitLab Omnibus instance**.
```bash
ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_OMNIBUS_HOST>
```
```bash
ssh -i <SSH_HOST_KEY> root@<GITLAB_OMNIBUS_HOST>
```
1. Verify Gitaly is running.
```bash
sudo gitlab-ctl status gitaly
```
```bash
sudo gitlab-ctl status gitaly
```
1. View Gitaly logs.
```bash
sudo gitlab-ctl tail gitaly
```
```bash
sudo gitlab-ctl tail gitaly
```
> You should see many recent gRPC requests relating to **Test Project** (you can see the references more clearly if you grep the output, e.g. `sudo gitlab-ctl tail gitaly | grep test-project`).
> You should see many recent gRPC requests relating to **Test Project** (you can see the references more clearly if you grep the output, e.g. `sudo gitlab-ctl tail gitaly | grep test-project`).
1. Stop Gitaly services.
```bash
sudo gitlab-ctl stop gitaly
```
```bash
sudo gitlab-ctl stop gitaly
```
1. Verify Gitaly (and only Gitaly) is stopped.
```bash
sudo gitlab-ctl status
```
```bash
sudo gitlab-ctl status
```
1. Navigate back to **Test Project** in your web browser. On the project page select the dropdown that says **main** under the project title. Ordinarily you would be able to select a Git branch to switch to. Now you see an error and the branch list will not load.
......@@ -208,55 +208,55 @@ sudo gitlab-ctl status
1. Return to your GitLab instance SSH session. Check Gitaly's recent log entries.
```bash
sudo gitlab-ctl tail gitaly/current
```
```bash
sudo gitlab-ctl tail gitaly/current
```
> Note the many errors in the log output.
> Note the many errors in the log output.
1. Enter `exit` to exit the SSH session on your GitLab Instance server.
1. SSH back into your **GitLab Runner server**.
```bash
ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_RUNNER_HOST>
```
```bash
ssh -i <SSH_HOST_KEY> root@<GITLAB_RUNNER_HOST>
```
1. Navigate into your cloned **Test Project**.
```bash
cd ~/test-project
```
```bash
cd ~/test-project
```
1. Try to fetch any new changes from the remote repo on the GitLab instance.
```bash
git fetch
```
```bash
git fetch
```
> You may see error 503 in the output, indicating Gitaly is not reachable and can not handle the request.
> You may see error 503 in the output, indicating Gitaly is not reachable and can not handle the request.
1. Enter `exit` to exit the SSH session on your GitLab Runner server.
1. Re-initiate an SSH session on your **GitLab Omnibus instance**.
```bash
ssh -i <SSH_HOST_KEY>.pem ec2-user@<GITLAB_OMNIBUS_HOST>
```
```bash
ssh -i <SSH_HOST_KEY> root@<GITLAB_OMNIBUS_HOST>
```
1. Restart Gitaly services.
```bash
sudo gitlab-ctl start gitaly
```
```bash
sudo gitlab-ctl start gitaly
```
1. Check Gitaly logs.
```bash
sudo gitlab-ctl tail gitaly/current
```
```bash
sudo gitlab-ctl tail gitaly/current
```
> The output should now show successful gRPC requests.
> The output should now show successful gRPC requests.
1. Return to **Test Project** in your web browser. Refresh the page. You should now be able to navigate around the repository, view files, and check out branches.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment