DAP Incorporation for CICD Labs

- **Cryptic Error Messages**: Jobs fail with unclear errors like "error in libcrypto" that provide no obvious solution

- **Trial and Error Debugging**: Developers waste hours trying random fixes without a systematic troubleshooting approach

- **Hidden Configuration Issues**: SSH keys, environment variables, and formatting problems cause silent failures

- **No Troubleshooting Framework**: The team lacks a structured methodology for diagnosing and resolving pipeline failures

- **Fear of Deployments**: Without confidence in their ability to debug issues, developers avoid setting up deployment automation

As a part of this course, you were provided with an SSH key to use for deployments. You will need to add this SSH key to GitLab to use it during your CI/CD process.

### Security Recommendation: Use 1Password for SSH Key Management

We strongly recommend using 1Password or similar secure key management solutions to generate and store SSH keys instead of storing them on disk. This provides better protection for your private keys. Follow the [1Password SSH key generation guide](https://docs.gitlab.com/user/ssh_advanced/#generate-an-ssh-key-pair-with-1password) for detailed instructions.

> **Security Warning:** Storing SSH keys on disk poses security risks. Private keys should be protected and never shared. Use secure key management practices.

To add your SSH key to GitLab, follow these steps:

1. Navigate to your project in GitLab.

1. In the left sidebar, go to **Settings > CI/CD**.

1. Select **Expand** next to **Variables**.

1. In **Group variables (inherited)**, you will see a variable named `SSH_PRIVATE_KEY`

      To demonstrate a common SSH related error, we will copy the SSH key and create a new variable based on its value:

1. You are currently viewing the inherited variables at project level. To interact directly with these group-level variables, click on the group name next to the `SSH_PRIVATE_KEY` variable.

1. Select **Expand** next to **Variables**.

1. Select the **Copy** icon next to the **value** of the `SSH_PRIVATE_KEY` variable .

1. Select **Add variable**.

1. Set the **Type** to **file**.

1. In the `key` field, enter `SSH_INVALID_KEY`.

1. Set the variable Visibility to **Visible**.

1. In the value, paste your SSH key.

1. Delete the new line at the end of the key value. Doing this will create an error when we try to use the key.

1. Select **Add variable**.

      Your new SSH key variable will now be accessible during any CI/CD jobs you run in your group. Now, let’s create a job to test the SSH connection.

1. Navigate to your CI/CD project.

1. Select **Build > Pipeline Editor**.

        - eval $(ssh-agent -s)

    ```

1. Next, we will set up a pem file from our SSH key variable, setting the required permissions on the file so it can be used for SSH purposes. Note that we are using the `SSH_INVALID_KEY` variable.

1. Next, we will set up a pem file from our SSH key variable, setting the required permissions on the file so it can be used for SSH purposes. Note that we are using the `SSH_PRIVATE_KEY` variable, which is a group-level variable. You can see the full listing of our variables by navigating to your group, and selecting **Settings > CI/CD** and then selecting the **Variables** sub-menu.

    ```yaml

    deploy app:

      script: 

        - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'

        - eval $(ssh-agent -s)

        - chmod 400 "$SSH_INVALID_KEY"

        - ssh-add "$SSH_INVALID_KEY"

        - chmod 400 "$SSH_PRIVATE_KEY"

        - ssh-add "$SSH_PRIVATE_KEY"

        - mkdir -p ~/.ssh

        - chmod 700 ~/.ssh

    ```

        script: 

          - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'

          - eval $(ssh-agent -s)

          - chmod 400 "$SSH_INVALID_KEY"

          - ssh-add "$SSH_INVALID_KEY"

          - chmod 400 "$SSH_PRIVATE_KEY"

          - ssh-add "$SSH_PRIVATE_KEY"

          - mkdir -p ~/.ssh

          - chmod 700 ~/.ssh

          - ssh-keyscan -t rsa,ed25519 $ip >> ~/.ssh/known_hosts

          - ssh-keyscan -t rsa,ed2519 $ip >> ~/.ssh/known_hosts

          - ssh root@$ip 'ls /'

      ```

        script: 

          - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'

          - eval $(ssh-agent -s)

          - chmod 400 "$SSH_INVALID_KEY"

          - ssh-add "$SSH_INVALID_KEY"

          - chmod 400 "$SSH_PRIVATE_KEY"

          - ssh-add "$SSH_PRIVATE_KEY"

          - mkdir -p ~/.ssh

          - chmod 700 ~/.ssh

          - ssh-keyscan -t rsa,ed25519 $ip >> ~/.ssh/known_hosts

          - ssh-keyscan -t rsa,ed2519 $ip >> ~/.ssh/known_hosts

          - ssh root@$ip 'ls /'

        environment:

          name: prod

          url: "http://$ip:80"

      ```

When you commit these changes, you will see an error in your deploy job. To view this error, navigate to the **Build** > **Pipelines** page and view the failed pipeline and job. The output should look similar to the one below:

  ```bash

  $ eval $(ssh-agent -s)

  Agent pid 3211

  $ chmod 400 "$SSH_INVALID_KEY"

  $ ssh-add "$SSH_INVALID_KEY"

  Error loading key "/builds/training-users/session-eff7bd34/iuztj7px/cicd-demo.tmp/SSH_INVALID_KEY": error in libcrypto

  ```

Let’s try to figure out what happened!

## Task B.1. Isolate the command that causes the error

The first logical step is to isolate the command that is causing the error. We can see in the logs that the `ssh-add "$SSH_INVALID_KEY"` command looks to cause the error.

With the command isolated, we can consider some ways to verify the command. One option would be to try running the command locally if possible. This can help rule out potential issues with the runner. For this example, we can assume the runners are working correctly, meaning the issue lies in the actual command itself.

In these cases, often the variable/input of the command is the main source of the error. From the error message, it looks that the key is not formatted correctly. Let's consult the documentation to see why.

## Task B.2. Search the Documentation

Often, common errors will be present in our documentation with solutions to the problems. To find this error:

1. Try searching for the error in the [documentation](https://docs.gitlab.com/). The first result you get is an article titled: **Using SSH keys with GitLab CI/CD**. Click onto this page.

1. If you scroll to the Troubleshooting section of this page, you will see a section on the exact error we are facing.

1. Commit your changes, and let the pipeline run. You should see that the deploy app job has failed.

      The documentation tells us that the issue is not having a new line at the end of the key. Let’s try adding one and see if it fixes the problem.

Let's figure out what went wrong, and how to fix it!

1. Return to your variables by selecting **Settings > CI/CD > Variables**.

## Task B. Debugging the pipeline with Duo Agent Platform

1. Select the group next to the `SSH_INVALID_KEY` variable.

1. Click the `deploy app` job. You should see an error message similar to `Unknown key type: "ed2519"` in the job log. We are going ot use Duo Agent Platform (DAP) to help us resolve this vulnerability. Duo Agent Platform is GitLab's AI solution that can help you build issues, review merge requests, remediate vulnerabilities, and in this case, fix a broken pipeline.

1. Expand the group variable section and select the **Edit** icon next to the `SSH_INVALID_KEY` variable.

1. Click on the **New chat** button in the top right.

1. Add a new line to the end of the variable value, then select **Save Changes**.

1. In the `Choose an agent` menu, select **CI Expert**. 

      To test if this fixes the error:

1. Write in the chat box "Can you help me fix this broken pipeline?" and press Enter. The CI Exeprt agent will analze your job log, and recommend a solution.

1. Navigate back to your CI/CD project.

1. Write in the chat box "Can you create a merge request to fix the broken pipeline?" and press Enter. The CI Expert agent will work to create a merge request. When the agent asks you for permission to create a commit and a merge request, press **Approve**.

1. Select **Build > Pipelines** from the left sidebar.

> When it comes to any actions that make changes to your repo, GitLab Duo agents will always ask for permission to do so. This is to ensure you remain the Human In the Loop (HITL) when it comes to using DAP.

1. Select **New pipeline**.

1. Click on the Merge Request link provided by the chat, or access the Merge Request via **Code > Merge Requests**.

1. Leave all values as default and select **New pipeline** again. You will now see the job complete successfully!

1. Review the Merge Request to see what changes were made, and click on the **Merge** button to merge the code.

## Task C. Clean Up Deploy Job

## Postface

By working through a real-world SSH deployment error, you developed a systematic troubleshooting framework that transformed their team's confidence. Your team learned to isolate failing commands, search documentation for known issues, and verify fixes methodically rather than guessing randomly. The structured approach—identify the error, isolate the command, consult documentation, apply the fix, and verify—reduced average debugging time from hours to minutes. Additionally, by refactoring the deploy job to use `before_script` for setup tasks, your team improved code readability and made their pipelines easier to maintain and troubleshoot in the future. With these troubleshooting skills and a working SSH deployment configuration, Tanuki Enterprises now confidently automates deployments to production, knowing they can quickly diagnose and resolve any pipeline issues that arise.

By working through a real-world deployment error, you developed a systematic troubleshooting framework that transformed their team's confidence. Your team learned to isolate failing commands, utilize agentic systems, and verify fixes methodically rather than guessing randomly. The structured approach—identify the error, isolate the command, consult DAP, apply the fix, and verify—reduced average debugging time from hours to minutes. Additionally, by refactoring the deploy job to use `before_script` for setup tasks, your team improved code readability and made their pipelines easier to maintain and troubleshoot in the future. With these troubleshooting skills and a working deployment configuration, Tanuki Enterprises now confidently automates deployments to production, knowing they can quickly diagnose and resolve any pipeline issues that arise.

## Lab Guide Complete