UBI-FIPS Runner cannot create directory '/builds'

Summary

I implemented the logic to run tests against gitlab-runner:ubi-fips-v15.10.1 and ran the Verify Run pipeline with manual jobs does not leave any job in skipped state test (which is passing on gitlab-runner:alpine). But it has errors under ubi-fips-v15.10.1. Here is the pipeline it creates. In the prep job it's throwing mkdir: cannot create directory '/builds': Permission denied and in the build jobs it's throwing Password: su: Authentication failure.

Steps to reproduce

1. setup your local environment to be able to run tests

   Environment variables needed (The Environment Variable QA_RUNNER_CONTAINER_IMAGE determines which Runner image will be run against)

    export GITLAB_QA_USER_AGENT="<value from 1Password>"
    export GITLAB_QA_ACCESS_TOKEN="<value from 1Password>"
    export GITLAB_QA_DEV_ACCESS_TOKEN="<value from 1Password>"
    export GITLAB_USERNAME="gitlab-qa"
    export GITLAB_PASSWORD="<value from 1Password>"
    export GITLAB_ADMIN_USERNAME="<value from 1Password>"
    export GITLAB_ADMIN_PASSWORD="<value from 1Password>"
    export GITLAB_QA_USERNAME_1="gitlab-qa-user1"
    export GITLAB_QA_PASSWORD_1="<value from 1Password>"
    export QA_PRAEFECT_REPOSITORY_STORAGE="nfs-file22"
    export QA_RUNNER_CONTAINER_IMAGE="gitlab-runner:ubi-fips-v15.10.1"

2. cd into the qa folder inside your GitLab repo
3. bundle install to make sure you have all gems needed
4. run bundle exec bin/qa Test::Instance::All https://staging.gitlab.com -- qa/specs/features/browser_ui/4_verify/pipeline/run_pipeline_with_manual_jobs_spec.rb

Example Project

What is the current bug behavior?

The test fails with the message of, and the pipeline looks like this:

1) Verify Run pipeline with manual jobs does not leave any job in skipped state
     Failure/Error:
       show.wait_until(max_duration: 300, sleep_interval: 2, reload: false) do
         project.latest_pipeline[:status] == 'success'
       end

     QA::Support::Repeater::WaitExceededError:
       Wait failed after 300 seconds
     # ./qa/support/repeater.rb:74:in `repeat_until'
     # ./qa/support/waiter.rb:20:in `wait_until'
     # ./qa/page/base.rb:76:in `wait_until'
     # ./qa/specs/features/browser_ui/4_verify/pipeline/run_pipeline_with_manual_jobs_spec.rb:88:in `block (4 levels) in <module:QA>'
     # ./qa/scenario/actable.rb:16:in `perform'
     # ./qa/specs/features/browser_ui/4_verify/pipeline/run_pipeline_with_manual_jobs_spec.rb:85:in `block (3 levels) in <module:QA>'
     # ./qa/specs/runner.rb:67:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/template.rb:50:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/bootable.rb:52:in `launch!'

Top 1 slowest examples (359.91 seconds, 100.0% of total time):
  Verify Run pipeline with manual jobs does not leave any job in skipped state
    359.91 seconds ./qa/specs/features/browser_ui/4_verify/pipeline/run_pipeline_with_manual_jobs_spec.rb:84

The wait is waiting for the pipeline to show success, but it fails.

What is the expected correct behavior?

I'ld expect the test to pass like it does for gitlab-runner:alpine and generate a pipeline that looks like this. This was generated by running the same test after running unset QA_RUNNER_CONTAINER_IMAGE which uses the default image of gitlab-runner:alpine

Relevant logs and/or screenshots

Failing pipeline screenshot

Passing pipeline screenshot

Possible fixes