When running a script in CI, the system does not have the same available commands and packages as the calling system

I couldn't find any related issues. If there are any, please close this and redirect me appropriately.

Summary

When running a script in the CI runners python gitlab_deploy.py, the shell it is running in does not have the installed packages available (in this case, aws is not found, though it was pip installed) when pushing commands through os.system('...')

Steps to reproduce

this is our .gitlab-ci.yml setup for this task:

deploy_staging:
    image: python:2
    cache:
        paths:
            - .pip_cache
        key: "pip_aws"
    when: manual
    before_script:
        - mkdir -m 0700 -p .pip_cache
        - date
        - pip install --cache-dir .pip_cache --user awscli --upgrade
    script:
        - cd bin/
        - python gitlab_deploy.py

Within gitlab_deploy.py, we have a task that copies several files/ syncs several folders to our s3 bucket via this block:

        for file_name in MANUALLY_COPIED_FILES:
            os.system(
                'aws s3 cp air_lms/static/{file} s3://{bucket}/{file} --acl public-read'.format(
                    file=file_name,
                    bucket=self.bucket,
                ))
        for directory in MANUALLY_COPIED_DIRS:
            os.system(
                'aws s3 sync air_lms/static/{dir} s3://{bucket}/{dir} --acl public-read'.format(
                    dir=directory,
                    bucket=self.bucket,
                ))

this results in the following error (repeated)

$ python gitlab-deploy.py veelo_random
sh: 1: aws: not found
...

However, when I just run the aws commands in-line in the script block, for instance

    script:
        - cd bin/
        - python gitlab_deploy.py
        - cd src/static/
        - aws s3 cp ./discovery/build/assets/discovery-0.1.0.min.js s3://bucket/ --acl public-read
...

this works without issue.

What is the current bug behavior?

I get the above aws: not found error

What is the expected correct behavior?

the aws files should sync appropriately with the installed awscli package and keys setup in the project.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

This bug happens on gitlab.com

Possible fixes

Unknown