Commit ca94968d authored by Neil Spink's avatar Neil Spink

restructure and update documentation

parent fc796d4b
Pipeline #43507375 passed with stages
in 4 minutes and 1 second
......@@ -46,7 +46,7 @@ deploy:
script:
- pip3 install awscli
- aws s3 cp source/source.zip s3://$S3_BUCKET/
- aws s3 cp deployment/website-monitor-list.yml s3://$S3_BUCKET/
- aws s3 cp source/website-monitor-list.yml s3://$S3_BUCKET/
- aws cloudformation package --template-file ./deployment/aws-create-lambda.json --s3-bucket $S3_BUCKET --output-template template-export.yml
- aws cloudformation deploy --template-file template-export.yml --stack-name aws-lambda-price-grabber-stack --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset
- aws lambda update-function-code --function-name grab-price --s3-bucket $S3_BUCKET --s3-key source.zip
......
#AWS Lambda Price Grabber
## Prerequisites for developers computer
## About
This is a project that creates a web scraper (or web crawler) to read prices off online shops.
Ubuntu 18.04.1 LTS Operating system.
It contains 2 AWS lambda functions which are running python3 code. The first is reading a jobs file which lists
all the URLs to be scanned, then it triggers the second function which grabs the price.
The python in this repository runs under Python 2.7
## Prerequisites for developers computer
The python in this repository runs under Python 3.6
To setup source control:
It was developed this under the Ubuntu 18.04.1 LTS Operating system but will also work under Windows 10.
On Linux command line, install and download using these 2 commands:
sudo apt-get install git
git config --global user.name "Neil Spink"
git config --global user.email "neilspink@gmail.com"
git clone git@gitlab.com:neilspink/aws-lambda-price-grabber.git
git clone https://gitlab.com/neilspink/aws-lambda-price-grabber.git
## About
To develop and test the AWS cloud formation templates you need to install the AWS CLI and for the second command
below be able to prove the region and a programatic access information YOUR-SECRET-ID & YOUR-SECRET-KEY :
pip install awscli
aws configure
This project is part of a multi week project and about running an AWS lambda function that is reading the price off an online shop.
To run the crawler locally you need to update/install the packages used by the program:
I've tagged the solution each week
pip3 install -U -r requirements.txt requirements_unit_test.txt
Blog | Tag
--- | ---
https://www.clouded.ch/lambda-web-scraper-project-week-1/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week1
https://www.clouded.ch/lambda-web-scraper-project-week-2/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week2
https://www.clouded.ch/lambda-web-scraper-project-week-3/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week3
https://www.clouded.ch/lambda-web-scraper-project-week-4/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week4
https://www.clouded.ch/lambda-web-scraper-project-week-5/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week5
https://www.clouded.ch/lambda-web-scraper-project-week-6/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week6
https://www.clouded.ch/lambda-web-scraper-project-week-7/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week7
## Notes
To be manually upload the source code to AWS, you would need all packages with the source code, so run we install them
to the source directory, after which you need to ZIP everything:
All AWS Lambda functions start with the prefix name grab_
pip3 install -r ../source/requirements.txt -t ../source
The deployment directory has a README for setting up the solution on Amazon.
## Solutio-t ../sourcen structure
All AWS Lambda functions start with the prefix name lambda_ and the unit tests with test_
You must install required libraries running, in the build folder run lib-download.sh
To manually ZIP the source code and upload to AWS, you would need all libraries with the function, so run this command:
sudo pip install -r ../source/requirements.txt -t ../source
sudo pip install -r ../source/requirements.txt
Unit tests can be run using nose2 which you might need to install:
......@@ -58,4 +58,65 @@ Test data is found at the end of each grab_ function, take the json values { ...
if __name__ == '__main__':
events = {
...
}
\ No newline at end of file
}
# Commands for developing and testing the AWS cloud formation.
Do not use your AWS root account ever! You need to go into the AWS IAM console and create an admin user, if you don't
already have one. You need a pair of access keys which you can generate from the security credentials tab when looking
at the user profile.
## Setup AWS for GitLab
You can upload the aws-create-lambda-json in the AWS CloudFormation console or use these commands to create S3 bucket,
the user for the GitLab pipeline and required permissions to deploy the solution.
If you use these commands, then you will need to change the 'ParameterValue' which is the S3 Bucket Name. You need to
update the name in the manual deployment section below and in the files; aws-create-lambda.json and .gitlab.ci.yml too
aws cloudformation create-stack --stack-name grabber-foundation --template-body file://./deployment/aws-create-foundation.json --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=S3BucketName,ParameterValue=aws-lambda-price-grabber
Once the stack is created you need to go into the IAM console -> Security Credentials tab and "create access key".
Sign into GitLab, then settings -> CI / CD -> Variables and add AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY making them
protected and giving in the access key you just created.
The grab_invoke function has been hardcoded to the S3 Bucket and needs updating. Also the role arn information needs to
be updated.
You must update the .gitlab.ci.yml file with the S3 bucket name, after which you can in GitLab run the pipeline under
CI / CD or using the commands in the following section.
## Manual Solution Deployment
These allow you deploy as GitLab does and to develop it further. It faster to use the CLI than wait on GitLab to do the job
aws s3 cp ./test.zip s3://aws-lambda-price-grabber/
aws configure set default.region eu-central-1
aws configure set aws_access_key_id YOUR-SECRET-ID
aws configure set aws_secret_access_key YOUR-SECRET-KEY
aws cloudformation package --template-file ./deployment/aws-create-lambda.json --s3-bucket aws-lambda-price-grabber --output-template template-export.yml
aws cloudformation deploy --template-file template-export.yml --stack-name aws-lambda-price-grabber-stack --capabilities CAPABILITY_IAM
aws cloudformation delete-stack --stack-name aws-lambda-price-grabber-stack
## Blog posts
During each week of development a blog was posted and the GIT repository was tagged.
Blog | Tag
--- | ---
https://www.clouded.ch/lambda-web-scraper-project-week-1/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week1
https://www.clouded.ch/lambda-web-scraper-project-week-2/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week2
https://www.clouded.ch/lambda-web-scraper-project-week-3/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week3
https://www.clouded.ch/lambda-web-scraper-project-week-4/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week4
https://www.clouded.ch/lambda-web-scraper-project-week-5/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week5
https://www.clouded.ch/lambda-web-scraper-project-week-6/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week6
https://www.clouded.ch/lambda-web-scraper-project-week-7/ | https://gitlab.com/neilspink/aws-lambda-price-grabber/tags/week7
\ No newline at end of file
#!/bin/bash
# you don't need to run this unless you ran -> sudo pip install -r ../source/requirements.txt -t ../source
# you don't need to run this unless you ran manual build, i.e. sudo pip3 install -r ../source/requirements.txt -t ../source
cd ../source
shopt -s extglob
sudo rm -- *.!(py|txt)
sudo rm -- *.!(py|txt|yml)
sudo rm -R -- */
sudo rm six.py
cd ..
......
#!/bin/bash
pip3 install -r ../requirements.txt
pip3 install -r ../requirements_unit_test.txt
\ No newline at end of file
#!/bin/bash
# use to manually build solution
pip3 install -r ../requirements.txt -t ../source
# Commands for developing and testing the AWS cloud formation.
Do not use your AWS root account ever! You need to go into the AWS IAM console and create an admin user, if you don't
already have one. You need a pair of access keys which you can generate from the security credentials tab when looking
at the user profile.
## Install the AWS CLI and configure.
pip install awscli
aws configure
## Setup AWS for GitLab
You can upload the aws-create-lambda-json in the AWS CloudFormation console or use these commands to create S3 bucket,
the user for the GitLab pipeline and required permissions to deploy the solution.
If you use these commands, then you will need to change the 'ParameterValue' which is the S3 Bucket Name. You need to
update the name in the manual deployment section below and in the files; aws-create-lambda.json and .gitlab.ci.yml too
aws cloudformation create-stack --stack-name grabber-foundation --template-body file://./deployment/aws-create-foundation.json --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=S3BucketName,ParameterValue=aws-lambda-price-grabber
Once the stack is created you need to go into the IAM console -> Security Credentials tab and "create access key".
Sign into GitLab, then settings -> CI / CD -> Variables and add AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY making them
protected and giving in the access key you just created.
The grab_invoke function has been hardcoded to the S3 Bucket and needs updating. Also the role arn information needs to
be updated.
You must update the .gitlab.ci.yml file with the S3 bucket name, after which you can in GitLab run the pipeline under
CI / CD or using the commands in the following section.
## Manual Solution Deployment
These allow you deploy as GitLab does and to develop it further. It faster to use the CLI than wait on GitLab to do the job
aws s3 cp ./test.zip s3://aws-lambda-price-grabber/
aws configure set default.region eu-central-1
aws configure set aws_access_key_id YOUR-SECRET-ID
aws configure set aws_secret_access_key YOUR-SECRET-KEY
aws cloudformation package --template-file ./deployment/aws-create-lambda.json --s3-bucket aws-lambda-price-grabber --output-template template-export.yml
aws cloudformation deploy --template-file template-export.yml --stack-name aws-lambda-price-grabber-stack --capabilities CAPABILITY_IAM
aws cloudformation delete-stack --stack-name aws-lambda-price-grabber-stack
......@@ -118,7 +118,7 @@
"arn:aws:s3:::aws-lambda-price-grabber/*",
"arn:aws:cloudformation:eu-central-1:*:stack/*/*",
{ "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/price-grabber-role" }
]
]§
},
{
"Sid": "VisualEditor1",
......
......@@ -2,7 +2,7 @@ import unittest
from source.grabber import JobStorageOS, JOBS_FILENAME
class TestWebsiteMonitorList(unittest.TestCase):
STORAGE_PATH = './deployment/'
STORAGE_PATH = './source/'
def test_load_local_jobs(self):
storage = JobStorageOS(self.STORAGE_PATH)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment