Provision of cloud resources (virtual machine + disk) for DLE (!2) · Merge requests · PostgresAI / dle-se-ansible

Provision of cloud resources (virtual machine + disk) for DLE on:

Additionally:

Role "envoy": (commit)
- Add the Envoy repository only if the proxy_install variable is true
- Use the get_url module to download the key and use a higher timeout (1 minute)
New role: "authorized-keys" (adds additional public keys specified in the ssh_public_keys variable to the system)
New variables: dle_platform_project_name, dle_platform_org_key
Role "deploy-finish"
- replace localhost to 127.0.0.1
New variable: dle_config_file to copy an existing DLE configuration
- if the DLE configuration file is defined in the dle_config_file variable, this file will simply be copied to the DLE server instead of generating a new configuration file.
- example: ansible-playbook deploy_dle.yml --extra-vars "dle_host=user@server-ip-address dle_config_file='server.yml'"
Add retry for task "Install system packages"
Fix the execution of the playbook using the tag
- previously, it was impossible to perform a playbook with a tag (for example --tags proxy)

DigitalOcean

Quick Start (example):

Requirements:

Personal Access Token

Provision:

Export DO_API_TOKEN

export DO_API_TOKEN=dop_v1_a05c2xxxxxxxxxxxxx1e44bef

Run playbook:

ansible-playbook deploy_dle.yml --extra-vars \
  "provision=digitalocean \
  server_name=vitaliy-dle-test \
  server_type=m-2vcpu-16gb \
  server_image=ubuntu-22-04-x64 \
  server_location=sfo3 \
  volume_size=100"

description of variables:

server_name: dblab-server # (required) This is the name of the Droplet. Must be formatted by hostname rules.
server_type: m-2vcpu-16gb # (required) Please see https://slugs.do-api.dev/ for current slugs.
server_image: ubuntu-22-04-x64 # (required) This is the slug of the image you would like the Droplet created with.
server_location: sfo3 # (required) This is the slug of the region you would like your Droplet to be created in.
volume_size: 100 # (required) This is the storage for zpool_disk (size in gigabytes)
ssh_key_name: "" # (optional) SSH key to be added to the DLE server. If not specified, all ssh keys will be added (if exists).
ssh_key_content: "" # (optional) if specified, the contents of the public key will be added to the cloud. Specified in a pair with the ssh_key_name variable.
state: present # 'present' to create (default) or 'absent' to delete

(Optional) Destroy VM

ansible-playbook provision.yml --extra-vars "provision=digitalocean server_name=vitaliy-dle-test server_location=sfo3 state=absent"

Hetzner Cloud

Quick Start (example):

Requirements:

API Token

Provision:

Export HCLOUD_API_TOKEN

export HCLOUD_API_TOKEN=HxzUBrUxxxxxxxxxxxxxxxxx9DJZi

Run playbook:

ansible-playbook deploy_dle.yml --extra-vars \
  "provision=hetzner \
  server_name=vitaliy-dle-test \
  server_type=ccx22 \
  server_image=ubuntu-22.04 \
  server_location=ash \
  volume_size=100"

description of variables:

server_name: dblab-server # (required) The Name of the Hetzner Cloud server.
server_type: ccx22 # (required) The Server Type of the Hetzner Cloud server. Please see https://docs.hetzner.cloud/#server-types
server_image: ubuntu-22.04 # (required) Image the server should be created from. Please see https://docs.hetzner.cloud/#images
server_location: ash # (required) This is the region where the server will be created (ash - us-east "Ashburn, VA"). Please see https://docs.hetzner.com/cloud/general/locations/, https://docs.hetzner.cloud/#locations
server_network: "" # (optional) The name of the private network. If specified, the server will be added to this network (must be created in advance).
volume_size: 100 # (required) This is the storage for zpool_disk (size in gigabytes)
ssh_key_name: "" # (optional) SSH key to be added to the DLE server. If not specified, all ssh keys will be added (if ssh_public_keys is not defined).
ssh_key_content: "" # (optional) if specified, the contents of the public key will be added to the cloud. Specified in a pair with the ssh_key_name variable.
state: present # 'present' to create (default) or 'absent' to delete

(Optional) Destroy VM

ansible-playbook provision.yml --extra-vars "provision=hetzner server_name=vitaliy-dle-test server_location=ash state=absent"

AWS

Quick Start (example):

Requirements:

Access key id and secret

Provision:

Export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

export AWS_ACCESS_KEY_ID=AKIA*********KVSQY
export AWS_SECRET_ACCESS_KEY=noUI**************f8X3P2

Run playbook:

ansible-playbook deploy_dle.yml --extra-vars \
  "provision=aws \
  server_name=vitaliy-dle-test \
  server_type=r5.large \
  server_image=ami-0557a15b87f6559cf \
  server_location=us-east-1 \
  volume_size=100 \
  ssh_key_name=Vitaliy"

Note: For AWS, specifying the ssh_key_name variable is required.

description of variables:

server_name: dblab-server # (required) The Name tag for the instance.
server_type: r5.large # (required) Instance type to use for the instance, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html.
server_image: ami-0557a15b87f6559cf # (required) ami ID to use for the instance. ami-0557a15b87f6559cf is Ubuntu 22.04 LTS on us-east-1
server_location: us-east-1 # (required) The AWS region to use. See the Amazon AWS documentation for more information http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region.
server_network: "" # (optional) The subnet ID in which to launch the instance (VPC). If specified, the server will be added to this network (must be created in advance). If server_network is not specified, use default vpc subnet
volume_size: 100 # (required) This is the storage for zpool_disk (size in gigabytes)
volume_type: "" # (optional) The volume type. Default "gp3"
ssh_key_name: "" # (required) Name of the SSH access key to assign to the instance - must exist in the region the instance is created.
ssh_key_content: "" # (optional) if specified, the contents of the public key will be added to the cloud. Specified in a pair with the ssh_key_name variable.
state: present # 'present' to create (default) or 'absent' to delete

(Optional) Destroy VM

ansible-playbook provision.yml --extra-vars "provision=aws server_name=vitaliy-dle-test server_location=us-east-1 state=absent"

GCP

Quick Start (example):

Requirements:

Create a service account
Create and save the JSON key for the service account and point to them using GCP_SERVICE_ACCOUNT_CONTENTS variable.

Provision:

Export GCP_SERVICE_ACCOUNT_CONTENTS

export GCP_SERVICE_ACCOUNT_CONTENTS='{
  "type": "service_account",
  "project_id": "my-project",
  "private_key_id": "c764349XXXXXXXXXX72f",
  "private_key": "-----BEGIN PRIVATE KEY-----\nXXXXXXXXXX==\n-----END PRIVATE KEY-----\n",
  "client_email": "my-sa@my-project.iam.gserviceaccount.com",
  "client_id": "111111112222222",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-sat%40my-project.iam.gserviceaccount.com"
}'

Run playbook:

# minimal (example)
ansible-playbook deploy_dle.yml --extra-vars \
  "provision=gcp \
  server_name=vitaliy-dle-test \
  server_type=n2-standard-4 \
  server_image=projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts \
  server_location=us-central1-a \
  volume_size=100"

# extended (example)
ansible-playbook deploy_dle.yml --extra-vars \
  "provision=gcp \
  server_name=vitaliy-dle-test \
  server_type=n2-standard-4 \
  server_image=projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts \
  server_location=us-central1-a \
  volume_size=100 \
  volume_type=local-ssd \
  gcp_project=postgres-ai \
  ssh_key_content=\"ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAgEA0TU9YoE5MwvOKKxpGn8CoU4bkd2Lh5JNSN5/...w+YyNozpEzbGg63wX2uPq35NlVL6Bn/whzcMINzKKCc7AVGbk\""

Note: For GCP, there is no need to specify the ssh_key_name variable, the ssh_key_content variable is used only.

description of variables:

server_name: vitaliy-dle-test # (required) The Name for the instance.
server_type: n2-standard-4 # (required) A reference to a machine type which defines VM kind. See https://cloud.google.com/sdk/gcloud/reference/compute/machine-types/listhttps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html
server_image: projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts # (required) The source image to create the system disk. See https://cloud.google.com/sdk/gcloud/reference/compute/images/list
server_location: us-central1-a # (required) A reference to the zone where the machine resides. See https://cloud.google.com/sdk/gcloud/reference/compute/zones/list
volume_size: 100 # (required) This is the storage for zpool_disk (size in gigabytes)
volume_type: "" # (optional) Specifies the disk type to use to create the instance. If not specified, the default is pd-ssd. See https://cloud.google.com/sdk/gcloud/reference/compute/disk-types/list
ssh_key_content: "" # (optional) if ssh_key_content is defined, add this public key to the instance. Otherwise, get the user public key ('~/.ssh/id_rsa.pub') from the system (if exists).
gcp_project - (optional) The Google Cloud Platform project to use. by default, the project data from GCP_SERVICE_ACCOUNT_CONTENTS is used.
state: present # 'present' to create (default) or 'absent' to delete

(Optional) Destroy VM

ansible-playbook provision.yml --extra-vars "provision=gcp server_name=vitaliy-dle-test server_location=us-central1-a state=absent"

issue https://gitlab.com/postgres-ai/database-lab/-/issues/486

Edited Mar 24, 2023 by Vitaliy Kukharik

Provision of cloud resources (virtual machine + disk) for DLE

Additionally:

DigitalOcean

Hetzner Cloud

AWS

GCP

Merge request reports