Commit 4d13ef84 authored by Jon Tavernier's avatar Jon Tavernier

correct yaml

parent 3c13aabc
......@@ -7,21 +7,12 @@ I have used Apache Airflow. Google offers Apache Airflow as-a-service via its C
1. How to launch cluster programmatically?
- for poc testing, gcloud can be used: gcloud composer environments create. this can then be automated elsewhere.
- reference: https://cloud.google.com/composer/docs/how-to/managing/creating
- example
```bash
gcloud composer environments create jon-airflow-poc \
--location=us-central1 \
--zone=us-central1-a \
--machine-type=n1-highcpu-4 \
--python-version=3 \
--node-count 3 \
--env-variables='CITY=Chicago'
```
- See example in commands below.
- other notable options:
- `disk-size`: default is 100 GB. used for the node VMs. min is 20 GB.
- notable missing options:
- I see no way to set the GCS bucket at cluster creation time.
1. How are airflow config options set?
- I see no way to set the GCS bucket at cluster creation time. So, have to grab that after cluster is created.
1. How are Airflow config options set?
- at cluster creation time, the `airflow-configs` option can be used.
- some properties are blocked: https://cloud.google.com/composer/docs/concepts/airflow-configurations
- after the cluster is running, the `update-airflow-configs` option can be used, but it will cause the cluster to enter an updating state.
......@@ -42,7 +33,7 @@ I have used Apache Airflow. Google offers Apache Airflow as-a-service via its C
--set 'DAG_GIT_VERSION' 'BlAH'
```
1. What functionality is beta vs released?
-
- Need to look into...
1. What GCP components are created?
- Kubernetes Engine
- GCP auto creates a k8s cluster for each Composer environment we create.
......@@ -57,14 +48,14 @@ I have used Apache Airflow. Google offers Apache Airflow as-a-service via its C
- GCP Console > Composer > Environment > Airflow web UI link.
- Reference: https://cloud.google.com/composer/docs/how-to/accessing/airflow-web-interface
1. How are upgrades handled?
-
- Need to look into.
1. What's the cluster lifecycle? I think it will be:
1. Create Cluster
1. Get Cluster's DAGs GCS Location
1. Sync DAGs to Cluster's DAGs GCS Location
1. Profit
## Example Commands
### Example Commands
```bash
# get some help
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment