Commit f395ecef authored by Jon Tavernier's avatar Jon Tavernier

add logs notes, sync example, bug found

parent b1931ab5
......@@ -60,7 +60,8 @@ Fun fact: This [Cloud Composer: Qwik Start lab](
- What's interesting is that the "data/ folder and logs/ folder are not subject to capacity limits."
- "Cloud Composer synchronizes the dags/ and plugins/ folders uni-directionally by copying locally and synchronizes data/ and logs/ folders bi-directionally by using Cloud Storage FUSE."
- Regarding logs:
- The Airflow Web UI displays these.
- Stackdriver has these too.
### Example Commands
......@@ -101,6 +102,18 @@ gcloud composer environments storage dags import \
--location us-central1 \
--source ./dags
# alternative: manually sync to gcs
DAGS_BUCKET=$(gcloud composer environments describe jon-airflow-poc \
--location us-central1 \
--format=json | jq -r '.config.dagGcsPrefix')
gsutil -m rsync \
-d \
-r \
./dags \
# see DAGs
gcloud composer environments storage dags list \
--environment jon-airflow-poc \
......@@ -867,6 +880,10 @@ I launched an Airflow cluster than visited the Admin Configuration page to grab
- What is a node pool?
- "When you create a container cluster, the number and type of nodes that you specify becomes the default node pool. Then, you can add additional custom node pools of different sizes and types to your cluster. All nodes in any given node pool are identical to one another."
# Gotchas
1. The `@once` DAG schedule is busted in GCP Cloud Composer. Tasks will not run.
# Local Development
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment