Skip to content
  • Tim Swast's avatar
    Composer: initial workflow and REST samples (#1468) · a4de1a37
    Tim Swast authored
    * Add quickstart example DAG.
    
    Add first airflow example DAG. This DAG demonstrates how to call
    Dataproc from airflow.
    
    Change-Id: I2f0464da37f11085dfc927f4f4d96d6a99634aad
    
    * Add region tags for cloud.google.com.
    
    Also, adjusts indentation to 4 spaces so that `flake8` passes.
    
    Change-Id: Id3e1fbfa5bd41a4e11e0cd7991b058b063241e6b
    
    * Add composer_quickstart_steps region tag.
    
    Change-Id: Ib0bbacd91529fac272e2da0c72abca5db9b4fad8
    
    * Test Composer workflow sample.
    
    The GCP Airflow operators only work on Python 2.7 (a constraint
    they inherited from Apache Beam), so I've made some changes to nox.py to
    exclude them from the Python 3 builds.
    
    Change-Id: Ia10971dd5a7b14279b236041836e317e79693258
    
    * Add DAG that runs BigQuery and notifies user.
    
    Add a sample Airflow DAG that will create BigQuery dataset, run
    BigQuery, export results to Cloud Storage, notify user by email when
    results are ready, and clean up the dataset.
    
    Change-Id: Ia8242df29223d910b2d269a9bb93720b35470b7a
    
    * Composer: add sample to get GCS bucket for DAGs
    
    Since there are not Cloud client libraries for Composer yet, this sample
    uses the REST API directly via the google-auth and requests libraries.
    
    Sample to be used at https://cloud.google.com/composer/docs/how-to/using/managing-dags
    
    Also, enables Kokoro testing for Composer samples. (Uses Python 2.7
    since Cloud Composer is currently restricted to Python 2.7)
    
    Change-Id: Icb37e079992c88eedc06cdcc3d72db5106d10ef5
    
    * Add tests for BQ notify DAG.
    
    Requires master copy of Airflow for `bigquery_get_data` operartor.
    
    Change-Id: I73cd2cfb2458b67bed1a77e65966d5018e8bb45d
    
    * Composer: Fix flake8 errors.
    
    Change-Id: I2856bc6cb866bd6f7abbac8de3323797a83c9857
    
    * Composer: add region tags to notification DAG sample.
    
    Change-Id: I657e052fa851daa7c72045762090a2e27dd406d3
    
    * Set machine type in quickstart to n1-standard-1.
    
    The default machine type was n1-standard-4, which exceeds trial
    project quota. This CL changes the machine type to n1-standard-1
    since a more powerful machine is not necessary for quickstart
    sample data.
    
    Change-Id: I46af68c29145f7a7ce303afdad4708bda7d9e6dd
    
    * Add composer config to travis test env.
    
    Change-Id: I9c27c75cbea8c5ed4edf859d26980e24ea270eea
    a4de1a37
This project manages its dependencies using pip. Learn more