Skip to content
  • Ben Sully's avatar
    Add dagster_databricks package for Databricks integration (#2468) · 19146d4d
    Ben Sully authored
    * Add dagster-databricks package
    
    This package is closely modeled off the dagster_aws.emr subpackage and
    provides the databricks_pyspark_step_launcher resource and the
    DatabricksRunJobSolidDefinition solid for running Databricks jobs.
    
    * Reference Databricks docs in dagster-databricks configs module
    
    * Move build_pyspark_zip into dagster_pyspark utils module
    
    * Fix style/minor issues in dagster-databricks
    
    Specifically:
    
    - triple single quotes instead of triple double quotes for docstrings
    - single quotes instead of double quotes everywhere else
    - oneline docstrings where possible; start on same line everywhere else
    - rename 'is_terminal' to 'has_terminated'
    - use 'databricks_run_id' instead of 'run_id' for clarity
    - make DatabricksJobRunner.client a property
    - remove unnecessary blank lines
    
    * Add references to Databricks storage docs in 'main' script
    
    * Add comment explaining global vars in databricks_step_main.py
    
    * Fix Python 2 issues in dagster-databricks
    
    * Check invariants when setting up storage in Databricks job
    
    * Fix dependencies in dagster-databricks/tox.ini
    
    * Move 'secret_scope' field into inner credentials object to simplify Databricks storage
    
    * isort dagster-databricks
    
    * Add pylint to tox.ini for dagster_databricks
    
    * Install dagster-databricks in 'make install_dev_python_modules'
    
    * Reference GitHub issue for better storage setup in databricks_step_main.py
    
    * Uncomment dagster-azure related config
    
    * Replace assert_called_once with call_count for Python3.5 compat
    
    * Fix lint errors in databricks.py
    
    * Improve handling of libraries by including required libs by default
    
    * Fix version to match other dagster libraries
    
    * Specify supported_pythons to exclude Python 3.8 from dagster-databricks tests on buildkite
    
    See https://github.com/dagster-io/dagster/issues/1960.
    
    * Add README for dagster-databricks
    
    * Install dagster-databricks in dagster-examples tox.ini
    
    * Update snapshot test for dagster example using databricks
    
    * Add API docs for dagster_databricks
    
    * Add coveragerc for dagster-databricks
    19146d4d
This project manages its dependencies using pip. Learn more