-
Ben Sully authored
* Add dagster-databricks package This package is closely modeled off the dagster_aws.emr subpackage and provides the databricks_pyspark_step_launcher resource and the DatabricksRunJobSolidDefinition solid for running Databricks jobs. * Reference Databricks docs in dagster-databricks configs module * Move build_pyspark_zip into dagster_pyspark utils module * Fix style/minor issues in dagster-databricks Specifically: - triple single quotes instead of triple double quotes for docstrings - single quotes instead of double quotes everywhere else - oneline docstrings where possible; start on same line everywhere else - rename 'is_terminal' to 'has_terminated' - use 'databricks_run_id' instead of 'run_id' for clarity - make DatabricksJobRunner.client a property - remove unnecessary blank lines * Add references to Databricks storage docs in 'main' script * Add comment explaining global vars in databricks_step_main.py * Fix Python 2 issues in dagster-databricks * Check invariants when setting up storage in Databricks job * Fix dependencies in dagster-databricks/tox.ini * Move 'secret_scope' field into inner credentials object to simplify Databricks storage * isort dagster-databricks * Add pylint to tox.ini for dagster_databricks * Install dagster-databricks in 'make install_dev_python_modules' * Reference GitHub issue for better storage setup in databricks_step_main.py * Uncomment dagster-azure related config * Replace assert_called_once with call_count for Python3.5 compat * Fix lint errors in databricks.py * Improve handling of libraries by including required libs by default * Fix version to match other dagster libraries * Specify supported_pythons to exclude Python 3.8 from dagster-databricks tests on buildkite See https://github.com/dagster-io/dagster/issues/1960. * Add README for dagster-databricks * Install dagster-databricks in dagster-examples tox.ini * Update snapshot test for dagster example using databricks * Add API docs for dagster_databricks * Add coveragerc for dagster-databricks
This project manages its dependencies using pip.
Learn more