Databricks SQL operators#21363
Merged
Merged
Conversation
alexott
changed the title
pateash
reviewed
Feb 6, 2022
Contributor
Author
|
Thank you for review @pateash , but this is really far from review state - more refactoring is coming |
mik-laj
reviewed
Feb 6, 2022
alexott
force-pushed
the
databricks-sql-operator
branch
from
da9a71f to
29bded7
Compare
alexott
force-pushed
the
databricks-sql-operator
branch
2 times, most recently
from
e74f505 to
74d2e87
Compare
Contributor
Author
|
@potiuk Jarek - would it be possible to review the changes? |
potiuk
approved these changes
Feb 26, 2022
Member
|
You need to rebase @alexott |
No documentation & tests yet
Still need to fix existing tests & add tests for Databricks SQL hook & operator
This includes: * identifying SQL Endpoint by name * allow to output results into a CSV/JSON/JSONL file * fix tests for DatabricksHook * address most of the comments
…rator Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
Split documentation for operators into separate pages & add more content and examples.
alexott
force-pushed
the
databricks-sql-operator
branch
from
d95f421 to
958c6be
Compare
Contributor
Author
|
@potiuk done. thank you for review |
Member
|
Tests are failing though :( |
alexott
force-pushed
the
databricks-sql-operator
branch
from
958c6be to
855aee4
Compare
Contributor
Author
|
🤦 forgot that tests are referring to the |
potiuk
approved these changes
Feb 27, 2022
jedcunningham
added
the
changelog:skip
84 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds new operators to Databricks provider:
DatabrickSqlOperatorthat allows to execute SQL commands against Databricks SQL Endpoints and Databricks clusters.DatabricksCopyIntoOperator(built on top ofDatabrickSqlOperator) that allows to import data into Databricks tables.This operator uses the same connection as other Databricks operators (although it could be discussed), if having a dedicated connection make sense as we can further customize it with specific input fields, etc.
Another possible improvement - make the
databricks-sql-connectordependency optional, but I'm not sure how to make it correctly in Airflowcloses: #21030
closes: #21376