Add Databricks Support
Databricks needs to be added either as an External connection (in the same manner as Data Factory) or as a source. Our flow of data is:
Source system > Data Factory > Databricks > Other (eg. Power BI, SQL Database)
We can add storage accounts at the moment, but I would prefer the option to read from the Hive metastore. Would also be nice to have Databricks as part of the lineage.
Troy Holland
shared this idea
2 comments
-
Anders Boje Hertz
commented
I showed that doing catalogue of hive metastore is doable today here https://www.linkedin.com/feed/update/urn:li:activity:6752223374140870656/
I am in the making of a blog post about it.Just a few hours ago I also managed to create Data lineage in real-time from Databricks to Purview.
Here is a screenshot of the result: https://www.evernote.com/l/ALVDHvuwFr1LMJfsBRLo7KRLS8d6hFdOimYI will bring up a post within a week or two how to do this.
-
Mara Cidri
commented
Yes, it would be great to heve Databricks as part of the Lineage.