Capture metadata of ETL processes designed in Data Factory
Big data platform are mainly powered by two major components in their architecture; Source-to-target mapping and ETL processes. So the analyst performing analytics on a specific dataset needs to understand where the data came from, which business rules applied on the data while in transient,...etc. So, an analyst could create a lineage diagram to help them understand the data movement in details. In the same time, data engineers could generate an impact analysis on ETL procesees and target datasets in case any data source was changed.
The short version is Data Catalog to harvest metadata from Data Factory that describes the ETL process
Is here any progress on this from Microsoft ?
Shannon Lowder commented
This would be awesome! Reading the JSON definition of the ADF pipelines should be simple.