Unit Testing for ADF Projects
There has to be support for automated testing of Azure Data Factory pipelines - perhaps as part of Visual Studio ADF project suite.
This is the saddest product I have every used. Can't add automated quality tests to establish CICD. :(
John Aven commented
Is this an issue that ADF should handle or is it something that should be handled by the developer. If unit testing is needed, then UI development isn't the right course. This can be done, but you need to use the APIs.
This UI driven culture of developing code is appalling, its difficult to debug and you cannot incrementally add value to your business.
This is more of a basic requirement for any pipeline to mock custom job. We are using UI to build these jobs and need something to verify the pipeline. Do you agree if we use System and Job related attributes to create a custom pipeline and verify actual pipeline before deployment?
Real projects out there are always required to go through defined processes like unit testing and automated testing. Unless Microsoft supports a way to come-up with unit testing, it will be difficult for developer community to design reliable applications.
Saul Cruz commented
This is a must, right now most of our ETLs stored procs are unit tested, however our pipelines in ADF are still breaking because of a lack of an automated unit test, we unit test this manually using a separate data factory (test environment), however this is very tedious and time consuming. It'd be nice to have a way of running simple and quick tests and then teardown the environment automatically
Additionally to unit testing, it would be great to have different deployment slots as in other azure services. Multiple environments would also allow easier testing.
Only 37 votes - says a lot about our industry's attitude to quality!