Support ADF Projects in Visual Studio 2017
Currently Visual Studio 2017 does not support Azure Data Factory projects.
Despite the Azure SDK now being included in VS2017 with all other services the ADF project files aren't.
Can you please include this feature so developers can upgrade from VS2015?
Absolute madness this hasn't been addressed. 2.5 years later!!!
You can author SSIS projects for ADF v2 in VS 2019 but not ADF v2 native Projects - it's just weird.
Veli-Jussi Raitila commented
@Nick While you are technically correct, unreliable installation and/or bugs within the tooling are beside the point.
Let me clarify my position. What I am advocating for is a consistent, streamlined, end-to-end developer experience. As it stands now, Azure Data Factory has been created as this isolated microcosm with it's own web-based authoring UX. Sure, you can link it up to a (Azure DevOps) Git repository, but even the suggested CI/CD process there is weird and unintuitive with a completely detached 'adf_publish' branch to where actual deployment artifacts are pushed. They've sort of reinvented the wheel by devising this totally unique flow which only applies to Data Factories. This rather myopic picture extends all the way to monitoring - granted one can jump through hoops in order to patch up some unified view by using Azure Monitor.
This would be totally fine if you really could develop, deploy, test and run data pipelines in isolation. But to my experience this is rarely the case. The way the tooling has been designed, perpetuates a model which advocates an army of specialized data engineers as a bottleneck in order to produce any meaningful value. As soon as you introduce any dependencies to the overall solution - such as a relational database, it becomes less obvious what the development process and the associated, preferably automated, deployment steps would look like. If somebody can point me to any non-trivial example, I'm happy to re-evaluate my position.
Until then, however, what I'm asking for is to reimagine the tooling to be aligned with the value stream of a working end-to-end solution instead of just focusing on the "data pipes" as an isolated problem. The fact that Data Factory has to be deployed as a monolith does not exactly help here either.
What I'm saying is the current incarnation of ADF v2 - and it not integrating properly with other development tooling such as Visual Studio - is just a symptom of a larger underlying issue.
I could also talk about the shortcomings of a visual authoring experience for creating and understanding any reasonably complex data pipelines - as well as code reusability, but that's a different topic altogether.
Hey nick, the last time I checked (which was a while ago so maybe something changed) you could integrate with public git sources. Some enterprises don’t feel good storing source code in the cloud, and are using on premise solutions. For example, customers using on premise TFS with integrated Git. To my understanding, there was not a way to have the UI write back to this kind of source code repository, which would be possible for UI development with this feature being made available. With that said, we stopped waiting and just learned to write the pipelines with arm template code.
I don't agree. The web UI is far more reliable than SSDT. SSDT never installs properly first time and if you don't have admin rights you're out of luck. The web UI is fully integrated with Git, DevOps etc. so all these arguments about deploying, branching etc. don't make sense.
Veli-Jussi Raitila commented
The DataOps story using Microsoft tooling is woefully inadequate without this. One should really be able to work on all the interrelated components within a data platform with unified tooling, commit into version control and deploy the changes into the appropriate environments e.g. with Azure DevOps. Having ADF support in Visual Studio is one piece of the puzzle and it seems that the teams developing various Azure services are not thinking about this holistically.
Now that the SSDT support for Azure SQL DW database projects is (finally) in preview, and ADF is the preferred Microsoft-provided tool for orchestrating data transformations, it should be fairly obvious that these things are closely related.
Sad that this is the top-voted feedback item for Data Factory and not even a comment from the team in 2 years...
Nick Hopkins commented
Hopefully it goes without saying that this request extends to VS 2019 as well.
Antonio Policastro commented
I don't know if anyone will ever read this, but I need to express my frustration.
PLEASE do something about this request. Thank you.
In just over a month this request will be celebrating its 2nd birthday 🍰🥂
Brian Green commented
Like everybody else. PLEASE add this to VS2017 and stop wasting our time.
did any one connected ADF with HDFS to ADL data transfer.. if yes then please let me know. They have provide tutorial but in this they are using integration service of windows to connect with on prim HDFS.. on prime HDFS is in linux then how we can use ADF.
Kev moore commented
Hi. Granted the Web authoring tool is getting much better. However, we really need the ability to manage our source code in the same way as all of our other projects. Namely, using VS to develop. Push to Dev env for testing, merge to master and push to prod. At the moment, and changes to the pipelines or connectors have to be done in the web, then we have to export the ARM template (making sure connectors are correct for each branch (and we have a lot)). Then merge through VS and deploy manually.
As the VSTS CI/CD pipeline is being promoted as DevOps, surely this should include Data Factory?
Please support ADV version 2 in Visual Studio. The Vision if Visual development must still support CI/CD!
James Denning commented
Actually theADF v2 web design surface I think is pretty amazing, it works well and doesn't crash half as much as visual studio would. What a lot of commenters on here really need is a proper CI/CD approach which would be facilitated by a totally different feature asked for elsewhere (programmatic/API capability to export the ARM template).
atin kothari commented
Can we please get VS 2017 to support Azure Data Factory projects?
Please the visual interface on ADF V2 is great for testing but using visual studio json templates would be really great too .. especially when working against multiple environments and pipelines
Manish Pradhan commented
This is very vital to support movement of ADF projects across environments using Visual studio configurations as well as for ease of development. Please support.
Is anyone using VS2015 anymore?
We are living in a world where analyzing data is more and more important. That means data movement and integration are vital. ADF development in Visual Studio makes this easier and quicker than front-end UI development. Everything in a data analytics cloud solution should be easy to build, maintain and manage source code not just some components. Please bring this add-in functionality back into Visual Studio.
Debashish Sarkar commented
This was an unpleasant surprise while migrating from VS2015 to VS2017 for us. MS, please support ADF in VS2017!