Support ADF Projects in Visual Studio 2017
Currently Visual Studio 2017 does not support Azure Data Factory projects.
Despite the Azure SDK now being included in VS2017 with all other services the ADF project files aren't.
Can you please include this feature so developers can upgrade from VS2015?
@Nick I take your point. And it certainly seems that with e.g. Azure Synapse Analytics, Microsoft is taking clear steps towards to what I'm advocating for. And yes, I agree that this is less about Visual Studio specifically.
While cloud-based UIs are fine, I also enjoy the possibility to work on things locally/offline (at least to a certain extent) as well as behind limited connectivity. Granted that is not often trivial to achieve in data-centric projects, but you can still attempt to decouple the design and runtime from each other.
Whatever the re-imagined developer workflow is to look like, just embedding the web-based visual design surface within Visual Studio doesn't cut it for me - I haven't actually asked for that - and it is not the only way of "supporting ADF projects" either. So I actually agree with you there that there is no technical difference if that is all that is done - except perhaps that you do not have to keep bouncing between your browser and other developer tooling.
@Veli-Jussi I agree that the existing ADF CICD process is isolated and doesn't work properly. However I'd rather see effort on improving the ADF web UI, Git platform integration and CICD rather than building a .VSIX for Visual Studio. I don't see any different between an army of engineers building pipelines (or SSIS packages) in Visual Studio vs an army of engineers building it in the Web UI. I agree there is a gap in data tooling between ADF web development and "everything else" (SSIS, Database Projects etc.) in Visual Studio but to my mind we should be migrating out of Visual Studio and into cloud based development tools. In fact that appears to be the roadmap. I disagree with your last point. Visual Studio does not afford us any advantage in managing complex ETL over a web UI. Regardless of the tool, the same source code is generated.
Absolute madness this hasn't been addressed. 2.5 years later!!!
You can author SSIS projects for ADF v2 in VS 2019 but not ADF v2 native Projects - it's just weird.
@Nick While you are technically correct, unreliable installation and/or bugs within the tooling are beside the point.
Let me clarify my position. What I am advocating for is a consistent, streamlined, end-to-end developer experience. As it stands now, Azure Data Factory has been created as this isolated microcosm with it's own web-based authoring UX. Sure, you can link it up to a (Azure DevOps) Git repository, but even the suggested CI/CD process there is weird and unintuitive with a completely detached 'adf_publish' branch to where actual deployment artifacts are pushed. They've sort of reinvented the wheel by devising this totally unique flow which only applies to Data Factories. This rather myopic picture extends all the way to monitoring - granted one can jump through hoops in order to patch up some unified view by using Azure Monitor.
This would be totally fine if you really could develop, deploy, test and run data pipelines in isolation. But to my experience this is rarely the case. The way the tooling has been designed, perpetuates a model which advocates an army of specialized data engineers as a bottleneck in order to produce any meaningful value. As soon as you introduce any dependencies to the overall solution - such as a relational database, it becomes less obvious what the development process and the associated, preferably automated, deployment steps would look like. If somebody can point me to any non-trivial example, I'm happy to re-evaluate my position.
Until then, however, what I'm asking for is to reimagine the tooling to be aligned with the value stream of a working end-to-end solution instead of just focusing on the "data pipes" as an isolated problem. The fact that Data Factory has to be deployed as a monolith does not exactly help here either.
What I'm saying is the current incarnation of ADF v2 - and it not integrating properly with other development tooling such as Visual Studio - is just a symptom of a larger underlying issue.
I could also talk about the shortcomings of a visual authoring experience for creating and understanding any reasonably complex data pipelines - as well as code reusability, but that's a different topic altogether.
Hey nick, the last time I checked (which was a while ago so maybe something changed) you could integrate with public git sources. Some enterprises don’t feel good storing source code in the cloud, and are using on premise solutions. For example, customers using on premise TFS with integrated Git. To my understanding, there was not a way to have the UI write back to this kind of source code repository, which would be possible for UI development with this feature being made available. With that said, we stopped waiting and just learned to write the pipelines with arm template code.
Nick McDermaid commented
I don't agree. The web UI is far more reliable than SSDT. SSDT never installs properly first time and if you don't have admin rights you're out of luck. The web UI is fully integrated with Git, DevOps etc. so all these arguments about deploying, branching etc. don't make sense.
The DataOps story using Microsoft tooling is woefully inadequate without this. One should really be able to work on all the interrelated components within a data platform with unified tooling, commit into version control and deploy the changes into the appropriate environments e.g. with Azure DevOps. Having ADF support in Visual Studio is one piece of the puzzle and it seems that the teams developing various Azure services are not thinking about this holistically.
Now that the SSDT support for Azure SQL DW database projects is (finally) in preview, and ADF is the preferred Microsoft-provided tool for orchestrating data transformations, it should be fairly obvious that these things are closely related.
Sad that this is the top-voted feedback item for Data Factory and not even a comment from the team in 2 years...
Nick Hopkins commented
Hopefully it goes without saying that this request extends to VS 2019 as well.
Antonio Policastro commented
I don't know if anyone will ever read this, but I need to express my frustration.
PLEASE do something about this request. Thank you.
In just over a month this request will be celebrating its 2nd birthday 🍰🥂
Brian Green commented
Like everybody else. PLEASE add this to VS2017 and stop wasting our time.
did any one connected ADF with HDFS to ADL data transfer.. if yes then please let me know. They have provide tutorial but in this they are using integration service of windows to connect with on prim HDFS.. on prime HDFS is in linux then how we can use ADF.
Kev moore commented
Hi. Granted the Web authoring tool is getting much better. However, we really need the ability to manage our source code in the same way as all of our other projects. Namely, using VS to develop. Push to Dev env for testing, merge to master and push to prod. At the moment, and changes to the pipelines or connectors have to be done in the web, then we have to export the ARM template (making sure connectors are correct for each branch (and we have a lot)). Then merge through VS and deploy manually.
As the VSTS CI/CD pipeline is being promoted as DevOps, surely this should include Data Factory?
Please support ADV version 2 in Visual Studio. The Vision if Visual development must still support CI/CD!
James Denning commented
Actually theADF v2 web design surface I think is pretty amazing, it works well and doesn't crash half as much as visual studio would. What a lot of commenters on here really need is a proper CI/CD approach which would be facilitated by a totally different feature asked for elsewhere (programmatic/API capability to export the ARM template).
atin kothari commented
Can we please get VS 2017 to support Azure Data Factory projects?
Please the visual interface on ADF V2 is great for testing but using visual studio json templates would be really great too .. especially when working against multiple environments and pipelines
Manish Pradhan commented
This is very vital to support movement of ADF projects across environments using Visual studio configurations as well as for ease of development. Please support.
Is anyone using VS2015 anymore?