Change Data Capture feature for RDBMS (Oracle, SQL Server, SAP HANA, etc)
Is it possible to have CDC features on ADF please ?
Can you please give SOME kind of update or any information on this. It is 2021 now and many competitors have already had this functionality for a long time like Fivetran, Streamset, Talend, Qlik, Oracle etc. If Microsoft keeps delaying this many will likely look to switch tools as CDC is an important and better technology to ingest the data to Data/Delta Lake!
Do we have any dates when this feature can be implemented?
Kevin Brown commented
Desperately needed to help take on large/ MF based data stores.
that would be a great feature to develop and maintain data pipelines
Jason Steele commented
Is this the same as "streaming support" as mentioned by Mark Kromer here: https://twitter.com/KromerBigData/status/1205577192098238464?s=20 ?
If so, then I would definitely like this.
I want to be able to submit rows/JSON docs as and when they are insertedupdated in the source.
Abhishek Narain commented
Please fill in the survey here for providing us more information arounds CDC - http://aka.ms/cdcsurvey.
Incremental data copy from SAP HANA Tables to azure cloud ADLS
A great idea.
It would simplify the task to keep up-to-date copies of table/view objects of an other system in Azure SQL. A nice feature would be, if the initial replication would also create all required tables in Azure SQL automatically.
The sync operation could be optionally bi-directional. This would allow to build closed loop applications very easily.
another vote for incremental data copy. In my use case I would to incrementally copy data from S3 to Azure.
Mike Westaway commented
I believe this would deliver the same effect as 'incremental data copy' so these could be combined https://feedback.azure.com/forums/270578-data-factory/suggestions/15903187-incremental-data-copy
Ben Hatton commented
How about built-in support for CDC from SQL Server sources, or even better, a more flexible 'audit table' support for any database keeping custom audit trails that are organised with a incrementing index?
Incremental processing is really quite important for any meaningful implementation. This changes the slice concept from regular interval to a set defined by pointer positions.
The key here is that a pointer needs to be kept from run to run to keep track of the last processed position in such a table - this could be stored in a sink location of choice?
Maybe this just needs a reference implementation/template if we think it already can be done with a bit of clever design...?
Provide the ability to copy changed records in a single directional data flow from source to destination. For example, one way synchronize from an on-prem SQL Server to Azure SQL Data Warehouse.
Nirav Shah commented
Currently we can Copy data , but it would be great to have data sync activity that can keep two data sets in Sync and does not need to copy app the data. Some way to get delta between data sets.
There can be basic restrictions like both data sets needs to be of same type and it can start with Azure SQL and SQL supported.