How can we improve Microsoft Azure Data Factory?

Move Activity

Activity that copies and then deletes.

916 votes
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    Max Smiley shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    34 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      Submitting...
      • Keith Patton commented  ·   ·  Flag as inappropriate

        this is blocking our adoption of azure data factory. We are happy to do a custom activity but have no working examples around how to delete a file using a file system linked service tied to on prem data source. I am coompletely bamboozled as to how a triggered copy activity could be of use to anyone without the ability to delete or archive the files as part of the operation, what am i missing!?

      • Nik commented  ·   ·  Flag as inappropriate

        is there an ETA on this ?? It's been stagnant for way too long...

      • Nik commented  ·   ·  Flag as inappropriate

        This definitely needs to be there...it just makes sense and fits the need

      • Sunitha commented  ·   ·  Flag as inappropriate

        Definitely need Move and Delete features added in ADF. The entire point of using such tools is for its simplicity while focusing on more important logic, If such basic features are missed out, then it is so disappointing especially when you take so much efforts in learning them.

      • Derek Adams commented  ·   ·  Flag as inappropriate

        I've had nothing but praises for the Data Factory until I realized that I couldn't delete a file after copying it. So disappointing! A ton of work wasted unless this feature is added.

      • Leeten commented  ·   ·  Flag as inappropriate

        Hi there, Whilst the ADF 'Move/Delete/Rename' base function is being considered as an imporvement, can we use a Web activity to call the REST API provide by Data Lake to move or delete a file?
        BUT, what are the instruction specific to calling Data Lake's REST API, and what are the url points etc..?

        I've looked at this page, but it's fairly generic - NOT specfic to Data Lake:

        https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

        Any help would be great.

      • Rodrigo commented  ·   ·  Flag as inappropriate

        IT can't be so complex "rm" ftp instruction ..... ADF base technology is "copy" files....I can imagine Microsoft IT engineer had this idea during implementation.

      • Ron DeFreitas commented  ·   ·  Flag as inappropriate

        This is an absolute must have. Either as a "Move" Activity or a "Delete" Activity. It's absurd that this platform doesn't provide this capability.

      • Anonymous commented  ·   ·  Flag as inappropriate

        I'm not fussed whether this is implemented as delete after copy or rename the file, or even move the file to another directory. The fact is copy it's useless without this functionality. It's almost the most basic integration pattern there is. I realise I can write a custom activity in .Net, but that shouldn't be necessary for such a simple use-case.

      • Julian commented  ·   ·  Flag as inappropriate

        Do you have any information about this feature?
        Move and delete source file is a common and important scenario..

      • James Devlin commented  ·   ·  Flag as inappropriate

        We really need this feature. It's not great learning ADF and then finding out you have to also learn how to implement Custom Activity and then also find you have to learn Azure Batch.

        Learning is a part of being an IT professional but at some point you just want to deliver a feature :/

      • Craig R commented  ·   ·  Flag as inappropriate

        I'm split on this one. I recommend "renaming" the file to achieve what you're looking for "and" avoid deleting. Below are my thoughts, and please correct me if I'm wrong.

        Part of me says deleting files should be avoided because data loss is inevitable.
        No matter how many checks you put in place, someone at sometime will introduce some code that does something wrong, and it may not even be your code (e.g. downstream consumer that deletes the output data so carefully protected).

        And the other side of me realizes that in many cases moving data is a necessity, whether due to requirements of a system you don't control that outputs files of the same name, or even to deal with space concerns.

        My recommendation is to avoid deletion and instead rename files. Rename them and it actually "can" move them on many of these technologies like ADLS. You may find you can achieve what you want "and" not delete the data. After all, the path is really just a label applied to a set of related data presented abstractly as a "file"... the folders don't exist in many of these systems anymore (e.g. you can vi a folder on linux because a folder on linux is just a special file)

      • Anonymous commented  ·   ·  Flag as inappropriate

        Just absolutely nuts that this does not exist already. Really shows me that the technology is nowhere near mature.

      ← Previous 1

      Feedback and Knowledge Base