How can we improve Microsoft Azure Data Factory?

← Data Factory

Throw Error Activity

If my pipeline orchestrates an asynchronous operation like processing an Azure Analysis Services model that the pattern is to start the operation asynchronously then loop and check the status. If the status is failed the REST API just says status=Failed but the REST API does not return an HTTP 500 status code so ADF does not fail. So I need a new Throw Error Activity component that will let me build an expression for the ErrorMessage property and throw an error message. Better yet there would be a property which does not throw an error if the property is set to false but does throw an error if the property is set to true.

800 votes
Vote
Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close
You have left! (?) (thinking…)
Greg Galloway shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

25 comments

Sign in
(thinking…)
Sign in with: Microsoft
Forgot password?
Create a password
Signed in as (Sign out)
Close
Close
Submitting...
An error occurred while saving the comment
  • Zoltan Peter commented  ·   ·  Flag as inappropriate

    I added a Copy data action with fake source and sink. Source is specified as QUERY and the sql query is: RAISERROR('error message', 16,1)
    Very lame solution, but it will stop your pipeline and show up as error in the pipeline Monitor.
    But yeah a feature like this is sorely missing.

  • Michael Erickson commented  ·   ·  Flag as inappropriate

    We solved this issue by writing a simple Azure Function App that throws an error message. We pass parameters to it using headers or posting a JSON object. We also have a test parameter "IfTrue" or "IfDefined" to check for true or if null or whitespace. Function is "public static async Task<IActionResult> ThrowError(...)". It returns a "new BadRequestObjectResult" on error or "new OkObjectResult" if no error. The function app is very useful for alternate error return as well, which provides a "IF" like capability.

  • Rodney Joyce commented  ·   ·  Flag as inappropriate

    Not having this leads to so much more complicated code... we should be able to rely on the Alerting framework to catch exceptions and not have to do Logic Apps to send emails etc

  • Zach commented  ·   ·  Flag as inappropriate

    Comon. We need this. We have SPs that return if QC checks are OK or failed and on failure we want the ability to End the Pipeline. Basic functionality. Microsoft is really dropping the ball with Data Factory. Need management for that???

  • Tristan Griffith commented  ·   ·  Flag as inappropriate

    It's over 2 years since this was originally posted. We really need this feature as there are scenarios where an unconditional clean-up activity is needed but we also need to be alerted to failures.

    If you have a clean-up activity triggered by the completion of some other activity, any failures in the activity triggering the clean-up activity get masked and the overall pipeline will say that it "Succeeded".

    The current workaround is to connect a "Web" activity that queries a known unreachable endpoint to force the overall pipeline to fail. It would be great if instead we had proper support to explicitly throw an error (or an option on an activity to fail the overall pipeline on failure).

  • mbitter@rockwellautomation.com commented  ·   ·  Flag as inappropriate

    There needs to be some kind off stop execution command as part of ADF. Sequential 4 eaches continue to run even with a failure. We had to code and create an exception to make it stop. This is not ideal and no fun to support a year later with new resources.

  • Miika Varis commented  ·   ·  Flag as inappropriate

    Definitely needed. We have few pipelines with pre-run checks. We need easy way to prevent full pipeline run if these checks fail.

    There is also another scenario where early prevention would make things smoother; when ETL-pipeline's Extract to staging returns 0 rows (nothing to load any more). Unnecessary running of Transform and Load logic returns errors which does not look good for monitoring and alerts.

  • Joel Cochran commented  ·   ·  Flag as inappropriate

    @DavidP - Both of these requests seem focused on Failure scenarios, which is not sufficient. Early termination should not be limited to Failure, but rather a controlled exit.

  • Joel Cochran commented  ·   ·  Flag as inappropriate

    To add to this, we need a pipeline activity to Terminate a running pipeline AND specify the reported Status [not just 'Failed'. Sometimes it isn't about failure, it's about preventing a failure. It's similar to TRY..CATCH for handling expected exceptions.

  • Anonymous commented  ·   ·  Flag as inappropriate

    please keep adding necessary features to make it suitable counterpart of ssis on cloud.

  • Anugrah Gupta commented  ·   ·  Flag as inappropriate

    Many a times I have come across that during the execution of a pipeline (such as a variable's True or False value), we want the pipeline to "Fail".

    Currently, I am to achieve this using a If Condition Activity (where the variable's value is checked), followed by a Web Activity to trigger Rest API for cancelling the pipeline run.

    https://docs.microsoft.com/en-us/rest/api/datafactory/pipelineruns/cancel

    It would have been great if similar to "Execute Pipeline" if there can be an activity to Kill/Terminate/Cancel the pipeline's run.

  • Nick McDermaid commented  ·   ·  Flag as inappropriate

    Great minds think alike - I am doing exactly the same thing. I think I'll do a database call with RAISERROR. Nasty but I'm already working around the fact that you can't capture refreshId directly from the initial REST call. If we got raiserror capability in ADF and conditional workflow it would make all of my pipelines waaaay simpler.

  • Gaurav Gangwar commented  ·   ·  Flag as inappropriate

    Please add some error handling feature in DF only.

    Because most of the time it's not allowed to use Logic App.

← Previous 1

Data Factory

Categories

Feedback and Knowledge Base

(thinking…)
Hosted by UserVoice