Throw Error Activity
If my pipeline orchestrates an asynchronous operation like processing an Azure Analysis Services model that the pattern is to start the operation asynchronously then loop and check the status. If the status is failed the REST API just says status=Failed but the REST API does not return an HTTP 500 status code so ADF does not fail. So I need a new Throw Error Activity component that will let me build an expression for the ErrorMessage property and throw an error message. Better yet there would be a property which does not throw an error if the property is set to false but does throw an error if the property is set to true.
Greg Galloway
shared this idea
27 comments
-
Thomas Alauzet
commented
To add to this, we need a pipeline activity to Terminate a running pipeline when an error occurs.
I had issue with Azure Function and Databricks where exeptions errors would not cause the pipeline to return a failure. We need a TRY/CATCH for handling expected exceptions.The linked images clearly show that we should see an excution failure in the pipeline...
-
George Mav
commented
This is a must have for orchestrating flows etc. I wonder this was not implemented as standard from day 1.
-
Zoltan Peter
commented
I added a Copy data action with fake source and sink. Source is specified as QUERY and the sql query is: RAISERROR('error message', 16,1)
Very lame solution, but it will stop your pipeline and show up as error in the pipeline Monitor.
But yeah a feature like this is sorely missing. -
Michael Erickson
commented
We solved this issue by writing a simple Azure Function App that throws an error message. We pass parameters to it using headers or posting a JSON object. We also have a test parameter "IfTrue" or "IfDefined" to check for true or if null or whitespace. Function is "public static async Task<IActionResult> ThrowError(...)". It returns a "new BadRequestObjectResult" on error or "new OkObjectResult" if no error. The function app is very useful for alternate error return as well, which provides a "IF" like capability.
-
Rodney Joyce
commented
Not having this leads to so much more complicated code... we should be able to rely on the Alerting framework to catch exceptions and not have to do Logic Apps to send emails etc
-
Zach
commented
Comon. We need this. We have SPs that return if QC checks are OK or failed and on failure we want the ability to End the Pipeline. Basic functionality. Microsoft is really dropping the ball with Data Factory. Need management for that???
-
Tristan Griffith
commented
It's over 2 years since this was originally posted. We really need this feature as there are scenarios where an unconditional clean-up activity is needed but we also need to be alerted to failures.
If you have a clean-up activity triggered by the completion of some other activity, any failures in the activity triggering the clean-up activity get masked and the overall pipeline will say that it "Succeeded".
The current workaround is to connect a "Web" activity that queries a known unreachable endpoint to force the overall pipeline to fail. It would be great if instead we had proper support to explicitly throw an error (or an option on an activity to fail the overall pipeline on failure).
-
mbitter@rockwellautomation.com
commented
There needs to be some kind off stop execution command as part of ADF. Sequential 4 eaches continue to run even with a failure. We had to code and create an exception to make it stop. This is not ideal and no fun to support a year later with new resources.
-
Andy Steinke
commented
This is definitely a needed feature that would greatly simplify common development problems.
-
demolition porg
commented
Please add this to data factory, there are a number of scenario's where this would be useful.
-
Miika Varis
commented
Definitely needed. We have few pipelines with pre-run checks. We need easy way to prevent full pipeline run if these checks fail.
There is also another scenario where early prevention would make things smoother; when ETL-pipeline's Extract to staging returns 0 rows (nothing to load any more). Unnecessary running of Transform and Load logic returns errors which does not look good for monitoring and alerts.
-
Konstantinos Michas
commented
Although I don't have high hopes since this feedback raised back in 2018 and already in mid 2020, it seems to me that nobody cares to step up in MS Azure Dev team. I gave it a try anyway and created this one today
-
Sven Gfeller
commented
For the time beeing, I am doing it like this where "Fail" is a boolean variable.
-
Joel Cochran
commented
@DavidP - Both of these requests seem focused on Failure scenarios, which is not sufficient. Early termination should not be limited to Failure, but rather a controlled exit.
-
Plecenik, David
commented
Possible duplicate of https://feedback.azure.com/forums/270578-data-factory/suggestions/34301983-throw-error-activity
The comments there mention a workaround of including an activity that is guaranteed to fail, such as a lookup that does not return any value, or a lookup on a SQL server using RAISERROR to provide a failure message.
-
Joel Cochran
commented
To add to this, we need a pipeline activity to Terminate a running pipeline AND specify the reported Status [not just 'Failed'. Sometimes it isn't about failure, it's about preventing a failure. It's similar to TRY..CATCH for handling expected exceptions.
-
Anonymous
commented
please keep adding necessary features to make it suitable counterpart of ssis on cloud.
-
prashant gupta
commented
we need it
-
Anugrah Gupta
commented
Many a times I have come across that during the execution of a pipeline (such as a variable's True or False value), we want the pipeline to "Fail".
Currently, I am to achieve this using a If Condition Activity (where the variable's value is checked), followed by a Web Activity to trigger Rest API for cancelling the pipeline run.
https://docs.microsoft.com/en-us/rest/api/datafactory/pipelineruns/cancel
It would have been great if similar to "Execute Pipeline" if there can be an activity to Kill/Terminate/Cancel the pipeline's run.
-
Nick McDermaid
commented
Great minds think alike - I am doing exactly the same thing. I think I'll do a database call with RAISERROR. Nasty but I'm already working around the fact that you can't capture refreshId directly from the initial REST call. If we got raiserror capability in ADF and conditional workflow it would make all of my pipelines waaaay simpler.
