Allow pipeline schedule to skip if already it is running (ADF V2)
Please add a feature to skip the schedule if the current schedule is already running.
For example I have a pipeline schedule for every 1 minute, if the pipeline is still running, the next schedule will start which causes the overlap in pipeline execution.
Right now I'm updating some records in SQL table which takes time until then the next schedule is starting which is again updating the same records because the previous pipeline schedule execution is not completed.
adm, tech commented
I don't want to add complex code to my pipeline just to prevent execution when the same pipeline is already/still running. I can do something similar in the Windows Task Scheduler, Task Properties, Settings: "If the task is already running, the following rules apply'. I like the pipeline Concurrency setting but this option would make it even better!
demolition porg commented
Yes, definitely need this.. surprised there's not a tickbox to just allow this in the trigger. Currently we're using code to check but this has be put as an azure function in each pipeline - which is not good, it really should be an option in the trigger
Thomas Andersen commented
You can specify concurrency for tumbling window triggers, but that will queue the pipeline runs. It would be nice if you could, as this proposal suggests, skip if already running.