Allow designation of "non-critical" output, for which data can be discarded on overflow
We use our Stream Analytics jobs in our alerting, to process tens of thousands of events into possible alerts. The stream analytics jobs write to multiple outputs; some of these are critical (containing the alerts), others are non-critical (container .e.g data to produce graphs for debugging/enrichting alerts).
When the non-critical outputs fail, our job will stop working, thereby interrupting our alerting flow. However, this debug data could simply be discarded if that allows the other outputs to keep functioning.
A workaround for this would be to create 2 jobs:
- A first job for the critical output (keeping the error policy to "retry")
- A second job for non-critical output (changing the policy to "drop")