Apache Beam on Azure Databricks
Apache beam is an open source batch and streaming engine with unified model that runs on any execution engine, including Spark. It has powerful semantics that elegantly solves real world challenges in both streaming and batch processing. It recently got also some Scala based abstractions on top of it, which enables succinct and correct expressiveness of windowing, triggering, out of order events and further more. It also has been chosen from some successful cloud born companies that are challenged with vast amounts of data.