How can we improve Microsoft Azure Functions?

Add support for Kafka Streams from HD Insight

Azure Functions should be able to be triggered from Apache Kafka. The triggered function should be able to be configured for a specific consumer group, with options to explicitly commit the consumer's offset.

Ideally Kafka Streams could be supported, with special support for local storage of KTables.

23 votes
Sign in
Check!
(thinking…)
Reset
or sign in with
    Password icon
    Signed in as (Sign out)

    We’ll send you updates on this idea

    Joe Wood shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    7 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
      Password icon
      Signed in as (Sign out)
      Submitting...
      • Joe Wood commented  ·   ·  Flag as inappropriate

        Good progress with Event Hubs. Would be great to see Azure Functions get full access to KTable/KStreams for state, plus partition support.

      • Anonymous commented  ·   ·  Flag as inappropriate

        I suggest that Kafka stream would be an advanced use case. Just being able to have functions triggered based on being a consumer in a consumer group with relevant scaling, and being able to produce messages to topics in Kafka would be a fantastic start.

      • Joe Wood commented  ·   ·  Flag as inappropriate

        Kafka Streams (KTable and KStream) is only in the Java API client right now. It provides the ability to aggregate and join data from inbound messages and store in a local data table (actually a RocksDB). The store is local to the processing pipelines (i.e. consumer group member). The resulting local writes are also replicated out as messages on separate topics (so the state is recoverable), and the whole thing is balanced using the same partition rules. See - https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/

        The projection, join and aggregation functions used in the Java client API are very LINQ like. Hopefully we'll see this in the C# and JavaScript clients soon.

      • Joe Wood commented  ·   ·  Flag as inappropriate

        That's great news. Realize that Java isn't natively supported on Azure Functions yet, so the support for Kafka Streams wouldn't apply yet. However, it would be good to think about how an elastic cluster of Kafka Stream based consumers would run on a "servless" platform.

      Feedback and Knowledge Base