Add support for Kafka Streams from HD Insight
Azure Functions should be able to be triggered from Apache Kafka. The triggered function should be able to be configured for a specific consumer group, with options to explicitly commit the consumer's offset.
Ideally Kafka Streams could be supported, with special support for local storage of KTables.
We now have a Kafka trigger here: https://github.com/azure/azure-functions-kafka-extension.
Good progress with Event Hubs. Would be great to see Azure Functions get full access to KTable/KStreams for state, plus partition support.
Any updates? Business appreciate Kafka support.
I suggest that Kafka stream would be an advanced use case. Just being able to have functions triggered based on being a consumer in a consumer group with relevant scaling, and being able to produce messages to topics in Kafka would be a fantastic start.
Dzmitry Safarau commented
Any updates to this? Kafka support would really make us consider switching to azure functions.
Kafka Streams (KTable and KStream) is only in the Java API client right now. It provides the ability to aggregate and join data from inbound messages and store in a local data table (actually a RocksDB). The store is local to the processing pipelines (i.e. consumer group member). The resulting local writes are also replicated out as messages on separate topics (so the state is recoverable), and the whole thing is balanced using the same partition rules. See - https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
Christopher Anderson commented
We've got support for C# kafka client it looks like - https://github.com/Microsoft/CSharpClient-for-Kafka. It even has a balanced consumer implementation for us to use, like the Event Hubs client.
What's KTables? Some feature of Kafka?
That's great news. Realize that Java isn't natively supported on Azure Functions yet, so the support for Kafka Streams wouldn't apply yet. However, it would be good to think about how an elastic cluster of Kafka Stream based consumers would run on a "servless" platform.