Community

Loading...

Event Hubs

121 Ideas

emulator

An Azure Event Hub emulator (preferably in a container) to do local testing

Ashish Chhabria

Company Response

7/17/23: This feature is currently being developed. Public preview is planned for Q1 of 2024. 6/15/23: This feature is included in the roadmap.5/29/24: The feature just went out into public preview. You can read more about it here: https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/introducing-local-emulator-for-azure-event-hubs/ba-p/4146454

Allow more than 10 hubs per event hub namespace

Event hub namespaces have a hard limit of 10 hubs. It would be great to be able to raise that limit to 100 or 1000 or whatever via a support request.We use Azure Event Hubs with Kafka. A namespace for us is a kafka server and a hub is a kafka topic. Now we have to split topics per kafka server based on Azure's 10 hubs hard limit, which is counterintuitive and complicates our setup.

Kasun Indrasiri

Company Response

Available in Event Hubs Premium GA.

Using a filter with EventHub data

ill we be able to use a filter in the future for the EventHub data? I'm designing a new solution for a customer and it looks like EventHubs are great for sending and receiving our near-realtime data. Downside is when receiving the data, we receive all data of all our devices in the world. Most of the time, our clients only want to see data of one (or a few) device. We could of course filter the data by ourselves, client side, but this would cost a lot of bandwith.

Company Response

Filters are in our roadmap. Filters will be tied to Consumer Groups. For filtering, you can also use the built-in no-code event processing features which uses Azure Stream Analytics. Filter and ingest to Azure Synapse SQL using the Stream Analytics no code editor | Microsoft Docs

Docker image for Event Hub in local/dev/test environments

A container that acts like an event hub so that in local/dev/test/ci environments it's easy to spin up and tear down application stacks that are dependent on event hub.

Ashish Chhabria

Company Response

7/17/23: This feature is currently being developed. Public preview is planned for Q1 of 2024. 6/15/23: This is planned to be included as part of the Event Hubs Emulator feature. 11/19/24: Emulator released with Kafka protocol support (Introducing Kafka Support in Event Hubs emulator | Microsoft Community Hub ) and with AMQP protocol support (Introducing Local Emulator for Azure Event Hubs | Microsoft Community Hub )

Geo Data Replication support

When setting up Azure policy for tag inheritance, you cannot exclude secondary namespaces, even though by definition the secondary namespaces cannot be updated under any circumstances. So when you do an automated deployment the policy kicks in and deployment fails wit the error Cannot update a namespace that is secondary.The only suggested way to get around this from Microsoft is to break the pairing (10 minutes) delete secondary namespace's event hubs (2-5 minutes) and set up tags enable pairing gain (10 minutes). I am setting this as the baseline, and it turns out it will be at least 25 minutes per namespce pairing. Imagine have multiple namespaces!! Even if you automate the process it still is a downtime of 25 minutes. Which is not acceptable. So please allow tags to be updated on event hub.How to recreate this situation.Create an ARM template for EventHub with GeoRecovery (with no tags set up)Create a policy to inherit tags from Resource Group/subscription if not set (provided tags are set up for the RG/subscription)Deploy the ARM - This will succeed, as when the primary and secondary namespaces are still different stand alone namespaces, tags will be inherited on resource creation and pairing happens normally.Re-deploy the same template - This will fail, as the primary and secondary are now paired secondary cannot be updated. Even though you are deploying the same event hub definition with no tags, ARM will assume you are trying to update the tags (even on an incremental update).Now break the pairing, delete the event hubs, deploy again - this will succeed, but you have just added a lot of manual work and have gained yourself a downtime of 15-20 minutes.Now imagine a large scale enterprise having at least 50 namespaces.Possible solution -1. PoweShell to exclude scope on policy assignment - Defies the point of having an easy template on the portal, but have to be a scripting genius to get this right.2. Set up policies based on naming convention..

Ashish Chhabria

Company Response

June 2024 : Geo-replication for data was released as a public preview for Event Hubs Dedicated tier. For more details, check out the documentation at Azure Event Hubs geo-replication - Azure Event Hubs | Microsoft Learn 10/2/2023: The new date for Public preview availability is on end of Q1 of 2024. 7/17/23: Geo Data replication feature is currently available as a private preview for select customers. Public preview is planned for Q4 of 2023. 2/26/23 : Update titile from "Allow Secondary EventHub's tags to be updated via ARM/PS/Policies" to "Geo Data Replication support".We are working on revamping our data replication and Geo DR features. So this ask will be addressed with the new changes.

Remove the EventHub Message Size limit of 1MB

Currently we have a size limit on the incoming messages on an EventHub as 1MB. Some processes batch the messages before sending them, to avoid making several POST requests. Even though the batch size is small as 10, the size is unpredictable when the message is sent, as it hits the 1MB mark. Please could we remove the size limit or at least increase the limit form 1MB to 5MB/10MB?Kafka does not have a size limit. Google has a limit of 10MB.

Ashish Chhabria

Company Response

05/21/2024: Public Preview available for Event Hubs dedicated tier. Check out the documentation at Quickstart: Send and receive large messages with Azure Event Hubs (preview) - Azure Event Hubs | Microsoft Learn10/9/23: Public preview is planned for H1 2024. 7/17/23: We are currently enabling this feature as a private preview for select customers.

Provide Event Hub viewer / explorer

It would be good if some sort of viewer/explore was provided for looking at the events within an Event Hub. This could be done either via the portal or Azure Storage Explorer.When developing I currently use https://marketplace.visualstudio.com/items?itemName=Summer.azure-event-hub-explorer to view events as they are produced.However, it would be good if a tool existed that let you browse events within the retention period. With the possibility to query by timestamp & partition key. As well as the event body, it would be good to be able to see the properties and other metadata.We use Event Hub as the integration point between multiple systems and being able to diagnose issues by querying events would be extremely useful.

Completed

All Ideas

Ashish Chhabria

Company Response

3 months ago

Event Hubs explorer is now generally available on the Azure Portal. Announcing the General Availability of Event Hubs Data Explorer | Microsoft Community Hub

Data Lake Gen2 Capture Support

Currently you can capture to Data Lake Gen1 or Blob but not Data Lake Gen 2. Sad times.

Kasun Indrasiri

Company Response

This is currently supported. Please check https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-enable-through-portal#azure-data-lake-storage-gen-2.

I want to see number of outgoing messages by consumer group

I can load up graphs showing outgoing messages by the event hub, but I want to see more granularly down to outgoing messages by the consumer group.

Kasun Indrasiri

Company Response

Now you can track the consumer group level information using logs related the application group level metrics. First you need to enable Application Metrics logs and then you can use log analytics to monitor the consumer group level data consumption. Monitoring Azure Event Hubs data reference - Azure Event Hubs | Microsoft Learn

Add an automatic scale down to better handle fluctuating workloads.

The current auto-inflate feature is great to handle gradual increase in workload without downtime. For peak loads, however, it means billing is done on the highest throughput of a given period if no action is taken. This is detrimental to a number of use-cases.

Kasun Indrasiri

Company Response

This feature is currently being developed for Dedicated and Premium SKU. However that is not planned for standard SKU yet.