Stream Analytics
Azure Stream Analytics is a fully managed real-time stream computation service hosted in Microsoft Azure, which provides highly resilient, low latency, and scalable complex event processing of streaming data. Azure Stream Analytics enables developers to easily combine streams of data with historic records or reference data to derive business insights easily and quickly.
-
Autoscale for streaming unit
At present, Stream analytics throws memory exception when there is not enough streaming units. It should automatically scale based on the streaming unit requirement.
262 votesneed-feedback ·
AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
You can autoscale ASA jobs based on pre-defined schedule or dynamically based on input load. This can be setup using Azure Automation. You can learn more about how to do this here: https://aka.ms/asaautoscale
-
Take input from CosmosDB
Currently Stream Analytics accepts data from Blob storage. Support to get data from DocumentDB should be added.
176 votes -
Stream Analytics VNet support
Stream Analytics should integrate into Virtual Network (VNet) so I can access my services seemlessly without requiring cross-internet access.
146 votesstarted ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
We are working on providing VNET support through a single tenant offering where you can create a Stream Analytics cluster with a min of 36 SUs and then run your jobs on this cluster. You will be able to privately access your resources like Event Hubs, SQL DB/DW etc using Azure Private Link.
-
Have a smaller unit size (i.e. for lab/demo environments)
Stream Analytics is great - but it's a bit much to pay for a full unit hour when you have very low processing requirements. I've got a couple demo and lab environments I'm working with now and have to keep stopping the jobs as each costs about $80/month to run.
The simplest fix for this would be to have a smaller unit size ("shared" unit?) to make it easier to get started with projects.
120 votesunder review ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
we will consider this request.
-
Stream into existing Power BI Datasets
Hey,
it would be great to have the possibility to create a dataset in PowerBi Desktop and use as output in ASA.108 votes -
Configurable Late Arrival Tolerance for IoT Edge
This is a critical feature. ASA on IoT Edge is not really usable in production without being able to change the default late arrival tolerance of 5 seconds to a larger value.
95 votes -
Call to a REST endpoint to invoke custom code
Call from my query to a REST endpoint to invoke custom code.
81 votes -
Azure Stream Analytics to handle case sensitive columns (currently getting InputDeserializerError)
Azure Stream Analytics currently does not support two column with names only differing by case as input.
//input data
""shopperId"":""1234567"",
""shopperid"":"""",//ASA returns InputDeserializerError error in input
{"Source":"eventhubinput","Type":"DataError","DataErrorType":"InputDeserializerError.InvalidData","BriefMessage":"Could not deserialize the input event(s) from resource 'Partition: [0], Offset: [12894685888], SequenceNumber: [104569]' as Json. Some possible reasons: 1) Malformed events 2) Input source configured with incorrect serialization format","Message":"Input Message Id: Partition: [0], Offset: [12894685888], SequenceNumber: [104569] Error: Index was out of range. Must be non-negative and less than the size of the collection.\r\nParameter name: index","ExampleEvents
ASK to Azure Stream Analytics team
Please consider to change Azure Stream Analytics to…
72 votes -
Allow creating and configuring stream analytics from the azure CLI
It is stunningly difficult to automate a deployment of Stream Analytics from a devops scenario. ARM templates seem to either not be supported or not work very well and there is no support for the Azure CLI. The sole option is to drop down to powershell which means that our deployments now need to make use of yet another technology. Please support the Azure CLI
64 votes -
On-prem SQL as input and output
Would like to push my data from on prem SQL -> Stream analytics + ML model -> Output of ML model back to on-prem SQL table -> Host power BI on corp. By this way, any PII/HBI data can still be on-prem and other numeric data can flow via the cloud and back. If the channel from cloud to on-prem sql can be secure and easily established, it would greatly help.
63 votesunder review ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
Reopening this item since we have additional votes and heard about more comments related to this.
Few comments/workaround in the meantime:
- For PII/HBI data: ASA on IoT Edge and Azure Stack for PII – this allows pre-processing at the Edge without sending data to the cloud.
- Streaming + DB can work side-by-side on ASDE (Azure SQL Database Edge) -
57 votesunplanned ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
At this time, we are not planning to add a direct connection to Elastic Search. However, as a workaround, you can output data to blob storage or SQL tables, and index them with Elastic Search. More information here: https://docs.microsoft.com/en-us/azure/search/search-semi-structured-data
Please let the feedback keep coming. We are listening!
-
Add HDInsight (HBase) to output
Adding HDInsight (HBase) as an output for an job. HBase provides random, realtime read/write access.
56 votes -
Add a simple Datetime format function
I need to insert the timestamp as a column in a Azure Table with the following .NET format ("yyyy-MM-dd-HH-mm"), I thought the CONCAT and DATENAME/DATEPART functions would help me, and I ended up with this, but it is not producing the output I need:
CONCAT(DATENAME(yyyy,System.Timestamp),'-',CAST(DATEPART(mm,System.Timestamp) AS NVARCHAR(MAX)),'-',DATENAME(dd,System.Timestamp),'-',DATENAME(HH,System.Timestamp),'-',DATENAME(mi,System.Timestamp)),
"2015-10-12T05:17:37.807Z" is formatted like: '2015-10-12-5-17" and I am expecting "2015-10-12-17-17"
56 votes -
54 votes
-
Time series database sink
It would be great to be able to stream the data into a time series database. Currently Azure does not have support for time series, but it would be nice to see support for KairosDB or OpenTSB
49 votesunder review ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
We see more and more demand for output to time series databases so reopening this item.
Let us know which database you are using. -
Add an output sync to Azure Operational Insights
Azure Operational Insights provides a very fast search and indexing mechanism to data. Its a perfect visualization experience on top of the data we are sending from EventHub->StreamAnalytics
43 votes -
Ability to output data to dynamically generated tables.
We want incomming data for multiple of our customers input through one event hub and then output to a table per customer. The tables don't need to be created by stream analytics, we can do that from code when the customer creates an account with us. I just need to be able to direct the output to the correct table from my Stream Analytics query.
41 votes -
Allow REST Endpoints as reference data input
Possibility to define REST endpoints as reference data inputs. Including refresh rate settings similar to SQL Database.
41 votes -
Enhance Security for ASA, Managed Services Identities , Key Vault Bindings
Add support for managed services identities and storing input and output configuration details and secrets in Key Vault.
38 votes -
User Defined Python Functions
Ability to define functions in python as part of a stream query similar to what has been done with Azure Machine Learning. The ability to then return the results back into the query would be great. Treat it like a SQL Stored Proc almost.
37 votesneed-feedback ·AdminAzure Stream Analytics Team on UserVoice
(Product Manager, Microsoft Azure)
responded
You can define Machine Learning UDF in your ASA jobs. With this, any of your python models built using popular frameworks like Tensorflow, scikit-learn, pytorch etc can be deployed on Azure Kubernetes and made available as a function in your job.
https://docs.microsoft.com/en-us/azure/stream-analytics/machine-learning-udf
- Don't see your idea?