Azure Synapse Analytics

We would love to hear your ideas for new features for Azure Synapse Analytics. Below, enter a new idea or upvote an existing one. The Synapse engineering team pays attention to all requests.

If instead you need a technical question answered or help, try the these options: DocumentationMSDN forum, and StackOverflow. If you need support, please open a support ticket.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. Knox & Polybase

    The Apache Knox Gateway (“Knox”) provides perimeter security so that the enterprise can confidently extend Hadoop access to more of those new users while also maintaining compliance with enterprise security policies.

    More and more enterprises are using it as the only way to access to the cluster data.

    The security is more than never a point of attention so it would be great if Polybase could connect to a Hadoop cluster using the Knox Gateway.

    6 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  2. polybase: allow calculated columns on external tables

    With the help of calculated columns it would be possible to create a hash column over all columns to check via pushdown if a record has changed on the external table compared to a local table. This would speed up the development of delta loads.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  3. Option to disable polybase validation on external table creation scripts

    When External Tables creation scripts are executed without existing files to back them, the scripts fail. Provide an option to disable validation for automated deployments where the data does not yet exist however the schema and external tables should be created.

    Creation of objects should not have any data dependencies.

    Not having External Tables deployed causes subsequent script dependencies (views, etc) to fail.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  4. ORC file format should distinguish blank and null

    ORC files do not seem to be able to distinguish blank strings and nulls.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  5. Limit the size of ORC files produced by CREATE EXTERNAL TABLE AS SELECT command

    Recently I worked with Microsoft support on an issue using CREATE TABLE AS SELECT to import data from external tables using ORC file format. Most queries completed just fine, but some ran into errors like "Java heap space". Those queries were hitting external tables backed by huge ORC files (ranged from 17GB to 33GB per file). The product team confirmed there is a design limit that makes those files unqualified. I then split the files further down to less than 12GB per file (total data volume was kept the same) and the CTAS queries then ran just fine.
    So obviously…

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  6. Polybase: support for external table with spatial data (Oracle, sdo_geometry)

    Needs support for accessing spatial data on an Oracle Instance. Oracle datatype sdo_geometry. Know there isn't a direct mapping to a sql datatype for it. Now using Linked Server with OPENQUERY to extract the needed data, but we prefer to migrate this solution to POLYBASE on SQL 2019 on-premise

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  7. Add support for SAS when connecting to Azure Storage

    Currently, you must provide the "root" Azure Storage Key when creating the database-scoped credential. As such, it is impossible to grant restricted access, particularly to specific Containers.

    This is supported in Azure SQL Database, and I was surprised to find this is not supported in Azure SQL Data Warehouse.

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  8. Support Azure Key Vault to store the SCOPED CREDENTIAL for Polybase

    I want to use Azure Key Vault to store the secret of my Scoped Credential that I use for Polybase. This makes is easier to store and changes secrets/passwords on one place only.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  9. Read {parquet} files with over 1024 fields

    We have parquet files with over 1024 fields, we want to ingest a subset of the fields into SQL DW. It appears we must define the whole parquet file and then the SQL limitation of 1024 columns kicks in.

    Ideas welcome. Thanks

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  10. Polybase Progress Monitoring

    Please implement DMVs that enable the progress of a Polybase load to be queried. Ideally, show all files being processed, and the completion state of each file. Keep the history for post-load reporting.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  11. Polybase: allow explicit naming of Clustered Columnstore Index in CTAS statement

    In the WITH clause of Create Table AS Select, allow the name for the Clustered Columnstore Index to be added like this:

    WITH

    (

    CLUSTERED COLUMNSTORE INDEX (CCI_myIndexName),  
    
    DISTRIBUTION = ROUND_ROBIN

    )

    Explicit names make maintenance much easier.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  12. Elastic Query - DW to DW Capability Hello Team, Currently seems Azure DW don't have support to issue cross data ware house queries.

    Hello Team, Currently seems Azure DW don't have support to issue cross data ware house queries.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  13. Top x from each blob, add blob name/identifier as a column

    A common pattern in source files is two tables with an unequal number of non-matching columns stacked on each other and an implied cross join.
    To save repeating the several columns on every row they appear once at the top.

    Eg
    Header #1
    One row of data
    Header #2 has many more columns
    1000 rows in table #2

    Ideally polybase would provide a simple method to either flatten the tables in one step on define table #1 and #2 separately but both pointing at the same blob so that they could be loaded with polybase into dw.

    Tables #2 is…

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  14. Add Silent Feature To Reject Rows

    I don't know if this is possible today, but having the rejected row output return to the client will sometimes ***** up external libraries and modules that are executing stored procedures or SQL directly. It would be nice if we could pass a silent option with the query or set it on the external table itself to not return feedback to the console, but still tie into the other Polybase reject row feature requests that output the rejected rows to a file.

    There are a few libraries people are using to execute SQL with Azure Data Warehouse that will fail…

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  15. PolyBase Distributed Output - Use PolyBase Head-Data Node and PolyBase Distributed Tables to distribute data to multiple targets.

    I would assume the distribution group would require same target type, and External Data Source to be predefined.
    CREATE EXTERNAL DISTRIBUTION GROUP ExternalDistributionGroup1 AS...
    (
    EXTERNALDATASOURCE1,EXTERNALDATASOURCE2,...(N)
    )
    CREATE EXTERNAL TABLE
    WITH DISTRIBUTION GROUP= ExternalDistributionGroup1 , DISTRIBUTION METHOD=HASH(DistributionKey), ETC.

    PolyBase supports Head+(n) Data Nodes... let's leverage Distributed technology in PolyBase, not just APS/ADW

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  16. Polybase - Match Parquet columns on name, not position

    We have a file schema that changes over time. Today, Polybase expects files to have the exact number of columns and will error on any mismatch.

    Polybase loads columns based on the order in the file. As Parquet has column names defined, it would be better if there was the option to match on name, allowing more fluid schema changes.

    Additional columns could therefore be ignored and any removed columns could insert NULL.

    Today if we were to add and remove a column, we would risk importing that into the wrong column.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  17. polybase created multiple folders with different timestamp but same files for rejected records(error rows)

    I have like 20 million rows in external table and there is rejected rows location specified. Why multiple datetime folder under rejectedrow get created when we do CTAS on this table which takes aroung 4 mins to execute. and these rejected rows folder contain same i.e, duplicate rejected records in all folders. Is this a bug . we need only one folder created.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  18. Why polybase external table is creating a new rejected folder when "select" statement is executed?

    Hi, we are facing a challange while making a "select" statement from external table. the problem is, it creates a "new" rejection folder with timestamp always under "RejectedRows" folder . why?

    We only required the folder is required while 1st table creation not everytime we read the data. Is this a Bug?

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  19. Enable SQLDB As Source For Elastic query (External Tables) in SQLDW

    I would like to be able to run DML queries and execute stored procedures against a SQLDB source that is on the same server as my SQLDW. The use case for this is to create a central ETL logging and control table structure.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  20. problem loading DATE data type with old dates in Spark 2.x Parquet format

    Spark 2.x uses a crazy hybrid Gregorian-Julian calendar for the DATE data type. When old dates in Parquet files are loaded to Synapse through Polybase or the COPY command, dates older than 1582-10-05 are wrong by a few days. And dates 0001-01-01 and 0001-01-02 cause an error in Polybase (or at least reject that row depending on your reject settings):

    2 rows rejected from external table [MyExtTable] in plan step 2 of query execution:
    Location: '/DatesRepro/dates.snappy.parquet' Column ordinal: 1, Expected data type: DATE.

    This issue has been raised as a bug (support case 120071524002114) and is unlikely to be fixed…

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  • Don't see your idea?

Feedback and Knowledge Base