Azure Synapse Analytics

We would love to hear your ideas for new features for Azure Synapse Analytics. Below, enter a new idea or upvote an existing one. The Synapse engineering team pays attention to all requests.

If instead you need a technical question answered or help, try the these options: DocumentationMSDN forum, and StackOverflow. If you need support, please open a support ticket.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. Add support for "Delta Lake" file format in Azure Data Lake Store / HDFS

    Today we can query data stored in parquet files on ADLS. It would be fantastic to extend this to support the new "Delta Lake" file format recently open-sourced by the DataBricks team ( see https://delta.io )

    This would allow us to take advantage of ACID guarantees that the delta format brings to the data lake.

    84 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    8 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  2. Polybase: support for external table with spatial data (Oracle, sdo_geometry)

    Needs support for accessing spatial data on an Oracle Instance. Oracle datatype sdo_geometry. Know there isn't a direct mapping to a sql datatype for it. Now using Linked Server with OPENQUERY to extract the needed data, but we prefer to migrate this solution to POLYBASE on SQL 2019 on-premise

    4 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  3. Support "RDBMS" TYPE for Create External Data Source

    Azure Synapse Analytics currently does not support creating external data sources to RDBMS databases such as Azure SQL, SQL Server, Orcale, etc.). However, this functionality is available in MS SQL Server 2019.

    10 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  4. problem loading DATE data type with old dates in Spark 2.x Parquet format

    Spark 2.x uses a crazy hybrid Gregorian-Julian calendar for the DATE data type. When old dates in Parquet files are loaded to Synapse through Polybase or the COPY command, dates older than 1582-10-05 are wrong by a few days. And dates 0001-01-01 and 0001-01-02 cause an error in Polybase (or at least reject that row depending on your reject settings):

    2 rows rejected from external table [MyExtTable] in plan step 2 of query execution:
    Location: '/DatesRepro/dates.snappy.parquet' Column ordinal: 1, Expected data type: DATE.

    This issue has been raised as a bug (support case 120071524002114) and is unlikely to be fixed…

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  5. Support Azure Key Vault to store the SCOPED CREDENTIAL for Polybase

    I want to use Azure Key Vault to store the secret of my Scoped Credential that I use for Polybase. This makes is easier to store and changes secrets/passwords on one place only.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  6. Polybase - Match Parquet columns on name, not position

    We have a file schema that changes over time. Today, Polybase expects files to have the exact number of columns and will error on any mismatch.

    Polybase loads columns based on the order in the file. As Parquet has column names defined, it would be better if there was the option to match on name, allowing more fluid schema changes.

    Additional columns could therefore be ignored and any removed columns could insert NULL.

    Today if we were to add and remove a column, we would risk importing that into the wrong column.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  7. add cache for On Demand Pool

    add a cache for On demand Query, I expect if I run the the same Query to hit a cache unless data source has changed

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  8. Support Avro format

    Please add support in Azure SQL DW for Avro files.

    10 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  9. polybase: allow s3 as external data source

    For building an on prem dwh it would be great, if polybase could add s3 as external data source. External tables could be used as the staging area and s3 object storage (e.g. open source minio) for the Persistent Staging Area (psa). Gzipped csv or parquet ly on s3 and could be queried on demand as a long time storage.

    8 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  10. polybase: allow calculated columns on external tables

    With the help of calculated columns it would be possible to create a hash column over all columns to check via pushdown if a record has changed on the external table compared to a local table. This would speed up the development of delta loads.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  11. Export data (simple custom export) to a single file

    Enable exporting data to a single file with custom export options: file format, max size, custom null, customer escape character, encryption, compression, etc

    9 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  12. Correct USE_TYPE_DEFAULT= FALSE rules for UTF-16 External file formats

    External file formats with UTF-16 encoding are not obeying the USETYPEDEFAULT= FALSE logic and are converting NULL marks in flat files to USETYPEDEFAULT = TRUE logic. There appears to be some kind of implicit conversion to UTF-8 causing NULL marks to no longer correctly appear.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  13. Polybase Progress Monitoring

    Please implement DMVs that enable the progress of a Polybase load to be queried. Ideally, show all files being processed, and the completion state of each file. Keep the history for post-load reporting.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  14. PolyBase: Support String Columns For Predicate Pushdown

    Predicate pushdown to Hadoop sources in SQL Server 2017 and 2019 CTP 2.2 works for date, time, and numeric data types (as the documentation at https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-pushdown-computation?view=sql-server-2017 correctly notes). It would be great if we could also push down string predicates from VARCHAR or NVARCHAR data types.

    Supporting LIKE filters would be nice, but even exact matches and support just for what's covered for numeric data types could help.

    11 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  15. Read {parquet} files with over 1024 fields

    We have parquet files with over 1024 fields, we want to ingest a subset of the fields into SQL DW. It appears we must define the whole parquet file and then the SQL limitation of 1024 columns kicks in.

    Ideas welcome. Thanks

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  16. ALTER EXTERNAL DATA SOURCE

    If a data source moves or changes then all external tables must be deleted, the data source deleted and recreated, then the external tables recreated.

    This may be experienced during a data warehouse migration, and during a disaster recovery.

    Support for ALTER EXTERNAL DATA SOURCE would make this a much more simple task.

    18 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  17. evaluate table structure for queries at table level (select count(*))

    Polybase does not evaluate datatypes for queries at table level like "select count()". When a numeric field contains alpha numeric data the record is not rejected when you use "select count()", but it is when you use "select *". It will be great when this is consistent.

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  18. Polybase: allow explicit naming of Clustered Columnstore Index in CTAS statement

    In the WITH clause of Create Table AS Select, allow the name for the Clustered Columnstore Index to be added like this:

    WITH

    (

    CLUSTERED COLUMNSTORE INDEX (CCI_myIndexName),  
    
    DISTRIBUTION = ROUND_ROBIN

    )

    Explicit names make maintenance much easier.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  19. Option to disable polybase validation on external table creation scripts

    When External Tables creation scripts are executed without existing files to back them, the scripts fail. Provide an option to disable validation for automated deployments where the data does not yet exist however the schema and external tables should be created.

    Creation of objects should not have any data dependencies.

    Not having External Tables deployed causes subsequent script dependencies (views, etc) to fail.

    5 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  20. polybase created multiple folders with different timestamp but same files for rejected records(error rows)

    I have like 20 million rows in external table and there is rejected rows location specified. Why multiple datetime folder under rejectedrow get created when we do CTAS on this table which takes aroung 4 mins to execute. and these rejected rows folder contain same i.e, duplicate rejected records in all folders. Is this a bug . we need only one folder created.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  SQL/Polybase  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3
  • Don't see your idea?

Feedback and Knowledge Base