SQL Data Warehouse

Do you have an idea or suggestion based on your experience with SQL Data Warehouse? We would love to hear it! Please take a few minutes to submit your idea or vote up an idea submitted by another SQL Data Warehouse customer. All of the feedback you share in these forums will be monitored and reviewed by the SQL Data Warehouse engineering team. By suggesting or voting for ideas here, you will also be one of the first to know when we begin work on your feature requests and when we release the feature.

Remember that this site is only for feature suggestions and ideas! If you have technical questions please try our documentation, MSDN forum, or StackOverflow. If you need support, please open a support ticket with us.

How can we improve Microsoft Azure SQL Data Warehouse ?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. PolyBase: allow escaping string delimiters within string fields

    If you want to quote the string columns, and if one of the quoted columns has a quote character in it, the proper way to encode this is to double the quote character. (Build a CSV file in Excel and you will see that's how it saves it. I would say Excel is the authority on CSV formats.)

    it appears it's impossible for Polybase to properly load that file. For example the following format is not interpreted right with STRING_DELIMITER = '"' since it divides the value 2.5" into two columns

    SizeDescription|SizeCode
    "3.5 feet"|"3.5 ft"
    "2.5 inches"|"2.5"""

    Please improve Polybase…

    201 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    started  ·  19 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  2. Polybase: allow field/row terminators within string fields

    If a line end is present within a text field (qualified) importing data using Polybase will fail.

    Loading this line will work fine:
    1;2.1;"zzzz"

    Loading this will fail:
    1;2.1;"zz
    zz"

    Code:

    CREATE EXTERNAL FILE FORMAT text_file_format_raw
    WITH
    (
    FORMAT_TYPE = DELIMITEDTEXT,
    FORMAT_OPTIONS
    (
    FIELD_TERMINATOR =';',
    STRING_DELIMITER = '0x22',
    USE_TYPE_DEFAULT = TRUE
    )
    );

    CREATE EXTERNAL TABLE dbo.test_external
    (
    col1 INT,
    col2 DECIMAL(2,1),
    col3 NVARCHAR(5)
    )
    WITH
    (
    LOCATION = 'test_failing.txt',
    DATA_SOURCE = azure_storage,
    FILE_FORMAT = text_file_format_raw
    );

    CREATE TABLE dbo.test
    WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = ROUND_ROBIN)
    AS
    SELECT * FROM dbo.test_external;

    179 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    started  ·  18 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  3. Expose File Properties as additional columns in External Tables

    It would be helpful if some of the properties of a BLOB file could be exposed in the external table such as: Last Modified, Filename, Size. It should be optional and could be defined in the EXTERNAL FILE FORMAT definition. Each Property should be appended to the end of the result set of an external table in the order given in the definition.

    72 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  4. More flexible date formats in Polybase

    Polybase currently has very limited date format support. All of the acceptable date formats have dashes or slashes. For example, I can't define a date format of yyyyMMdd since it doesn't have slashes or dashes. Please allow any date format I can express.

    Also, please allow more than one date format per external table. For example, what if I have a date column as yyyyMMdd and then a different column as yyyyMMddHHmm. With only being able to define one date format in the CREATE EXTERNAL FILE FORMAT statement I think I can't handle both formats.

    72 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    8 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  5. PolyBase: Support newline/carriage return in delimited text

    Polybase fails to read rows if a [delimited] text column contains a newline/carriage return character even if they are within a text delimiters

    take this example: I write a line to a external table using CETAS and cannot read the very same table again:

    CREATE EXTERNAL TABLE stage.testNewLine WITH
    (
    LOCATION = 'archive/NewLine',
    DATA_SOURCE = myDatasource,
    FILE_FORMAT = myFormat
    )
    AS
    SELECT
    'regular Text' AS Col1,
    'Line1
    Line2' AS Col2

    SELECT *
    FROM stage.testNewLine

    Everything that is written to SQL DW using INSERT should be able to be retrieved again using SELECT

    59 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  6. Function that returns the name of the current file in Polybase load

    Selecting from a Hive external table there's a virtual column called INPUT__FILE__NAME which gives you the name of the file where the current row came from.

    I wish that Polybase had a similar feature so that if we are loading a directory of several files we can read the filename and parse it to derive a new column. For example, we might have a directory with 20160504.csv and 20160505.csv and the contents of the file don't indicate the date so we need to use the filename and add it as a column to the downstream table.

    55 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  7. Polybase: PARTITIONED BY functionality when creating external tables.

    I have a large number of files which are read with Hive using a partitioning scheme. PARTITIONED BY functionality, which is so commonly used in HIVE is missing from polybase.

    51 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    8 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  8. Polybase to treat \N as NULL

    I was trying to load external file into SQL DW table using polybase. This external file has \N as value for null columns. For example :

    2,"C1008",1,\N,1,1,"2016-02-09 23:13:02","2014-02-21 20:11:36","2016-02-08 17:48:59","2008-07-16 14:17:40",947

    I defined external data format as below:

    CREATE EXTERNAL FILE FORMAT TextFileFormat
    WITH
    ( FORMAT_TYPE = DELIMITEDTEXT
    , FORMAT_OPTIONS ( FIELD_TERMINATOR = ','
    , STRING_DELIMITER = ''
    , USE_TYPE_DEFAULT = FALSE
    )
    );

    when i try to load this record into DW table, I am getting below error:

    Query aborted-- the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of…

    42 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  9. Import files via single COPY command from blob store.

    It shouldn't require 5+ unintelligible queries just to import a file from blob store. Redshift lets me import files (including zipped files) from S3 with a single COPY command.

    37 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  10. 36 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  11. Support for Large Objects in External Tables

    While you can read and write to blob storage with large objects like varchar(max) now you can't read them via external tables.

    E.g. If you define a field with varchar(max) in an external table you can't use it you need to use BCP or SSIS to get it into a physical table that is a heap or supported indexed table.

    Msg 102048, Level 16, State 1, Line 56
    Columns with large object types are not supported in external tables.

    33 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    17 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  12. Enable SQL DW as data source for external tables in SQL DB

    Querying SQL DW from SQL DB via external tables currently does not work due to an implicit SET LANGUAGE statement issued by the elastic queries component. It sets the Language to N'us_english' (not supported) vs. us_english (supported I guess). Same for SET NUMERIC_ROUNDABORT, set to OFF (not supported). While you can explicitly set the NUMERIC_ROUNDABORT to ON at run-time, you cannot do SET LANGUAGE us_english on SQL DB. Maybe other issuesd are to be found but this scenario is important.

    33 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  13. Polybase - Give the ability to specify a file name when outputting external tables and consolidate into a single file

    Creating an external table will result in multiple files being generated in blob storage. It would be great if the multiple files could be consolidated into a single file and be able to configure the file name.
    There are multiple files that contain 0 bytes. It would be better to have a single file and be able to specify the file name.
    This would help with idempotency to delete the file if I wanted to re-create the external table.

    25 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  14. Polybase support for other cloud data sources

    we currently has Polybase to Azure Blob Store.

    we need Polybase to the following data stores:

    HDInsight with push down compute?
    SQL Azure with push down compute?
    DocumentDB with push down compute?
    Azure Data Lake Store with push down to Azure Data Lake Analytics?

    22 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  15. support for fixed width flat files in Polybase

    Polybase currently only supports delimited flat files. It should support fixed width flat files.

    22 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  16. Support of StorageTable as DataSource

    Please add support of Azure TableStorage as PolyBase DataSource.
    To be honest I can't believe its not already available. :O

    20 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  17. Load subset of files in Azure blob storage to SQL data warehouse

    Let's say we have following files in Azure blob storage with same file format but different number/type of data in their columns:
    fileA_1.csv
    fileA_2.csv
    fileB_1.csv
    fileB_2.csv

    If PolyBase supports defining data source to blob storage with defined file title format (for instance: WHERE file.name = 'fileA_*.csv'), we will be able to transfer various files from same location to desired external tables.

    18 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  18. Predicate pushdown for ADLS sources

    As Azure SQL DW now also supports ADLS, it would be great if we could also leverage the distributed processing capabilities of ADLS to push computation down to ADLS in a similar way as it works for HDFS/Hadoop already

    18 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  19. 17 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
  20. Polybase - fix file format type default for decimals

    Currently it's not possible to load a file with empty decimal value and convert it to 0 when loading data.

    This will load fine:
    1;2.1;"zzzz"

    This will fail:
    1;;"zzzz"

    According to the documentation( https://msdn.microsoft.com/en-us/library/dn935026.aspx) when USE_TYPE_DEFAULT = TRUE is set in the file format definition empty value should be converted to 0. While it works for Integers, it doesn't work for decimals. USE_TYPE_DEFAULT = FALSE will work fine for decimals (but of course the value will be loaded as NULL).

    Code:
    CREATE EXTERNAL FILE FORMAT text_file_format_raw
    WITH
    (
    FORMAT_TYPE = DELIMITEDTEXT,
    FORMAT_OPTIONS
    (
    FIELD_TERMINATOR =';',
    STRING_DELIMITER = '0x22',
    USE_TYPE_DEFAULT…

    15 votes
    Sign in
    (thinking…)
    Sign in with: oidc
    Signed in as (Sign out)

    We’ll send you updates on this idea

    3 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1 3
  • Don't see your idea?

SQL Data Warehouse

Feedback and Knowledge Base