SQL Data Warehouse

Do you have an idea or suggestion based on your experience with SQL Data Warehouse? We would love to hear it! Please take a few minutes to submit your idea or vote up an idea submitted by another SQL Data Warehouse customer. All of the feedback you share in these forums will be monitored and reviewed by the SQL Data Warehouse engineering team. By suggesting or voting for ideas here, you will also be one of the first to know when we begin work on your feature requests and when we release the feature.

Remember that this site is only for feature suggestions and ideas! If you have technical questions please try our documentation, MSDN forum, or StackOverflow. If you need support, please open a support ticket with us.

How can we improve Microsoft Azure SQL Data Warehouse ?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. PolyBase: allow escaping string delimiters within string fields

    If you want to quote the string columns, and if one of the quoted columns has a quote character in it, the proper way to encode this is to double the quote character. (Build a CSV file in Excel and you will see that's how it saves it. I would say Excel is the authority on CSV formats.)

    it appears it's impossible for Polybase to properly load that file. For example the following format is not interpreted right with STRING_DELIMITER = '"' since it divides the value 2.5" into two columns

    SizeDescription|SizeCode
    "3.5 feet"|"3.5 ft"
    "2.5 inches"|"2.5"""

    Please improve Polybase…

    147 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      11 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
    • Polybase: allow field/row terminators within string fields

      If a line end is present within a text field (qualified) importing data using Polybase will fail.

      Loading this line will work fine:
      1;2.1;"zzzz"

      Loading this will fail:
      1;2.1;"zz
      zz"

      Code:

      CREATE EXTERNAL FILE FORMAT text_file_format_raw
      WITH
      (
      FORMAT_TYPE = DELIMITEDTEXT,
      FORMAT_OPTIONS
      (
      FIELD_TERMINATOR =';',
      STRING_DELIMITER = '0x22',
      USE_TYPE_DEFAULT = TRUE
      )
      );

      CREATE EXTERNAL TABLE dbo.test_external
      (
      col1 INT,
      col2 DECIMAL(2,1),
      col3 NVARCHAR(5)
      )
      WITH
      (
      LOCATION = 'test_failing.txt',
      DATA_SOURCE = azure_storage,
      FILE_FORMAT = text_file_format_raw
      );

      CREATE TABLE dbo.test
      WITH (CLUSTERED COLUMNSTORE INDEX, DISTRIBUTION = ROUND_ROBIN)
      AS
      SELECT * FROM dbo.test_external;

      134 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        11 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
      • More flexible date formats in Polybase

        Polybase currently has very limited date format support. All of the acceptable date formats have dashes or slashes. For example, I can't define a date format of yyyyMMdd since it doesn't have slashes or dashes. Please allow any date format I can express.

        Also, please allow more than one date format per external table. For example, what if I have a date column as yyyyMMdd and then a different column as yyyyMMddHHmm. With only being able to define one date format in the CREATE EXTERNAL FILE FORMAT statement I think I can't handle both formats.

        56 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          8 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
        • PolyBase: Support newline/carriage return in delimited text

          Polybase fails to read rows if a [delimited] text column contains a newline/carriage return character even if they are within a text delimiters

          take this example: I write a line to a external table using CETAS and cannot read the very same table again:

          CREATE EXTERNAL TABLE stage.testNewLine WITH
          (
          LOCATION = 'archive/NewLine',
          DATA_SOURCE = myDatasource,
          FILE_FORMAT = myFormat
          )
          AS
          SELECT
          'regular Text' AS Col1,
          'Line1
          Line2' AS Col2

          SELECT *
          FROM stage.testNewLine

          Everything that is written to SQL DW using INSERT should be able to be retrieved again using SELECT

          45 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
          • Function that returns the name of the current file in Polybase load

            Selecting from a Hive external table there's a virtual column called INPUT__FILE__NAME which gives you the name of the file where the current row came from.

            I wish that Polybase had a similar feature so that if we are loading a directory of several files we can read the filename and parse it to derive a new column. For example, we might have a directory with 20160504.csv and 20160505.csv and the contents of the file don't indicate the date so we need to use the filename and add it as a column to the downstream table.

            42 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              3 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
            • Expose File Properties as additional columns in External Tables

              It would be helpful if some of the properties of a BLOB file could be exposed in the external table such as: Last Modified, Filename, Size. It should be optional and could be defined in the EXTERNAL FILE FORMAT definition. Each Property should be appended to the end of the result set of an external table in the order given in the definition.

              38 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                3 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
              • Import files via single COPY command from blob store.

                It shouldn't require 5+ unintelligible queries just to import a file from blob store. Redshift lets me import files (including zipped files) from S3 with a single COPY command.

                30 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                • Polybase to treat \N as NULL

                  I was trying to load external file into SQL DW table using polybase. This external file has \N as value for null columns. For example :

                  2,"C1008",1,\N,1,1,"2016-02-09 23:13:02","2014-02-21 20:11:36","2016-02-08 17:48:59","2008-07-16 14:17:40",947

                  I defined external data format as below:

                  CREATE EXTERNAL FILE FORMAT TextFileFormat
                  WITH
                  ( FORMAT_TYPE = DELIMITEDTEXT
                  , FORMAT_OPTIONS ( FIELD_TERMINATOR = ','
                  , STRING_DELIMITER = ''
                  , USE_TYPE_DEFAULT = FALSE
                  )
                  );

                  when i try to load this record into DW table, I am getting below error:

                  Query aborted-- the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of…

                  26 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                  • Support for Large Objects in External Tables

                    While you can read and write to blob storage with large objects like varchar(max) now you can't read them via external tables.

                    E.g. If you define a field with varchar(max) in an external table you can't use it you need to use BCP or SSIS to get it into a physical table that is a heap or supported indexed table.

                    Msg 102048, Level 16, State 1, Line 56
                    Columns with large object types are not supported in external tables.

                    24 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      15 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                    • Polybase: PARTITIONED BY functionality when creating external tables.

                      I have a large number of files which are read with Hive using a partitioning scheme. PARTITIONED BY functionality, which is so commonly used in HIVE is missing from polybase.

                      22 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                      • Enable SQL DW as data source for external tables in SQL DB

                        Querying SQL DW from SQL DB via external tables currently does not work due to an implicit SET LANGUAGE statement issued by the elastic queries component. It sets the Language to N'us_english' (not supported) vs. us_english (supported I guess). Same for SET NUMERIC_ROUNDABORT, set to OFF (not supported). While you can explicitly set the NUMERIC_ROUNDABORT to ON at run-time, you cannot do SET LANGUAGE us_english on SQL DB. Maybe other issuesd are to be found but this scenario is important.

                        22 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                        • 20 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                          • Support of StorageTable as DataSource

                            Please add support of Azure TableStorage as PolyBase DataSource.
                            To be honest I can't believe its not already available. :O

                            18 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                            • Polybase support for other cloud data sources

                              we currently has Polybase to Azure Blob Store.

                              we need Polybase to the following data stores:

                              HDInsight with push down compute?
                              SQL Azure with push down compute?
                              DocumentDB with push down compute?
                              Azure Data Lake Store with push down to Azure Data Lake Analytics?

                              18 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                              • support for fixed width flat files in Polybase

                                Polybase currently only supports delimited flat files. It should support fixed width flat files.

                                17 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                • Predicate pushdown for ADLS sources

                                  As Azure SQL DW now also supports ADLS, it would be great if we could also leverage the distributed processing capabilities of ADLS to push computation down to ADLS in a similar way as it works for HDFS/Hadoop already

                                  14 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Load subset of files in Azure blob storage to SQL data warehouse

                                    Let's say we have following files in Azure blob storage with same file format but different number/type of data in their columns:
                                    fileA_1.csv
                                    fileA_2.csv
                                    fileB_1.csv
                                    fileB_2.csv

                                    If PolyBase supports defining data source to blob storage with defined file title format (for instance: WHERE file.name = 'fileA_*.csv'), we will be able to transfer various files from same location to desired external tables.

                                    13 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                    • 13 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Polybase - fix file format type default for decimals

                                        Currently it's not possible to load a file with empty decimal value and convert it to 0 when loading data.

                                        This will load fine:
                                        1;2.1;"zzzz"

                                        This will fail:
                                        1;;"zzzz"

                                        According to the documentation( https://msdn.microsoft.com/en-us/library/dn935026.aspx) when USE_TYPE_DEFAULT = TRUE is set in the file format definition empty value should be converted to 0. While it works for Integers, it doesn't work for decimals. USE_TYPE_DEFAULT = FALSE will work fine for decimals (but of course the value will be loaded as NULL).

                                        Code:
                                        CREATE EXTERNAL FILE FORMAT text_file_format_raw
                                        WITH
                                        (
                                        FORMAT_TYPE = DELIMITEDTEXT,
                                        FORMAT_OPTIONS
                                        (
                                        FIELD_TERMINATOR =';',
                                        STRING_DELIMITER = '0x22',
                                        USE_TYPE_DEFAULT…

                                        13 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Support Append Blob

                                          Support for Azure storage Append Blob with PolyBase External tables

                                          12 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1
                                          • Don't see your idea?

                                          SQL Data Warehouse

                                          Feedback and Knowledge Base