SQL Data Warehouse

Do you have an idea or suggestion based on your experience with SQL Data Warehouse? We would love to hear it! Please take a few minutes to submit your idea or vote up an idea submitted by another SQL Data Warehouse customer. All of the feedback you share in these forums will be monitored and reviewed by the SQL Data Warehouse engineering team. By suggesting or voting for ideas here, you will also be one of the first to know when we begin work on your feature requests and when we release the feature.

Remember that this site is only for feature suggestions and ideas! If you have technical questions please try our documentation, MSDN forum, or StackOverflow. If you need support, please open a support ticket with us.

How can we improve Microsoft Azure SQL Data Warehouse ?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. 4 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
    • Limit the size of ORC files produced by CREATE EXTERNAL TABLE AS SELECT command

      Recently I worked with Microsoft support on an issue using CREATE TABLE AS SELECT to import data from external tables using ORC file format. Most queries completed just fine, but some ran into errors like "Java heap space". Those queries were hitting external tables backed by huge ORC files (ranged from 17GB to 33GB per file). The product team confirmed there is a design limit that makes those files unqualified. I then split the files further down to less than 12GB per file (total data volume was kept the same) and the CTAS queries then ran just fine.
      So obviously…

      2 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
      • Add Polybase Rejected Rows DMV

        The REJECT_TYPE and REJECT_VALUE settings on a Polybase External Table are very useful for managing invalid data types, but they are very limited if you're using Polybase as a data loading mechanism. In order for this to be useful, we need visibility of which rows were rejected, even if it was within the tolerance.

        I would suggest a generic system view that logged the failed row as an untyped string (or JSON/XML string if in the SQL engine rather than the Azure SQL Datawarehouse engine).

        This would read something like:

        reject_id, object_id, query_date, reject_string

        The external table would have config…

        3 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
        • Polybase: PARTITIONED BY functionality when creating external tables.

          I have a large number of files which are read with Hive using a partitioning scheme. PARTITIONED BY functionality, which is so commonly used in HIVE is missing from polybase.

          11 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
          • Load subset of files in Azure blob storage to SQL data warehouse

            Let's say we have following files in Azure blob storage with same file format but different number/type of data in their columns:
            fileA_1.csv
            fileA_2.csv
            fileB_1.csv
            fileB_2.csv

            If PolyBase supports defining data source to blob storage with defined file title format (for instance: WHERE file.name = 'fileA_*.csv'), we will be able to transfer various files from same location to desired external tables.

            6 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
            • Support of StorageTable as DataSource

              Please add support of Azure TableStorage as PolyBase DataSource.
              To be honest I can't believe its not already available. :O

              17 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
              • Enable SQL DW as data source for external tables in SQL DB

                Querying SQL DW from SQL DB via external tables currently does not work due to an implicit SET LANGUAGE statement issued by the elastic queries component. It sets the Language to N'us_english' (not supported) vs. us_english (supported I guess). Same for SET NUMERIC_ROUNDABORT, set to OFF (not supported). While you can explicitly set the NUMERIC_ROUNDABORT to ON at run-time, you cannot do SET LANGUAGE us_english on SQL DB. Maybe other issuesd are to be found but this scenario is important.

                18 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                • Predicate pushdown for ADLS sources

                  As Azure SQL DW now also supports ADLS, it would be great if we could also leverage the distributed processing capabilities of ADLS to push computation down to ADLS in a similar way as it works for HDFS/Hadoop already

                  8 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                  • allow polybase export

                    support

                    EXEC sp_configure 'allow polybase export', '1';

                    3 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                    • Polybase to treat \N as NULL

                      I was trying to load external file into SQL DW table using polybase. This external file has \N as value for null columns. For example :

                      2,"C1008",1,\N,1,1,"2016-02-09 23:13:02","2014-02-21 20:11:36","2016-02-08 17:48:59","2008-07-16 14:17:40",947

                      I defined external data format as below:

                      CREATE EXTERNAL FILE FORMAT TextFileFormat
                      WITH
                      ( FORMAT_TYPE = DELIMITEDTEXT
                      , FORMAT_OPTIONS ( FIELD_TERMINATOR = ','
                      , STRING_DELIMITER = ''
                      , USE_TYPE_DEFAULT = FALSE
                      )
                      );

                      when i try to load this record into DW table, I am getting below error:

                      Query aborted-- the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of…

                      13 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                      • Enable AD authentication for Polybase

                        Currently SAS is the only option for Polybase to access blob storage or datawarehouse.

                        6 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                        • Polybase old dates into ORC files fails

                          The following CETAS statement fails. Apparently there's an issue with old dates and ORC files? Please fix.

                          CREATE EXTERNAL FILE FORMAT ORC_Snappy
                          WITH (
                          FORMAT_TYPE = ORC
                          , DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
                          );

                          create EXTERNAL TABLE dbo.testexternaltable WITH (DATA_SOURCE = myds, LOCATION = '/testexternaltable', FILE_FORMAT = ORC_Snappy) AS
                          select cast('1910-08-13' as date) as dt;

                          2 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                          • Knox & Polybase

                            The Apache Knox Gateway (“Knox”) provides perimeter security so that the enterprise can confidently extend Hadoop access to more of those new users while also maintaining compliance with enterprise security policies.

                            More and more enterprises are using it as the only way to access to the cluster data.

                            The security is more than never a point of attention so it would be great if Polybase could connect to a Hadoop cluster using the Knox Gateway.

                            4 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                            • Support Append Blob

                              Support for Azure storage Append Blob with PolyBase External tables

                              9 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                              • PolyBase: Support newline/carriage return in delimited text

                                Polybase fails to read rows if a [delimited] text column contains a newline/carriage return character even if they are within a text delimiters

                                take this example: I write a line to a external table using CETAS and cannot read the very same table again:

                                CREATE EXTERNAL TABLE stage.testNewLine WITH
                                (
                                LOCATION = 'archive/NewLine',
                                DATA_SOURCE = myDatasource,
                                FILE_FORMAT = myFormat
                                )
                                AS
                                SELECT
                                'regular Text' AS Col1,
                                'Line1
                                Line2' AS Col2

                                SELECT *
                                FROM stage.testNewLine

                                Everything that is written to SQL DW using INSERT should be able to be retrieved again using SELECT

                                32 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                • Polybase - Give the ability to specify a file name when outputting external tables and consolidate into a single file

                                  Creating an external table will result in multiple files being generated in blob storage. It would be great if the multiple files could be consolidated into a single file and be able to configure the file name.
                                  There are multiple files that contain 0 bytes. It would be better to have a single file and be able to specify the file name.
                                  This would help with idempotency to delete the file if I wanted to re-create the external table.

                                  4 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                  • 7 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Polybase support for other cloud data sources

                                      we currently has Polybase to Azure Blob Store.

                                      we need Polybase to the following data stores:

                                      HDInsight with push down compute?
                                      SQL Azure with push down compute?
                                      DocumentDB with push down compute?
                                      Azure Data Lake Store with push down to Azure Data Lake Analytics?

                                      13 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                      • PolyBase:allow encoding in file format and polybase will take care of encoding.

                                        PolyBase: Allow encoding in file format and polybase will take care of encoding.

                                        4 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Hive metastore integration for polybase / Azure SQL DW

                                          **Hive metastore integration for polybase / Azure SQL DW**
                                          I want to be able to seamless access/ import /join on tables that are already captured in my common hive metastore. This would significantly help integrating DW in our big data infrastructure and eliminate the huge duplicated maintenance effort of keeping external table definitions in sync.

                                          9 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1
                                          • Don't see your idea?

                                          SQL Data Warehouse

                                          Feedback and Knowledge Base