SQL Data Warehouse

Do you have an idea or suggestion based on your experience with SQL Data Warehouse? We would love to hear it! Please take a few minutes to submit your idea or vote up an idea submitted by another SQL Data Warehouse customer. All of the feedback you share in these forums will be monitored and reviewed by the SQL Data Warehouse engineering team. By suggesting or voting for ideas here, you will also be one of the first to know when we begin work on your feature requests and when we release the feature.

Remember that this site is only for feature suggestions and ideas! If you have technical questions please try our documentation, MSDN forum, or StackOverflow. If you need support, please open a support ticket with us.

How can we improve Microsoft Azure SQL Data Warehouse ?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. sql server 2016 polybase data lake store

    On prem Polybase should support Azure Data Lake Store and push down query to Azure Data Lake Analytics

    0 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
    • Limit the size of ORC files produced by CREATE EXTERNAL TABLE AS SELECT command

      Recently I worked with Microsoft support on an issue using CREATE TABLE AS SELECT to import data from external tables using ORC file format. Most queries completed just fine, but some ran into errors like "Java heap space". Those queries were hitting external tables backed by huge ORC files (ranged from 17GB to 33GB per file). The product team confirmed there is a design limit that makes those files unqualified. I then split the files further down to less than 12GB per file (total data volume was kept the same) and the CTAS queries then ran just fine.
      So obviously…

      2 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
      • Polybase: PARTITIONED BY functionality when creating external tables.

        I have a large number of files which are read with Hive using a partitioning scheme. PARTITIONED BY functionality, which is so commonly used in HIVE is missing from polybase.

        7 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
        • Load subset of files in Azure blob storage to SQL data warehouse

          Let's say we have following files in Azure blob storage with same file format but different number/type of data in their columns:
          fileA_1.csv
          fileA_2.csv
          fileB_1.csv
          fileB_2.csv

          If PolyBase supports defining data source to blob storage with defined file title format (for instance: WHERE file.name = 'fileA_*.csv'), we will be able to transfer various files from same location to desired external tables.

          4 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
          • Support of StorageTable as DataSource

            Please add support of Azure TableStorage as PolyBase DataSource.
            To be honest I can't believe its not already available. :O

            17 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
            • Enable SQL DW as data source for external tables in SQL DB

              Querying SQL DW from SQL DB via external tables currently does not work due to an implicit SET LANGUAGE statement issued by the elastic queries component. It sets the Language to N'us_english' (not supported) vs. us_english (supported I guess). Same for SET NUMERIC_ROUNDABORT, set to OFF (not supported). While you can explicitly set the NUMERIC_ROUNDABORT to ON at run-time, you cannot do SET LANGUAGE us_english on SQL DB. Maybe other issuesd are to be found but this scenario is important.

              18 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
              • Predicate pushdown for ADLS sources

                As Azure SQL DW now also supports ADLS, it would be great if we could also leverage the distributed processing capabilities of ADLS to push computation down to ADLS in a similar way as it works for HDFS/Hadoop already

                7 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                • allow polybase export

                  support

                  EXEC sp_configure 'allow polybase export', '1';

                  3 votes
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                  • Polybase to treat \N as NULL

                    I was trying to load external file into SQL DW table using polybase. This external file has \N as value for null columns. For example :

                    2,"C1008",1,\N,1,1,"2016-02-09 23:13:02","2014-02-21 20:11:36","2016-02-08 17:48:59","2008-07-16 14:17:40",947

                    I defined external data format as below:

                    CREATE EXTERNAL FILE FORMAT TextFileFormat
                    WITH
                    ( FORMAT_TYPE = DELIMITEDTEXT
                    , FORMAT_OPTIONS ( FIELD_TERMINATOR = ','
                    , STRING_DELIMITER = ''
                    , USE_TYPE_DEFAULT = FALSE
                    )
                    );

                    when i try to load this record into DW table, I am getting below error:

                    Query aborted-- the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of…

                    10 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                    • Enable AD authentication for Polybase

                      Currently SAS is the only option for Polybase to access blob storage or datawarehouse.

                      6 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                      • Polybase old dates into ORC files fails

                        The following CETAS statement fails. Apparently there's an issue with old dates and ORC files? Please fix.

                        CREATE EXTERNAL FILE FORMAT ORC_Snappy
                        WITH (
                        FORMAT_TYPE = ORC
                        , DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
                        );

                        create EXTERNAL TABLE dbo.testexternaltable WITH (DATA_SOURCE = myds, LOCATION = '/testexternaltable', FILE_FORMAT = ORC_Snappy) AS
                        select cast('1910-08-13' as date) as dt;

                        2 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                        • Knox & Polybase

                          The Apache Knox Gateway (“Knox”) provides perimeter security so that the enterprise can confidently extend Hadoop access to more of those new users while also maintaining compliance with enterprise security policies.

                          More and more enterprises are using it as the only way to access to the cluster data.

                          The security is more than never a point of attention so it would be great if Polybase could connect to a Hadoop cluster using the Knox Gateway.

                          4 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                          • Support Append Blob

                            Support for Azure storage Append Blob with PolyBase External tables

                            9 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              4 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                            • PolyBase: Support newline/carriage return in delimited text

                              Polybase fails to read rows if a [delimited] text column contains a newline/carriage return character even if they are within a text delimiters

                              take this example: I write a line to a external table using CETAS and cannot read the very same table again:

                              CREATE EXTERNAL TABLE stage.testNewLine WITH
                              (
                              LOCATION = 'archive/NewLine',
                              DATA_SOURCE = myDatasource,
                              FILE_FORMAT = myFormat
                              )
                              AS
                              SELECT
                              'regular Text' AS Col1,
                              'Line1
                              Line2' AS Col2

                              SELECT *
                              FROM stage.testNewLine

                              Everything that is written to SQL DW using INSERT should be able to be retrieved again using SELECT

                              28 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                1 comment  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                              • Polybase - Give the ability to specify a file name when outputting external tables and consolidate into a single file

                                Creating an external table will result in multiple files being generated in blob storage. It would be great if the multiple files could be consolidated into a single file and be able to configure the file name.
                                There are multiple files that contain 0 bytes. It would be better to have a single file and be able to specify the file name.
                                This would help with idempotency to delete the file if I wanted to re-create the external table.

                                4 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                • 7 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Polybase support for other cloud data sources

                                    we currently has Polybase to Azure Blob Store.

                                    we need Polybase to the following data stores:

                                    HDInsight with push down compute?
                                    SQL Azure with push down compute?
                                    DocumentDB with push down compute?
                                    Azure Data Lake Store with push down to Azure Data Lake Analytics?

                                    13 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                    • PolyBase:allow encoding in file format and polybase will take care of encoding.

                                      PolyBase: Allow encoding in file format and polybase will take care of encoding.

                                      4 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Hive metastore integration for polybase / Azure SQL DW

                                        **Hive metastore integration for polybase / Azure SQL DW**
                                        I want to be able to seamless access/ import /join on tables that are already captured in my common hive metastore. This would significantly help integrating DW in our big data infrastructure and eliminate the huge duplicated maintenance effort of keeping external table definitions in sync.

                                        8 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Function that returns the name of the current file in Polybase load

                                          Selecting from a Hive external table there's a virtual column called INPUT__FILE__NAME which gives you the name of the file where the current row came from.

                                          I wish that Polybase had a similar feature so that if we are loading a directory of several files we can read the filename and parse it to derive a new column. For example, we might have a directory with 20160504.csv and 20160505.csv and the contents of the file don't indicate the date so we need to use the filename and add it as a column to the downstream table.

                                          16 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            2 comments  ·  Polybase  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1
                                          • Don't see your idea?

                                          Feedback and Knowledge Base