Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Support CustomVision Cognitive API

    Custom Vision API is currently in preview => https://customvision.ai/
    How can we use trained model I have created into a U-SQL job ?

    Currently, calling web API is not supported.

    1 vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Provide built-in support for Geospatial types with Spatial indexing

      Many IoT and big data scenarios have a geospatial component to it that require geospatial operations such as point in polygon queries. Please provide built-in support of the SQL Server Geometry and Geography types with indexing support.

      4 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Add storage retention policy

        ADLS should have storage retention policies i.e. keep only last N days of contents under a folder (based on content creation date). This is similar to storage policies available in Cosmos and other similar big data storage solutions.

        Current ADLS API for content expiration look very restrictive – can only apply at file, and that too using a separate API (network) call.

        2 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Support text streams in Data Lake

          Currently, the ADL APIs only support binary streams. It'd be nice to have text streams as well, to, for instance, have the ability to easily read/write CSVs from pandas.

          2 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
          • Tool or powershell command to list all the folder sizes recursively.

            Please enable a feature - tool or powershell command to list all the folder sizes recursively. This will help to find the Top consumer of space.

            2 votes
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Paging on ListFileStatus

              Enable a way to page through files in a directory, top, skip, orderby etc.

              This way is it is possible to get the last file in a directory when ordered by filename or creation/modification date.

              It also allows users to skip to page #5 etc, without just going backwards and forwards.

              1 vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Support auth using service account principal in Azure Data Factory (ADF) linked service

                Currently only personal OAuth user token is supported what doesn't fit real-world production scenario.

                1 vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • 1 vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • 1 vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • USQL - Allow Constants to be assigned to Columns in the JOIN Conditions

                      In Join Conditions, Both sides of the == operator should be columns names, we cannot use values as in SQL. Can we add support for this?

                      For Example – JOIN ON Eng.EngagementSID == Prj.EngagementSID AND Eng.DataSourceSID == 2 should be allowed

                      4 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Allow inheritance in USQL pacakge

                        Support inheritance in USQL package so that we can have parent/child relation in packages.

                        E.g.

                        BasePacakge : Declare @baseValue = ‘Base”
                        DerivedPackage : base.@baseValue =”newValue” ??? I want to override base package value somehow.

                        Please add support of this.

                        2 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Count Distinct within Window Function (i.e. Over)

                          Today you have SUM, AVG, COUNT but Count Distinct is not a valid Window Function and I need it for logic later on in my process.

                          1 vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Data Lake Store in UK

                            Can we have UK South or UK West Azure Data lake Store?
                            Because of Data Protection Act we are restricted from using ADLS and ADLA. If this is on roadmap can we have an estimate time frame to expect this please?

                            3 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • 1 vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Estimated row count VS Actual row count

                                It would be good to have the Estimated vs Actual rows shown in the supervertexes to troubleshoot CBO's estimation errors. For details please read the stack overflow topic http://stackoverflow.com/questions/43589518/u-sql-job-performance

                                7 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Ability to profile data in data lake

                                  One of the key advantage for leveraging the data lake is exploration capability; having data profile handy would enhance the exploration experience for the user. Moreover, the “data profile” can be integrated with data catalog to further enhance the experience.

                                  This is about the statistics of the data such as minimum, max, avg, data type, length, discrete values, uniqueness, occurrence of null values, typical string patterns etc. This is to help the user understand if the data is appropriate or are there any anomalies? The reason for having the profiling capabilities in data lake is to offload data profiling capabilities…

                                  9 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Estimated rows count vs Actual rows count on U-SQL plan's supervertexes.

                                    It would be good to have the Estimated vs Actual rows shown in the supervertexes to troubleshoot CBO's estimation errors. For details please read the stack overflow topic alexander.sukhoborov@arkadium.com

                                    1 vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Increase number of U-SQL table files/partitions that can be read in a job

                                      Today, a job raises an error (or may timeout) of the query reads too many partitions or insertion generated files of a table (or set of tables). Please increase the number in a similar way to how the number of files in a file set is being increased.

                                      7 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Output to unstructure files with file size configuration

                                        Output files based on row numbers or file size would be helpful.

                                        1 vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Deploy a folder with its structure

                                          It'd be great if one can DEPLOY a folder, with all its contents and structure as is, in a single DEPLOY statement,

                                          2 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 10 11
                                          • Don't see your idea?

                                          Feedback and Knowledge Base