Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Allow completed leaf INSERT/OUTPUTs to complete even though job fails

    When you have job running for hours and many of the steps has completed at leafs, you still don't get the ouput if other parts of the job fails with an error.

    We should have an option on the OUTPUT/INSERTS that will complete what it has received regardless of the state of other vertices.

    0 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Add option to select files at any folder level in filesets

      Add option to filesets to select files at any folder level. E.g., I have the files

      root/f/data.csv
      root/a/b/f/file.csv
      root/a/f/file1.csv
      root/c/d/file2.csv

      Then

      @data = EXTRACT ..., filename string FROM "root/{**}/f/{filename}.csv" ...

      would extract from the files:

      root/f/data.csv
      root/a/b/f/file.csv
      root/a/f/file1.csv

      but not:

      root/c/d/file2.csv

      3 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        I agree to the terms of service
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • U-SQL Copy command

        Add the ability to quickly copy a file from one location in ADLS to another location in ADLS. If the file is compressed, this should NOT invoke decompression and re-compression steps like an extractor/outputter process would require.

        1 vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          I agree to the terms of service
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • Estimated Cost Property in API

          Can you please include the Estimated Cost Property in the Azure Data Lake API
          Please see enclosed what I'm referring to.

          3 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            I agree to the terms of service
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • Remove arbitrary 50 file limit and allow uploading of folders to Azure Data Lake from Azure Portal

            Let me upload folders and remove the arbitrary 50 file limit when uploading to ADL from the portal.

            1 vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              I agree to the terms of service
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Allow file copy on Data lake with Powershell and C#.NET

              I want to do file copy on Data lake with Powershell and C#.NET

              1 vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                I agree to the terms of service
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Allow Azure Data Lake to read/write to CosmosDB

                Support reading from and writing to CosmosDB for Azure Data Lake.

                2 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  I agree to the terms of service
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Support CustomVision Cognitive API

                  Custom Vision API is currently in preview => https://customvision.ai/
                  How can we use trained model I have created into a U-SQL job ?

                  Currently, calling web API is not supported.

                  1 vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    I agree to the terms of service
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Provide built-in support for Geospatial types with Spatial indexing

                    Many IoT and big data scenarios have a geospatial component to it that require geospatial operations such as point in polygon queries. Please provide built-in support of the SQL Server Geometry and Geography types with indexing support.

                    4 votes
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      I agree to the terms of service
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Add storage retention policy

                      ADLS should have storage retention policies i.e. keep only last N days of contents under a folder (based on content creation date). This is similar to storage policies available in Cosmos and other similar big data storage solutions.

                      Current ADLS API for content expiration look very restrictive – can only apply at file, and that too using a separate API (network) call.

                      3 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        I agree to the terms of service
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Support text streams in Data Lake

                        Currently, the ADL APIs only support binary streams. It'd be nice to have text streams as well, to, for instance, have the ability to easily read/write CSVs from pandas.

                        2 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          I agree to the terms of service
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                        • Tool or powershell command to list all the folder sizes recursively.

                          Please enable a feature - tool or powershell command to list all the folder sizes recursively. This will help to find the Top consumer of space.

                          2 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            I agree to the terms of service
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Paging on ListFileStatus

                            Enable a way to page through files in a directory, top, skip, orderby etc.

                            This way is it is possible to get the last file in a directory when ordered by filename or creation/modification date.

                            It also allows users to skip to page #5 etc, without just going backwards and forwards.

                            1 vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              I agree to the terms of service
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Support auth using service account principal in Azure Data Factory (ADF) linked service

                              Currently only personal OAuth user token is supported what doesn't fit real-world production scenario.

                              1 vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                I agree to the terms of service
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • 1 vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  I agree to the terms of service
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • 1 vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    I agree to the terms of service
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • USQL - Allow Constants to be assigned to Columns in the JOIN Conditions

                                    In Join Conditions, Both sides of the == operator should be columns names, we cannot use values as in SQL. Can we add support for this?

                                    For Example – JOIN ON Eng.EngagementSID == Prj.EngagementSID AND Eng.DataSourceSID == 2 should be allowed

                                    5 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      I agree to the terms of service
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Allow inheritance in USQL pacakge

                                      Support inheritance in USQL package so that we can have parent/child relation in packages.

                                      E.g.

                                      BasePacakge : Declare @baseValue = ‘Base”
                                      DerivedPackage : base.@baseValue =”newValue” ??? I want to override base package value somehow.

                                      Please add support of this.

                                      2 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        I agree to the terms of service
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Count Distinct within Window Function (i.e. Over)

                                        Today you have SUM, AVG, COUNT but Count Distinct is not a valid Window Function and I need it for logic later on in my process.

                                        1 vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          I agree to the terms of service
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Data Lake Store in UK

                                          Can we have UK South or UK West Azure Data lake Store?
                                          Because of Data Protection Act we are restricted from using ADLS and ADLA. If this is on roadmap can we have an estimate time frame to expect this please?

                                          5 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            I agree to the terms of service
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 11 12
                                          • Don't see your idea?

                                          Feedback and Knowledge Base