Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Add support for ADLS Gen2 to ADLA

    Would like the ability to use the ADL Analytics job-as-a-service to work with data in ADLS Gen2.

    10 votes
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
    • Support Event Grid

      Add event grid support to datalake as event grid provider each time that file was added/edited.

      34 votes
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        7 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Create Datalake with Folder and Permissions via ARM Template

        The use-case demands to create an EventHub, a Datalake and enable the Capture feature to save incoming data from the Eventhub in the Datalake, directly.

        Currently it is not possible to enable the capture feature on a new created Datalake, because both the folder path and the permissions are not set / created, yet. Setting permissions and creating folders is not possible via ARM templates.

        We want to be able to create a Datalake via an ARM Template and set permissions to it and create the folders.

        11 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
        • U-SQL database project template - Need option to sort the scripts

          Recently, I have started using U-SQL project template and it works good. But I don't have option to sort which files should be merged to final DDL and which one next.

          I assume that whichever is create first it comes first. It would be great if you can provide an option to sort the U-SQL files before it is merged into one DDL file/statements.

          2 votes
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            2 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • EnumerateDirectory with Pattern matching & MultipleFiles download

            1. We need EnumerateDirectory with Pattern matching and capability to traverse sub-folders, to download only required files.
            2. Should have an option to download selected files by passing the names, like download('a.csv', 'b.csv')

            Really not sure, how the people are living without pattern matching on ADLS files :(

            1 vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Please fix the python reducer bug

              Script will fail when using multiple python.reducer.

              "message": "An unhandled exception from user code has been reported when invoking the method ' Constructor ' on the user type ' Extension.Python.Reducer '",
              "description": "Unhandled exception from user code: \"The file ' C: \\ Users \\ hoch \\ AppData \\ Local \\ Temp \\ 612352AA6C1B0F31 \\ tmp6AF1_932036913D6A93D5 \\ bridgeTests.py ' already exists.\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised.",

              1 vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Please fix the python reducer bug

                Script will fail when using multiple python.reducer.

                "message": "An unhandled exception from user code has been reported when invoking the method ' Constructor ' on the user type ' Extension.Python.Reducer '",
                "description": "Unhandled exception from user code: \"The file ' C: \\ Users \\ hoch \\ AppData \\ Local \\ Temp \\ 612352AA6C1B0F31 \\ tmp6AF1_932036913D6A93D5 \\ bridgeTests.py ' already exists.\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised.",

                1 vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Support .zip format in OUTPUT

                  With FeaturePreviewMode we can now create a .gz output of a csv/txt/tsv file.
                  .zip output format should be supported as well

                  1 vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Support for Download all files at Folder level

                    Currently we can download individual files. But it would be easier to downlaod all files in a directory which belongs to the same category than downloading them individually,.

                    1 vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Can we have folder -file level search for Datalakestore account

                      When Datalke store account has lot of Data, it would be easier to search the folder -subfolder and file level search with partial or full text.

                      This would prevent expanding all folders of datalakestore account.

                      1 vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Bring HDInsight / Datalake Store and other advanced data services to Azure Stack

                        Bring HDInsight / Datalake Store and other advanced data services to Azure Stack.

                        18 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                        • Allow rowset variables to be converted to scalar values in USQL

                          I'm looking to make an output path string out of dates that are in an event-level table, so that the aggregate results can be saved and partitioned into paths according to their date.

                          It appears that when I'm selecting a date from a rowset, it cannot be converted into a scalar string and used as an output path..

                          37 votes
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            6 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • SSIS Component - Azure Data Lake Store Source - Please include Text Qualifier option to load the files

                            Currently "SSIS Component - Azure Data Lake Store Source" doesn't have option of Text Qualifier so while reading the file from ADL where the values are enclosed with a Text Qualifier like " couldn't be loaded.

                            It will be great to have Text Qualifier option to load such kind of files directly from ADL store.

                            42 votes
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Pass Rowsets to C# Code-Behind Constructor in U-SQL

                              When using code-behind C# in U-SQL jobs, I'd like to be able to pass Rowsets as a parameter to a UDO. This might look like:

                              USING new Namespace.MyReducer("ALL", @MyRowset);

                              2 votes
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Snapshot on ADLS

                                HDFS has supported snapshotting since at least Hadoop 2.4.1. To become a robust platform for running reliable and reproducible Data Science operations, ADLS should have the ability to snapshot or version data to a point in time.

                                See http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html for the base implementation. Are more robust example is included with MapR https://mapr.com/resources/mapr-snapshots/

                                5 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • trigger azure functions based on files received in azure data lake store

                                  It does not seem to be possible to trigger azure functions based on files received in azure datalake store. in blob storage this works like a charm, but for ADLS this isn't possible. So my proposal is improve integration azure functions - azure data lake store: ability to trigger azure functions based on received files in azure data lake store.

                                  45 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Add Data Lake and Data Lake Analytics to either Australia or Asia Pacific.

                                    As of October 2017, the closest region is still US. There needs to something better serving the Asia Pacific regions - either one of the AU or AP regions.

                                    12 votes
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      4 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • NFS mount ADLS in Linux

                                      add driver to mount ADLS in Linux, so users can read/write/execute files in ADLS directly in Linux

                                      24 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        3 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                      • USQL catalog - support IDENTITY or SEQUENCE

                                        In USQL managed table, we need column IDENTITY property similar to T-SQL or USQL catalog level SEQUENCE object.

                                        CREATE TABLE X
                                        (
                                        ID Int64 IDENTITY(1,1)
                                        )

                                        As a workaround we can use ROW_NUMBER(). But something like IDENTITY or SEQUENCE would mean we don't have to work out the next value.

                                        18 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                        • JSON support for data analytics in azure portal

                                          data analysis for JSON data hosted in data lake store need to be processed by data analytics jobs directly and easily. Lot's of customers waiting for that. There is NO clear guidance on how to use JSON data using U-SQL query, and we might be loosing lots of business as most of the customers now a days have complex JSON data to be queried.

                                          110 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            7 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 16 17
                                          • Don't see your idea?

                                          Feedback and Knowledge Base