Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. When trying to create a Lake Store, fix error 'Failed to register subscription'

    I tried creating a Data Lake Store,
    it turns out I dont have permissions to do that.
    The message was "Failed to register subscription"
    which implies a malfunction, and does not help me understand
    this is a permissions issue.

    1 vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)

      We’ll send you updates on this idea

      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • add Azure Data Lake diagnostic log support to Azure log integrator

      We need to collect and forward Azure Data Lake diagnostic logs to our SIEM, which is not currently supported by Azure log integrator.

      1 vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)

        We’ll send you updates on this idea

        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • U-SQL Database Project (Public Preview) needs to support Drop object

        ADLA DataLake Tools for Visual Studio - New project type U-SQL Database Project (Public Preview) needs to support Drop object commands. Currently there no way that I can find to update the definition of Procedures, Functions etc in USQL without dropping and creating the object. Which is fine, the problem is that the new Database project type throws an error - [DBProject]Drop object DDL statement not supported. So if USQL does not allow alters to Functions Views or Procedures how can we publish changes with the new project type? This basically means that this project type is useless for project…

        5 votes
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)

          We’ll send you updates on this idea

          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        • U-SQL Job when a file in Vertex causes an error display which file caused it

          If a U-SQL Job fails due to a bad value (such as a comma) in a text file row, the error details provides a numeric index of which file failed. It would be more useful to give us a direct link to the file that failed, or at least highlight the file in the Data tab that failed. Right now we have to take that numeric value and manually determine where it is in the Data tab file lists. It would save lots of time to make it much more obvious which file caused the failure.

          1 vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)

            We’ll send you updates on this idea

            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
          • U-SQL database project template - Need option to sort the scripts

            Recently, I have started using U-SQL project template and it works good. But I don't have option to sort which files should be merged to final DDL and which one next.

            I assume that whichever is create first it comes first. It would be great if you can provide an option to sort the U-SQL files before it is merged into one DDL file/statements.

            1 vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)

              We’ll send you updates on this idea

              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Create Datalake with Folder and Permissions via ARM Template

              The use-case demands to create an EventHub, a Datalake and enable the Capture feature to save incoming data from the Eventhub in the Datalake, directly.

              Currently it is not possible to enable the capture feature on a new created Datalake, because both the folder path and the permissions are not set / created, yet. Setting permissions and creating folders is not possible via ARM templates.

              We want to be able to create a Datalake via an ARM Template and set permissions to it and create the folders.

              7 votes
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)

                We’ll send you updates on this idea

                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
              • Azure Data Lake Quota

                Looking for a way to provide an Multi Level user-based quota on Azure Data Lake Storage. Is there a way to give a user some amount of storage capacity, and inform him gracefully when this Limit is reached? Once the user is way beyond a maximum Limit, to be able to decline additional create/append Operations?

                2 votes
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)

                  We’ll send you updates on this idea

                  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                • Pass Rowsets to C# Code-Behind Constructor in U-SQL

                  When using code-behind C# in U-SQL jobs, I'd like to be able to pass Rowsets as a parameter to a UDO. This might look like:

                  USING new Namespace.MyReducer("ALL", @MyRowset);

                  1 vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)

                    We’ll send you updates on this idea

                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                  • Make explicit the duplicated Key in the error when populating a SQL.MAP

                    If you're populating a SQL.MAP and by mistake you insert a duplicated key like the following:

                    new SQL.MAP<string, string>{
                    ...
                    {"Currency", "currency"},
                    ...
                    {"Currency", "old_currency"},
                    ...
                    }

                    you'll get the following build error: "...Evaluation of expression threw an exception: An item with the same key has already been added..."

                    If you've added hundred of pairs it'll be really tedious to identify where you placed the duplicated item, especially if it is not near its related item.
                    It'd be helpful to report the value of the duplicated key in the error message, so that the user doesn't waste his time to…

                    1 vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)

                      We’ll send you updates on this idea

                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                    • Create External Data Source with Connection String

                      Please make Azure DataLake able to accept a plain connection string on . Something similar to this:

                      CREATE DATA SOURCE MyAdoNetDBDataSource
                      FROM SQLSERVER
                      WITH
                      (
                      PROVIDER_STRING = "Server=mySampleDB.database.windows.net;Database=mySampleDB;User ID=myUser;Password=myPasswd;",
                      REMOTABLE_TYPES = (bool, byte, sbyte, short, ushort, int, uint, long, ulong, decimal, float, double, string, DateTime)
                      );

                      Right now, there are restrictions that PROVIDER_STRING cannot contain username/password, and CREDENTIAL parameter is required and must be made from Powershell. DataLake will be much easier to use if PROVIDER_STRING fully accepts ConnectionString credential, while the CREDENTIAL parameter is made optional.

                      6 votes
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)

                        We’ll send you updates on this idea

                        2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • Support Event Grid

                        Add event grid support to datalake as event grid provider each time that file was added/edited.

                        27 votes
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)

                          We’ll send you updates on this idea

                          7 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • EXTRACT all columns from a file

                          Would like the option to not have to define the column list in the EXTRACT statement in U-SQL and just do an EXTRACT * or something like that.

                          1 vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)

                            We’ll send you updates on this idea

                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Enahance built-in extractors to be able to handle files with different numbers of columns

                            Currently if a set of files have a different number of columns it can't be process in a U-SQL extract statement with the built-in extractors. Our use case is that we pull in delta loads form tables, and sometimes they add columns to the source tables.

                            For example all daily load files for TableA have 100 columns. Then, there is an application release to add 5 columns to TableA. We want to just be able to seamlessly pull in the additional 5 columns

                            1 vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)

                              We’ll send you updates on this idea

                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Provide scriptpath while submitting U-SQL from REST API

                              When submitting U-SQL job from REST API, there is no option to provide the path to the U-SQL script which can be from Blob or Data Lake Store similar to Data Factory or Powershell. If there is anything as such , please let me know

                              1 vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)

                                We’ll send you updates on this idea

                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Automatic ADLA table rebuild (i.e. compact) like in Hive

                                Today as inserts to an ADLA table creates more and more files we have to trigger a dedicated job to compact the table.

                                ALTER TABLE {table name}
                                REBUILD;

                                Such job can be expensive in AUs and time.
                                This is a classic housekeeping job which should be executed by the infrastructure like happens in Hive (Compactor enabled)

                                5 votes
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)

                                  We’ll send you updates on this idea

                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Support Cross Region Data Access

                                  Allow Data Lake Analytics to read and write data from Data Lake Storage accounts in other regions.

                                  Today, when we try to read a stream in an ADLS account in another region, we get an error ("File not found or access denied error').

                                  We are aware of the high network costs involved with a cross region read/write operation. However, there are certain cases where reading or writing data across Azure regions is required.

                                  23 votes
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)

                                    We’ll send you updates on this idea

                                    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Support File.Exist with dynamic path

                                    currently the File.Exist works with only scalar expression. We can not assign values (for e.g. file path) from the rowset to the variable and check that path exist or not and also check the contents of the file. For that have to write UDF. And though the ADLA is Linked with ADLS have to write additional code to connect to ADLS using u-sql UDF.

                                    1 vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)

                                      We’ll send you updates on this idea

                                      0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Support of Column.Exists feature

                                      I would like to add column to table only if it doesn't exists.

                                      Similar to this is desired :
                                      ALTER TABLE Test2 ADD COLUMN IF NOT EXISTS s string;

                                      OR

                                      !Column.Exists() Then add column.

                                      Currently this is not supported

                                      4 votes
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)

                                        We’ll send you updates on this idea

                                        1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Merge Statment in U-SQL

                                        Today only Insert and Delete is possible in U-SQL. It would be a great if Merge would exist aswell.

                                        Today we have dimensions that are pocessed in data lake with surrogate keys. These we would like to update the memberrs that already exists.

                                        5 votes
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)

                                          We’ll send you updates on this idea

                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Integration with Azure Search

                                          Ability to be able index a table or file set in Azure Data Lake Store from Azure Search. Complex Data Types included.

                                          2 votes
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)

                                            We’ll send you updates on this idea

                                            0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1 3 4 5 15 16
                                          • Don't see your idea?

                                          Feedback and Knowledge Base