Data Lake

You can use this set to communicate with the Azure Data Lake team. We are eager to hear your ideas, suggestions, or any other feedback that would help us improve the service to bet fit your needs.

If you have technical questions, please visit our forums.
If you are looking for tutorials and documentation, please visit http://aka.ms/AzureDataLake.

How can we improve Microsoft Azure Data Lake?

(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Tools to support to make column definition.

    It is little troublesome to define type of column when reading from file in U-SQL. It is nice to have default suggestion automatically like Excel.
    That can be an independent tool or integrated in Visual Studio also nice.

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  2. Provide a way to get full URL of Kona Job after submitting the kona job using API?

    Is there a way to get the full Kona/ADLA job url like below after submitting the job using API? I would like to know the job URL in my code after submitting the job, so that I can log the full url correctly.

    https://sangamkonaaccount.azuredatalakeanalytics.net/Jobs/6447bd83-de68-4095-b622-4fad4c16b461?api-version=2015-10-01-preview
    (fyi, I found this URL from Data Lake Visual studio plugging by querying my jobs)

    I am submitting the job using API:

    var response = await this.konaJobClient.Jobs.CreateAsync(this.konaResourceGroupName, this.KonaAccountName, new JobInfoBuildOrCreateParameters { Job = jobInfo }).ConfigureAwait(false);
    this.JobId = response.Job.JobId.ToString();

    3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
    under review  ·  Saveen responded

    Looking into extending the API to provide this information

  3. 3 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
  4. Move/Change Visual Studio Data Lake Tools Job View Cancel Button

    In Visual Studio. In the Cloud Explorer job view for Azure Data Lake tools can you please move or change the behaviour of the red cross cancel button? See attached.

    Currently the button resides right next to the job refresh button and does not offer a confirmation prompt.

    For long running jobs I often want to manually refresh the job graph more frequently, but have accidently clicked the cancel button! This ends up costing a lot of time and money in compute because of an imprecise click! Totally my fault, but we are all human!

    Could the "dangerous" cancel button…

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  5. Support the option to have my ADLA account span over multiple data centers

    and then execute U-SQL queries transparently local to the data center where I am accessing data.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  6. Allow failure thresholds for silent:true

    Allow to specify load fails if more than X (% or number) of rows fail to insert. bonus: The intention is to not reject all rows and give a false positive. Use a reject percentage or reject values. For example, SQL Server uses REJECT_TYPE and REJECT_VALUE.

    Why? this will save time (and cost) in a long job and help abort before the rest of the job runs.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  7. Allowing @Rowset OR @Table created in U-SQL to consume by C# function.

    @Ret1 =
    EXTRACT Datasource string
    FROM "Catalog/MySQL.EmployeeDb.txt"
    USING Extractors.Text();

    @Ret =
    SELECT USQLConsoleApp.ProcessConsole.ProcessRow(@Ret1)
    FROM(
    VALUES
    (
    1
    )) AS T(id);

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
  8. Allow vertex failure feedback

    Some of the failures do fail quite quickly while some others take such long time before getting back feedback. I can't categorically state the difference between these two types of failures. Allow being able to see the cause for failed vertices that are color coded with orange while script is still running.

    2 votes
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  9. Support syntactic sugar for creating simple parameterized views

    DROP VIEW IF EXISTS SearchLogView;
    CREATE VIEW SearchLogView(@datafile string) AS
    EXTRACT
    UserId int,
    Start DateTime,
    Region string,
    Query string,
    Duration int?,
    Urls string,
    ClickedUrls string
    FROM @datafile
    USING Extractors.Tsv();

    Which would be functionally equivalent (but shorter to write) to:

    DROP FUNCTION IF EXISTS SearchLogView;
    CREATE FUNCTION SearchLogView(@datafile string)
    RETURNS @searchlog
    AS BEGIN
    @searchlog =
    EXTRACT
    UserId int,
    Start DateTime,
    Region string,
    Query string,
    Duration int?,
    Urls string,
    ClickedUrls string
    FROM @datafile
    USING Extractors.Tsv();
    END;

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
  10. Add an ORDERED syntax that enforces an ordering of predicates

    DECLARE @outpref string = "/output/<replace_this_with_your_output_name>";
    DECLARE @out1 string = @outpref+"_agg.csv";
    DECLARE @out2 string = @outpref+"_top5agg.csv";

    @searchlog =
    EXTRACT UserId int,
    Start DateTime,
    Region string,
    Query string,
    Duration int?,
    Urls string,
    ClickedUrls string
    FROM "/Samples/Data/SearchLog.tsv"
    USING Extractors.Tsv();

    // enforce the ordering of the following block in cases where the order is important
    // this may be the case if for some reason the optimizer has a bug and this would be a
    // workaround or where semantically the meaning is not equivalent to reorder
    ORDERED {
    @rs1 =
    SELECT
    Region,
    SUM(Duration) AS TotalDuration
    FROM @searchlog
    GROUP BY Region;

    @res =
    SELECT…

    1 vote
    Sign in
    (thinking…)
    Sign in with: Microsoft
    Signed in as (Sign out)

    We’ll send you updates on this idea

    under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
1 2 4 Next →
  • Don't see your idea?

Feedback and Knowledge Base