Batch

How can we improve Azure Batch? Do you have an idea, suggestion or feedback based on your experience? We would love to hear it!

How can we improve Azure Batch?

You've used all your votes and won't be able to post a new idea, but you can still search and comment on existing ideas.

There are two ways to get more votes:

  • When an admin closes an idea you've voted on, you'll get your votes back from that idea.
  • You can remove your votes from an open idea you support.
  • To see ideas you have already voted on, select the "My feedback" filter and select "My open ideas".
(thinking…)

Enter your idea and we'll search to see if someone has already suggested it.

If a similar idea already exists, you can support and comment on it.

If it doesn't exist, you can post your idea so others can support it.

Enter your idea and we'll search to see if someone has already suggested it.

  1. Implement local Batch emulation

    Please allow full emulation of Batch jobs to facilitate local development and testing prior to deployment.

    140 votes
    Vote
    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      You have left! (?) (thinking…)
      under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
    • Support Managed Service Identity

      We made application that uses Managed Service Identity.
      It's easy and friendly way to access Azure Key Vault that contains some secrets.

      Actually, Azure Batch is not support Managed Service Identity.
      In this situation, We have to make another application between MSI enabled environment (Azure VM, Web Apps) and disabled environment (Azure Batch).

      We don't want writing secrets in application package, environment variables and clear text configuration.
      We want strongly Managed Service Identity in Azure Batch.

      76 votes
      Vote
      Sign in
      Check!
      (thinking…)
      Reset
      or sign in with
      • facebook
      • google
        Password icon
        Signed in as (Sign out)
        You have left! (?) (thinking…)
        under review  ·  4 comments  ·  Flag idea as inappropriate…  ·  Admin →
      • Support for allocating Azure Batch nodes using containers instead of VMs

        It would be great if Azure Batch could use the upcoming Windows Containers to allocate new nodes as containers instead of VMs, with an appropriate increase in auto-scale formula evaluation frequency.

        This could potentially reduce the time from job submission to executing at full capacity from the current 5-10 minutes to well under a minute.

        These are some of the scenarios where this would benefit us:

        In scenario one a customer might use hundreds of nodes to run a large job in a few minutes, but the wait for nodes to be allocated and then become available can tripple the…

        69 votes
        Vote
        Sign in
        Check!
        (thinking…)
        Reset
        or sign in with
        • facebook
        • google
          Password icon
          Signed in as (Sign out)
          You have left! (?) (thinking…)
          1 comment  ·  Flag idea as inappropriate…  ·  Admin →
        • Add diagnostics support for Azure Batch

          Currently Azure Batch does not support diagnostics/telemetry/performance counters. It would be really useful to have something like a New relic or an App insights integration.

          50 votes
          Vote
          Sign in
          Check!
          (thinking…)
          Reset
          or sign in with
          • facebook
          • google
            Password icon
            Signed in as (Sign out)
            You have left! (?) (thinking…)
            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
          • Support on Premise Batch Nodes

            It would be great to have support for on-premise nodes to support hybrid workloads. This could help with connectivity issues for specific types of specialized data sources. Having a "Batch Agent" that could be installed on any server so it can register in a pool and be sent work would be really valuable.

            48 votes
            Vote
            Sign in
            Check!
            (thinking…)
            Reset
            or sign in with
            • facebook
            • google
              Password icon
              Signed in as (Sign out)
              You have left! (?) (thinking…)
              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
            • Per-node resource statistics (CPU usage etc.)

              Currently the API allows to fetch resource stats for a whole pool, returning averages of cpu usage etc. See https://docs.microsoft.com/en-us/rest/api/batchservice/get-information-about-a-pool

              It would be great if one could query those statistics per-node as well. This is especially useful if you run only a single task per node and want to see how well it utilizes the node (by parallelisation etc.)

              39 votes
              Vote
              Sign in
              Check!
              (thinking…)
              Reset
              or sign in with
              • facebook
              • google
                Password icon
                Signed in as (Sign out)
                You have left! (?) (thinking…)
                1 comment  ·  Flag idea as inappropriate…  ·  Admin →
              • Need reimage support on VirtualMachineConfiguration-based pool

                Hi Batch Team, we would like to have reimage support on VirtualMachineConfiguration-based pools as well. We know that currently it is supported on CloudServiceConfiguration-based pools. This would greatly reduce the effort to maintain the nodes in the pool as we want to regularly reimage the nodes to keep them healthy without deleting the pool and recreating again. Thanks so much!

                29 votes
                Vote
                Sign in
                Check!
                (thinking…)
                Reset
                or sign in with
                • facebook
                • google
                  Password icon
                  Signed in as (Sign out)
                  You have left! (?) (thinking…)
                  under review  ·  2 comments  ·  Flag idea as inappropriate…  ·  Admin →
                • Visual Studio Project - Batch Application

                  There needs to be a simpler way to deploy Batch Applications, preferably directly from Visual Studio.

                  Having a mechanism to deploy my Azure Batch programs directly to my Batch Applications with a method similar to deploying Azure Web Apps would save heaps of time.

                  27 votes
                  Vote
                  Sign in
                  Check!
                  (thinking…)
                  Reset
                  or sign in with
                  • facebook
                  • google
                    Password icon
                    Signed in as (Sign out)
                    You have left! (?) (thinking…)
                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                  • Allow initial value for TargetDedicated when autoscale is enabled for a pool.

                    Otherwise it takes at least 5 minutes to start the tasks!

                    24 votes
                    Vote
                    Sign in
                    Check!
                    (thinking…)
                    Reset
                    or sign in with
                    • facebook
                    • google
                      Password icon
                      Signed in as (Sign out)
                      You have left! (?) (thinking…)
                      under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                    • Jobs of equal priority should be executed in submission order

                      If I have a fixed number of dedicated VMs, then it is important that queued jobs run in submission order. Sure I could use the priority to achieve this, but then I can't use priority for real business use case prioritisation. Sure I could reserve priority ranges, but this is getting complicated when it should just process them in submission order.

                      20 votes
                      Vote
                      Sign in
                      Check!
                      (thinking…)
                      Reset
                      or sign in with
                      • facebook
                      • google
                        Password icon
                        Signed in as (Sign out)
                        You have left! (?) (thinking…)
                        0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                      • 18 votes
                        Vote
                        Sign in
                        Check!
                        (thinking…)
                        Reset
                        or sign in with
                        • facebook
                        • google
                          Password icon
                          Signed in as (Sign out)
                          You have left! (?) (thinking…)
                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                        • Deploy certificate from Azure Key Vault to compute node

                          Rather than having to associate a certificate with a pool, being able to associate a certificate in a Key Vault that's been created with the -EnabledForDeployment flag would help centralize management of certificates being deployed to both VM's and Batch nodes.

                          15 votes
                          Vote
                          Sign in
                          Check!
                          (thinking…)
                          Reset
                          or sign in with
                          • facebook
                          • google
                            Password icon
                            Signed in as (Sign out)
                            You have left! (?) (thinking…)
                            under review  ·  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                          • Add support for Azure Batch in Azure Stack

                            I'm working with a client that requires extensive batch processing and does not want to utilize Azure cloud at this time. We are designing a system that incorporates Azure Service Fabric (on-premises) as part of the overall data integration framework. It would be great if we could add a native batch-processing facility that utilizes Azure tooling and design, but runs in legacy infrastructure (Windows AD networks).

                            12 votes
                            Vote
                            Sign in
                            Check!
                            (thinking…)
                            Reset
                            or sign in with
                            • facebook
                            • google
                              Password icon
                              Signed in as (Sign out)
                              You have left! (?) (thinking…)
                              0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                            • Make it possible to set memory usage limitation for a batch task

                              it would be nice if we can set the maximum memory usage for a specific task.
                              This could help us protecting concurrent tasks from failing when one task is consuming a lot of memory.

                              6 votes
                              Vote
                              Sign in
                              Check!
                              (thinking…)
                              Reset
                              or sign in with
                              • facebook
                              • google
                                Password icon
                                Signed in as (Sign out)
                                You have left! (?) (thinking…)
                                0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                              • Add Job based metrics in autoscale formula to facilitate the scaling of Dedicated nodes based on the number of JobManagerTasks

                                Now autoscaling only supports task-based metrics: $ActiveTasks, $RunningTasks, $PendingTasks, etc. It would be nice to have similar metrics for jobs to be able to scale the TargetDedicatedNodes count to the number of JobManagerTasks with AllowLowPriorityNode = false.

                                6 votes
                                Vote
                                Sign in
                                Check!
                                (thinking…)
                                Reset
                                or sign in with
                                • facebook
                                • google
                                  Password icon
                                  Signed in as (Sign out)
                                  You have left! (?) (thinking…)
                                  0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                • Custom job/node statistic/metrics

                                  There's a vote for per-node resource statistics for general metrics that are always relevant - CPU, RAM usage, etc.

                                  The ability to push custom statistics, probably through some kind of node SDK, would be great too. For example, if you're doing database work, you might want the rows p/s or if it was render work, pixels p/s or ETA for that section.

                                  Ideally, these metrics would also be available for the entire job. In the first example above, the entire job would show the sum of all rows p/s to get the total rows p/s. For ETA, it would be…

                                  6 votes
                                  Vote
                                  Sign in
                                  Check!
                                  (thinking…)
                                  Reset
                                  or sign in with
                                  • facebook
                                  • google
                                    Password icon
                                    Signed in as (Sign out)
                                    You have left! (?) (thinking…)
                                    1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                  • Provide a way to increase default quotas at the subscription level

                                    It would be nice if we could increase the Max cores, max jobs etc at a subscription level so that each new batch account we create will have those settings.

                                    I understand you need to know how many cores we can use at any given time but you also have another quota for maximum number of batch accounts we can have.

                                    I think you can determine our core potential just by multiplying the default limit by the limit on how many batch accounts we can have.

                                    5 votes
                                    Vote
                                    Sign in
                                    Check!
                                    (thinking…)
                                    Reset
                                    or sign in with
                                    • facebook
                                    • google
                                      Password icon
                                      Signed in as (Sign out)
                                      You have left! (?) (thinking…)
                                      1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                    • Allow B-Series VMs to be chosen

                                      I understand that the main use case of Batch is for high performance computing, but we have a use case where we have jobs that are I/On (disk / network) dependent, so don't take an awful lot of processing power and would run easily in Bingley Series VMs

                                      5 votes
                                      Vote
                                      Sign in
                                      Check!
                                      (thinking…)
                                      Reset
                                      or sign in with
                                      • facebook
                                      • google
                                        Password icon
                                        Signed in as (Sign out)
                                        You have left! (?) (thinking…)
                                        under review  ·  1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                      • Support uploading OutputFiles from CloudServiceConfiguration

                                        Hi Batch Team, we tried to upload output files with the outputfile property from the CloudTask (.Net) like descripted here: https://docs.microsoft.com/en-us/azure/batch/batch-task-output-files on starting the task we get the error "Nodes in CloudServiceConfiguration pools do not support uploading output files" Now we are implementing a workaround, but we would appreciate a build in solution.

                                        5 votes
                                        Vote
                                        Sign in
                                        Check!
                                        (thinking…)
                                        Reset
                                        or sign in with
                                        • facebook
                                        • google
                                          Password icon
                                          Signed in as (Sign out)
                                          You have left! (?) (thinking…)
                                          0 comments  ·  Flag idea as inappropriate…  ·  Admin →
                                        • Raise the number of concurrent tasks per node core

                                          The idea I have in mind is to use Azure Batch to export a >100 TB Azure Table, at a much faster rate than Azure Data Factory can manage.

                                          Azure Tables is quoted to cope with 10,000s of simultaneous requests. To maximize my dev velocity, I would create millions of lightweight tasks and let Azure Batch runtime handle all the scheduling details.

                                          But there is a limit of 4 concurrent tasks per core so this would make for extremely underutilized nodes and long runtime.

                                          https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit

                                          4 votes
                                          Vote
                                          Sign in
                                          Check!
                                          (thinking…)
                                          Reset
                                          or sign in with
                                          • facebook
                                          • google
                                            Password icon
                                            Signed in as (Sign out)
                                            You have left! (?) (thinking…)
                                            1 comment  ·  Flag idea as inappropriate…  ·  Admin →
                                          ← Previous 1
                                          • Don't see your idea?

                                          Batch

                                          Feedback and Knowledge Base