Securing resources using Azure Firewall, specific regions from which the build agent comes cannot be guaranteed even if a very wide ranges of IP addresses, there will be some failures. Only allowing access for all would make it successful, but it should work by blocking few IPs.