Prevent Azure CDN Bandwidth abuse
I have already made a couple of posts in regards to this issue on StackOverflow: http://stackoverflow.com/questions/35488753/how-to-prevent-azure-cdn-bandwidth-abuse-by-malicious-bandwidth-vampire-requests and on MSDN CDN forum: https://social.msdn.microsoft.com/Forums/azure/en-US/9e37ca24-b38d-4193-847b-f679eab76aa5/azure-cdn-bandwidth-abuse-by-malicious-bandwidth-vampire-requests?forum=azurecdn but so far, unfortunately, no good solutions to the problem were offered. So I thought it would be a good idea to post this idea here as well, and get a little more into detail on how to solve the issue since this little fix can easily be integrated into the upcoming WAF offering with the Premium SKU.
To summarize, the problem can simply be stated as follows: Any large multimedia content file such as an image or a video that is publically exposed via a CDN, can be downloaded continuously thousands of times non-stop via programmatically generated malicious requests which CDN cannot prevent, resulting in a huge bandwidth bill for a customer.
One can compare this problem to bank security flaw, which allows endless withdrawals of pennies from customer’s bank account at will, without any way to stop it.
While the issue described above may feel a little more on a defensive side, it still is a real issue at hand, and actually is the first thing that came to our mind when considering exposing content publicly via CDN. Everybody loves the speed and proximity of CDN but security of content against malicious “bandwidth-vampire” requests is also very important.
In our specific case, where we are building a website that hosts many images, we would like to be able to limit the number of requests that can be made for each image in a particular time frame. So for example, a request coming from a particular IP address, may not download a picture file more than 10 times in 1 hour. So what a customer should be able to do, is create a rule that when activated for a specific URI destined for a container protected by the rule, does the following:
1) Looks at the incoming URI.
2) Checks if the URI is for a file that is inside a container defined in a rule (image container for example)
3) Pulls out HitCount(integer) for this URI by requester’s IPAddress+URI(Key)
4) Checks if HitCount for this URI from that IP has exceeded the allowed hit count (defined by user) during (a time frame defined by user).
5) If HitCount is equal to the limit, then CDN serves 403. Else CDN serves the content.
6) When a timeframe defined by a user has elapsed, The HitCount goes back to 0, and requests can continue until HitCount limit has been reached again.
Why Technical Implementation is So Easy:
1) Azure Premium CDN already has the ability for the user to define a bunch of rules that filter incoming request based on header values, referrer values, etc. So this is would not require architecting anything new, same per-request filtering will be applied here.
2) A generic Dictionary data structure can be used for O(1) lookups . Key and Value are defined as follows:
Key: IPAddress+URI (stripped from query string, since appending a query string to a URI that ends with a .jpg (ex: .jpg?fdfdfdf=4455 ) still serves the picture file, thus stripping away the query string and combining it with IP address would guarantee to provide a unique key value for dictionary lookup)
Value: HitCount for that particular URI
So this is pretty much the core of what needs to happen, it really is very trivial to implement.
This would give a customer a piece of mind about being protected against such attacks and incurring humungous bill. And it also gives the customer ability to deliver content via speed and agility of Premium CDN, thus providing Azure with bigger profit!
So everyone wins, customer is happy and protected and Azure is making money. We don’t think AWS has implemented this feature yet, so why not be the first?
Does anyone have a solution for this?
Good thing that I am not paranoid! It appears that Amazon has already thought of such a thing: http://docs.aws.amazon.com/waf/latest/developerguide/tutorials-rate-based-blocking.html
I will have to take a closer look at this, and see if you can:
a) Customize rate-based blocking to only parts of your application / only certain containers
b) Customize rate-based blocking only for certain files