Collect IIS Advanced logs
Allow the collection and addition of custom fields using advanced logging or custom IIS modules. Example is to add x-forwarded-for to IIS logs in W3WC format.
Theo Carpenter
shared this idea
This feature request is still under review and team is actively prioritizing with existing backlog. Will keep the thread updated as we move forward.
16 comments
-
Florian Winter
commented
We have an application which returns a custom header, X-RequestId, which is a unique ID of every request, which also appears in other application logs. We configured IIS to log this header, but Azure Log Analytics doesn't pick it up. Being able to correlate request logs with application logs using request IDs is very helpful for making sense of logs and tracing the origins of errors. Azure Log Analytics should at least collect the fields that IIS has been configured to log.
-
David Maskell
commented
NYC-IT's solution is the easiest I have found until Azure supports the X-Forwarded-For by default in W3C logs.
It is worth mentioning you may need to change the index of CSVFields to match your logs as my x-forwarded-for field is at index 15.
I also used the following to make my parsed logs more readable:| extend cIP = substring(tostring(CSVFields[15]), 0, indexof(tostring(CSVFields[15]), ","))
| extend RealDatetime = todatetime(strcat(Date, " ", Time))
| extend FormattedDatetime = format_datetime(RealDatetime, 'dd/MM/y HH:mm:ss')
| where RealDatetime >= now(-1h)
| project FormattedDatetime,cIP,Method,Uri,Uristem,Uriquery,Port,csStatus,UserAgent
| sort by FormattedDatetime desc nulls last -
NYC-IT
commented
I was able to report on x-forwarded-for via Custom Logs, following an MS article. It is rudimentary but it is something. Microsoft, you can do so much better!!! I wish I had 100 votes!
IISCustom4_CL
| extend CSVFields = split(RawData, ' ')
| extend Date = tostring(CSVFields[0])
| extend Time = tostring(CSVFields[1])
| extend WAFIP = tostring(CSVFields[2])
| extend Method = tostring(CSVFields[3])
| extend Uristem = tostring(CSVFields[4])
| extend Uriquery = tostring(CSVFields[5])
| extend Port = tostring(CSVFields[6])
| extend UserAgent = tostring(CSVFields[9])
| extend ClientIP = tostring(CSVFields[16]) -
BenTheBuilder
commented
What I've been doing to collect ClientIP is using ApplicationHost.xdt transform on my WebApps to collect the first IP of the XFF until we can fix the ClientIp collection behind AppGW/AFD - this is a bandaide at best. But it does work in that you are able to get ClientIP in your WebApp IIS Logs from both StorageAccount and OMS logging.
https://blogs.msdn.microsoft.com/benjaminperkins/2015/03/03/making-changes-to-the-applicationhost-config-on-azure-websites/<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.webServer>
<arrHelper>
<trustedProxies xdt:Transform="SetAttributes(trustUnlisted)" trustUnlisted="true" />
</arrHelper>
</system.webServer>
</configuration> -
Subodh Patil
commented
We need at least X-Forwarded-For field in Azure Log Analytics
-
Subodh Patil
commented
Is there any update on this ?
-
Anonymous
commented
We really need this. We must be able to see where the client connects from.
-
Pascal Houde
commented
Looking for that x-forwarded-for to find its way from the App Gateway to the IIS Log files. We are troubleshooting an issue and the only way for us to prove out this issue is by seeing that x-forwarded-for header in the logs. I was really hoping that 4 years later this would be available.
-
Surya
commented
Dear OMS LA Team,
This thread is not updated for past 4 years, do this feature will be provided into Azure log analytics?
-
Magnus
commented
Maybe this could help with onprem iis: https://devcentral.f5.com/articles/x-forwarded-for-log-filter-for-windows-servers#.UUExZ1ckTTc
-
Andreas Larsen
commented
I just completed fending off a botnet attack behind Cloudflare, whereas I was getting Cloudflare IPs as the visitor IP and no way to find the original visitor IP in order to block them in the Cloudflare firewall without first disabling HTTP Proxy, effectively exposing my server IP and opening up the attack surface of my server, just in order to get the visitor IPs.
I talked to Cloudflare support and they refer to X-Forwarded-For header as being the de-factor standard for this scenario, widely supported in web farm and HTTP proxy solutions. If I were running a VM or possibly Azure Cloud Service, I would be able to enable logging of this, but not when using Azure Web Apps. This is such a standard scenario for Azure users, I am amazed there is no solution for this yet.
The two solutions I can think of are:
1) Change the code of all my Azure services to explicitly log the X-Forwarded-For request headers in my application logs. Really not desirable.2) Put my web apps and services behind an Azure firewall to give me more control of blocking IPs and viewing traffic details. We originally used Cloudflare for this, but their firewall offering is quite basic.
-
Chris Stanley
commented
Full support for Advanced Logging would be great, but what about just supporting additional custom fields in normal IIS 8.5 custom logging? It seems like this may be simpler and it would meet our needs to grab a quick custom entry like the x-forwarded-for.
https://www.iis.net/configreference/system.applicationhost/sites/site/logfile/customfields/add -
Austin McCollum
commented
X-Forwarded-For is very important for the data I work with.
-
Currently we only understand the default format. That field is not logged by default and must be enabled explicitly like described here http://www.iis.net/learn/get-started/whats-new-in-iis-85/enhanced-logging-for-iis85 but our parser doesn't know about it.
With the current thinking we don't want to special case just a handful of those known fields - since they can be customized heavily - but this is something that will be evolved once we will allow custom parsing of arbitrary logs - loosely tracked in the following two related ideas
http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/7113030-collect-text-log-files
http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519270-support-regular-expressions-regex-or-xpath-to-pe -
Anonymous
commented
Given that one critical functionality is to determine client activity then lacking the source IP from the X-Forwarded-For field where SNAT devices are involved is very short sighted
-
Anonymous
commented
Given that one critical functionality is to determine client activity then lacking the source IP where SNAT devices are involved is very short sighted