I want to be able to filter stuff I don't want to collect in logs. For example with ACS (in SCOM) I could apply filters that didn't collect system logins. I would like this functionality in all logs, for example I would want to filter IIS logs to remove data from certain IP addresses.
I can see customers wanting to use this type of functionality when the costs of data start to pile up.
This feature is already in progress, limited preview is expected later in 2018
Andrew Sears commented
Any updates? Relevant for AKS and HDInsight logging, currently it requires some custom setup.
where is the limited preview?
when is this custom event filtering going to be available? it needs to be targetable to hosts and groups.
the lack of it it is causing us significant ingestion costs - which means we have disabled OMS on a number of servers across the board due to lack of customisation.
Andreas Karz commented
We need a solution with which we can define which entries are sent to the OMS cloud. This is the only way to ensure that no sensitive data is sent to the OMS Cloud..
The best solution would be to define which log entries are sent. For example, only all entries that contain "xyz" – like Nagios rules.
Is this possible or is something planned in this direction? What would be really awesome, if it would be possible to implement such filters directly on the OMS gateway – e.g. by Regex definitions.
Steve Bogar commented
I would like more advanced filtering capability in the Data tab of the Settings solution.
Right now we can only filter based off Error/Warning/Information. I would like more granular control and be able to collect certain informational messages like Event 1074 but I don't necessarily want to store every single informational system message up in the cloud.
Is there any way to do this?
If you use Operations Manager, you can do this with Windows Events by defining your rules onprem - http://blogs.technet.com/b/momteam/archive/2014/08/27/anatomy-of-an-event-collection-rule-for-advisor-preview-advanced-targeting.aspx - rather than in the cloud policy (which is more simplified).
About the Security Intelligence Pack, I agree there needs to be some level of filtering there (certain EventID's are useful for a scenario but not for another - even within the same security space), so we are planning to have some configuration of what to collect in that sense.
Not sure of the feasibility for IIS - today we just copy over the FILES without looking at them/parsing them onprem... so looking for IP addresses might be overkill there. But we could allow something like a 'only certain websites' type of filtering for that, for now it's all or nothing.
Bottom line - it depends on the data type, but yes we understand what you are saying :-)