Need the ability to filter security events before they're collected
The security SP badly needs:
- security log filtering so that only specific scenarios can be consolidated to the cloud. Mind don't assume they're connected through OpsMgr, since OpInsights lacks any multitenancy, security scenarios are the first that would need separate workspaces to limit access to the data
- there are some common scenarios that need to be addressed on the collected data, for example there should be an easy way to discard logons by computer accounts (account ending with $).
- princing for Security IP risks to put OpInsights out of play, it's way too expensive, a single DC for 5K users can collect 3 to 5 GB per day, at premium prices og 5€/GB it is going to cost 15€ to 25€ per day, or between 6 to 9 K per year. Now if you add a few DCs and then standard servers you get to the tens of K per year. Way too expensive, customer will remain with an on-prem solution where they can tailor what they collect based on their budget and then can archive, put offline and so on.
Thank you for you patience,
Based on your feedback we have completed the work to enable FILTER security events before collection.
Here is the blog post on this:
We are working on Solution Targeting feature which will provide you ability to select scope of the computers per each solution. Please expect the public preview later on this year.
Andreas Karz commented
We need a solution with which we can define which entries are sent to the OMS cloud. This is the only way to ensure that no sensitive data is sent to the OMS Cloud..
The best solution would be to define which log entries are sent. For example, only all entries that contain "xyz" – like Nagios rules.
Is this possible or is something planned in this direction? What would be really awesome, if it would be possible to implement such filters directly on the OMS gateway – e.g. by Regex definitions.
Justin Grote commented
OMS Team, this is fantastic and thank you for listening to your customers on UserVoice. However, there's an immediately glaring omission: A "custom" option to specify your own event IDs.
For instance, maybe I want to collect object modifications to OMS but not object access, but our internal policy requires object access to be logged, the difference in logs is massive, but I cannot support that scenario.
Why not just add an additional "custom" button that lets me specify the event IDs in a comma separated way just like you do above? All the architecture is there, this is basically just an interface change.
Martin Ehrnst commented
How i see it, this is still very relevant. We use SCOM but i dont see how we can use you MP to filter events when OMS now has assessment packs that chose what logs and events to send.
Kirk Munro commented
Just to add to this, I'm relatively new to OMS and this is the first issue that drove me to the forums to log something. I have a virtual lab with 2 VMs running and the security data has exceeded my "free tier" limits for the past 2 days. From 2 VMs. And I'm not even one week into using OMS. I'm looking at OMS from a developer's perspective, but if I was evaluating this as a prospect, I'd want to be able to review the entire solution, and right off the bat this would scream "way too expensive/bloated", driving me back to on premises solutions. The existing solution clearly is taking an extremely lazy approach when it comes to what data is consumed by default. There has to be some way to remove/compress the benign information so that value can be had from this solution at much, much lower cost in terms of bandwidth and disk space usage.
Michael Voegtline commented
Allow for picking and choosing of Security And Audit logs and events so that the usage for that solution is not such a drain on data used.
Lars Villaume Jørgensen commented
Hey - would be great with an update on plans for a configurable collection policy. Not having this now is a showstopper in many cases, for this otherwise extremely useful solution.
Also look at the related thread http://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6658106-log-filtering (not sure if we'll execute them together or separately - as 'security' events have peculiarities...)
Let's get back to the issue:
- the security IP costs too much
- the security IP uses too much bandwidth
My quest is for a solution that gives me all the logs for all the systems, so that I can do forensic even on unplanned scenarios, that implements Technologies that compress/dedup the data both on the wire and on the final storage. :-)
Since I now you're skillful gals and guys I'm expoecting no less (with some magic, too). :-)
Regarding the orginal question: unless you're able to implement all of the above at cloud speed, we need both: ability to filter for a set of systems and ability to filter out useless events.
Can't we have both :-)
Collect these events from this set of computer
Collect event1, event2, event3 from Domain Controllers
Collect event1 from webservers
Collect event2, event2 from SQL servers
If I'm to choose, then filter on the events to collect. This will generate the least amount of data being uploaded. And thereby saving cost.
Question to all the commenters/voters here: is it *more* important for you to filter which EVENTs you pick up (i.e. only 'logon' events or only 'object access' or similar), or is it more important/more effective to just to limit the solution to a SET of MACHINEs ?
Weston Fraser commented
Security IP as it is now simply is too expensive. This is a shame since it's an extremely valuable IP that like other "security features" could drive the product. Security IP consistently uses 90% of the total data collected when compared to all other IPs. Until something is done to make this IP drastically more cost efficient, it's unusable.
I really need to second this. I just installed the agent (Direct Agent) on 1 Domain Controller. In it's own workspace. We have 45.000 employees and 12 Domain Controller. On this 1 Domain Controller, I send 8G of security data in 24 hours (and this is not the most busy DC we have). 12*8*30=2880G per month (minimum). That becomes way too expensive.
And the ability to analyse security logs would be a key mover for us to use OMS.
SCOM ACS already has the filtering functionality for security event collection. Is there a plan to extend ACS functionality so it can work with OMS? ACS can also provide a larger cache for collected security data in database before uploading them to OMS.