Intelligence Pack Updates
With IP's that are pushed out from Microsoft down to SCOM, it would be handy to have a RSS feed, or SCOM alert, that would notify that a new IP is being distributed, along with a list of changes.
This could be also for changes to the Ops Insights website.
If anything does go wrong with a change MS makes, especially if it affects SCOM clients/monitoring, we can at least be aware that a change has happened.
As I understand, MS push out changes to IP's, so we have no control on when those changes affect our environment.
Currently I've just hacked together a script to check the ManagementPackHistory table in my Ops Manager DB. But it's a basic solution so I know if anything happens.
We do multiple deployments a day, this is very challenging to do in a services world, especially for the cloud part: the portal and multiple other parts of the online service can change and be redeployed dozens of times each day, every day, on a typical week when developers check in new code, this goes thru a bunch of automated validation, and then gets pushed out.
We have ability to show/hide new features to selected tenants/workspaces – but there are a lot of teams that independently contribute to the portal and system for various scenarios, and check in and deploy their changes independently thru automated workflows.
We intentionally go ‘slower’ with the Intelligence Packs – because we know that they can affect on-premises. Those are generally pushed out at a slower cadence, but can still happen a few different times in the same week; the deployment mechanisms are similar, but we do hold them back longer and let them bake in our environments to observe the behavior, and have a comprehensive (and expanding) test matrix of systems/configurations.
We do have a mechanism to make the gallery tile ‘blink’ when we have released NEW intelligence packs — complete NEW scenarios to opt in or out of, I mean. Not just changes to libraries or existing pieces.
We could think of publishing a feed of at least this part, but again, the notice would still be very short, and often ‘small’ changes don’t get full documentation (fixes, as opposed to features)… but it would probably allow you to disable the auto MP update rule in production and leave it enabled in pre-prod/test system… when something is new, you take a look if in pre-prod everything looks fine, do your testing, decide if it is good, and then flip the rule on in prod for a few minutes to let it grab the updates, then stop it again.
I've put together this script to give some visibility on management pack updates pushed out from OMS into SCOM.
Saul Guttman commented
I like what this tool/service provides but I do not think that this tool/service can really be called enterprise ready if there is no release management. I have had two outages as a result of AOI running on production servers and I simply had to remove the service from all hosts in my environment. In my opinion the risk outweighs the benefit.
Glen Eustace commented
I don't believe there is any right answer for this, we haven't come up with one anyway. We have now been bitten twice with IP pushes that have impacted our production environment. We do appreciate that MS are taking as much care as possible with their releases but there is always a risk that something will break. From our perspective, we lose credibility as we have tried hard to get SCOM deployed and have assured people during deployment that in most cases adding monitoring has little or no impact on production services. When a service is then impacted by an IP push, which we didn't initiate and therefore hasn't gone through our change management process, SCOMs reputation suffers as it is seen as a system that breaks things.
Unfortunately, we don't have a test system with all our production servers (around 300) due to resourcing, and multihoming all SCOM agents to both test and prod could cause other issues. Our test setup is relatively small and we add agents as needed for test purposes.
Also to have a test environment - this means running two Ops Insights instances - one prod and one test.
A blinking tile would be good, but that means staying signed in - and we may not have the console visible at all times.
A feed is ideal. Even if it doesn't include a changelog, just a notification that xyz has been updated. Short notice is fine - any notice is better than no notice. Even if it's done as the same time as the update.
An example is the recent issue with an IP update for Microsoft.IntelligencePacks.Types.HealthServiceProxyConfiguration that incorrectly flagged as a system rule. This broke monitoring in SCOM on our 2003 servers that didn't have powershell. As I was unaware of an IP update we ended up escalating to Premier Support, as I was not aware of any changes that may affect our environment.
It would be awesome if you guys could add this feature.