Why ITOps still suffers from alert fatigue

It takes a lot of time, effort and money to configure centralized monitoring. Making it all the more frustrating that those carefully crafted alerts will probably just end up being ignored. So why has the whole of ITOps collectively decided to banish your monitoring alerts to their junk folders? The simple answer: alert fatigue.

During this blog we discuss the reasons behind alert fatigue, but more importantly, what can you do about it. Read on to find out more.

 

Alert fatigue is your #1 enemy

Whilst the reasons alerts are ignored remain largely consistent, knowing where to start is a little more complex.

You see, centralized monitoring is often caught in the middle of a vicious cycle. Whilst they have access to a tonne of rich infrastructure data - but because it lacks context - ITOps teams don’t know what to do with it. As such, system fatigue begins to spread and bad habits creep in.

Worse still, because the user experience offered by centralized monitoring tools are often a little out-dated, consumers would rather stick to their own niche tools. Meaning centralized monitoring don’t get the insights and feedback they need to customize alerts. Meaning centralized monitoring continues to adopt this one-size-fits-all approach. Meaning centralized monitoring don’t get the insights and feedback they need to customize alerts. And so on...

Like I said, a vicious cycle.

 

Game shows that are impossible to win

So a few people have decided to switch off from centralized monitoring, so what? That just means ITOps can focus on their day job. No biggy.

Wrong. A big fat wrong.

We've all heard of the phrase “work smarter, not harder”, well, ignoring alerts that are designed to monitor and report on the collective good will engender a culture that does the exact opposite. With that said, I'd like to introduce you to the blame game! As you're someone who works in IT you’re probably familiar with the rules, but for those of you that are new to the game, let’s do a quick re-cap:

 

  1. Keep all your monitoring data to yourself. Under no circumstances should this be shared.
  2. Shift the blame wherever possible. It’s not your fault the application went down.
  3. Look confident and assured. Don’t give other contestants reason to sniff blood.

 

And finally, if you’re the one that eventually gets shafted with the blame, you’ll be crowned this week’s loser. But don’t worry, there’s always next week…

*A terrible theme tune begins to play*

Weird game show analogies aside, a divisive IT culture - whilst not a particularly nice environment to be a part of - will severely impact your end users. Unable to access the systems and services they rely on, internal and external stakeholders will quickly become frustrated with IT. Not only that, core KPIs will take a dramatic turn for the worse. Mean Time to Repair (MTTR) stats will go through the roof, the number of incidents detected by customers vs operations will begin to look ugly. Basically any KPI that can be linked back to your holiday bonus could feasibly be affected.

It’s no wonder then that 43% of business leaders believe that IT can be significantly replaced by shadow IT. After all, if IT hasn’t got a handle on how their work rolls up and contributes to the greater good – how can the business be expected to?

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at UmbracoSquaredUp.Core.Models.ViewModels.Media.ImageViewModel..ctor(IPublishedContent image)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at UmbracoSquaredUp.Core.Extensions.PublishedContentExtensions.AsNestedContent[T](IPublishedContent node)
   at ASP._Page_Views_Partials_RTEImage_cshtml.Execute() in C:\home\site\wwwroot\Views\Partials\RTEImage.cshtml:line 14
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
   at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
   at System.Web.Mvc.RazorView.RenderView(ViewContext viewContext, TextWriter writer, Object instance)
   at System.Web.Mvc.BuildManagerCompiledView.Render(ViewContext viewContext, TextWriter writer)
   at Umbraco.Core.Profiling.ProfilingView.Render(ViewContext viewContext, TextWriter writer)
   at System.Web.Mvc.HtmlHelper.RenderPartialInternal(String partialViewName, ViewDataDictionary viewData, Object model, TextWriter writer, ViewEngineCollection viewEngineCollection)
   at System.Web.Mvc.Html.PartialExtensions.Partial(HtmlHelper htmlHelper, String partialViewName, Object model, ViewDataDictionary viewData)
   at UmbracoSquaredUp.Core.Helpers.TemplateHelper.ParseImages(String text, HtmlHelper helper)
   at ASP._Page_Views_Partials_grid_editors_rte_cshtml.Execute() in C:\home\site\wwwroot\Views\Partials\grid\editors\rte.cshtml:line 10
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
   at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
   at System.Web.Mvc.RazorView.RenderView(ViewContext viewContext, TextWriter writer, Object instance)
   at System.Web.Mvc.BuildManagerCompiledView.Render(ViewContext viewContext, TextWriter writer)
   at Umbraco.Core.Profiling.ProfilingView.Render(ViewContext viewContext, TextWriter writer)
   at System.Web.Mvc.HtmlHelper.RenderPartialInternal(String partialViewName, ViewDataDictionary viewData, Object model, TextWriter writer, ViewEngineCollection viewEngineCollection)
   at System.Web.Mvc.Html.PartialExtensions.Partial(HtmlHelper htmlHelper, String partialViewName, Object model, ViewDataDictionary viewData)
   at System.Web.Mvc.Html.PartialExtensions.Partial(HtmlHelper htmlHelper, String partialViewName, Object model)
   at ASP._Page_Views_Partials_grid_editors_base_cshtml.Execute() in C:\home\site\wwwroot\Views\Partials\grid\editors\base.cshtml:line 20
Joe EdwardsDigital Marketing Manager