Many quality improvement (QI) teams use interruptive alerts as part of their care improvement initiatives. Alerts are finely tuned to specific points in the workflow to influence behavior. Many quality improvement and informatics professionals agree they’re intended, ideally, to adhere to the Five Rights of Clinical Decision Support (CDS). For those more statistically inclined, a goal when designing an alert is for it to be very sensitive and specific. In other words, when an alert fires, it should correctly identify an action that must be taken by the targeted end-user (true positive). On the other hand, an alert that doesn’t fire should only do so when the end-user doesn’t warrant it (true negative).
Unfortunately, many hospital informatics and quality improvement teams inadvertently overuse interruptive alerts as blunt tools and suffer downstream consequences of poor results for both patients and clinicians. This issue can also lead to alert fatigue among end-users, which may spill over to detract from well built interventions. Ultimately, the principal concern is that this desensitization can fail to prevent the very quality and safety incidents the alerts are focused on.
If you’re interested in understanding how interruptive alerts affect care at your facility or within your healthcare system, we recommend two metrics.
First, the override rate is the number of overridden interruptive firings divided by the sum of non-neutral interruptive firings. This metric tells you if the alerts are, on average, being acknowledged or discarded. It’s important to dive into the meaning of “override” and “neutral” actions. If a user is shown an interruptive informational alert without any possible actions, it shouldn’t be considered an “override” when they cancel it out. After all, there was no other course of action for them to take. We define these as “neutral” actions. As a result, it is important to take the alert design into consideration when calculating this metric. Phrase Health uses a logic tree to determine the mapping of an “override,” “accepted,” and “neutral” action, but allows clients to customize these definitions based on local configurations or customizations.
Second, the Phrase Burden Index, also referred to as PBI, is the number of interruptive firings per potentially affected end-user per day. The ideal population of users to use in the denominator is the cohort limited by the alert’s rules definition. For example, if an alert’s rule is simply limited to a single department, then all the users in that department should be in the denominator. In order to estimate this in our analysis given the potential complexity of rule logic, a potentially affected end-user is one who has seen an alert firing sometime in the past 365 days.
The override rate and Phrase Burden Index can be used to answer different sets of questions for quality improvement and informatics teams.
Typically, the override rate is thought of as an indicator of how good an alert is. If an alert has a low override rate, people are often adhering to it and likely find it helpful. That association seems to bear out, although the average override rate is about 87% based on analysis of health systems in our benchmarking study, which would imply that interruptive alerts are difficult to implement optimally.
There are two faults with hastily coming to this conclusion. First, it doesn’t tell the story of what a provider does after the alert is overridden. Do they end up doing the action they were alerted to do, anyway? Or, do they go with a different approach? Further analysis with a tool designed to uncover that information would be necessary to help uncover if the intended outcome was achieved. As a result, it’s impossible to determine the effectiveness of an alert simply by looking at an override rate.
Second, good teams use override rates to quickly iterate. Much like user comments can be used as feedback to find malfunctions, override rates can be used to highlight iterative opportunities. One team leveraged a certain venous thromboembolism (VTE) prophylaxis alert’s high override rate as an impetus to speak with medical residents to understand how they were interpreting the alert. After just a few conversations, the team identified an improvement opportunity around confusing wording and were able to drop the override rate by approximately 20%. In this way, the override rate was used as a way to drive quick feedback and iteration.
The Phrase Burden Index helps solve different questions than the override rate, namely:
It’s important to note that interruptivity isn’t a bad thing. In fact, we’ll describe a “burdensome” alert that is actually one of the more useful and well-targeted at one institution. However, when an alert isn’t targeted correctly, being highly concentrated also means “annoying.”
The absolute value of the Phrase Burden Index does hold meaning. It roughly translates to “the number of interruptive alerts per day that a potential end-user sees.” However, since defining “burden” is up for interpretation and is a rough calculation, the relative measurement between alerts within an organization often provides the best way to start prioritizing optimization initiatives. A high burden score for one alert compared to others should raise a flag for quality improvement and informatics teams. Are the right providers being included? Is the alert arriving at the right time in the workflow?
Sepsis alerts fall within a relatively narrow Phrase Burden Index range, likely because many organizations spend significant time honing them. Among our sample, override rates are inversely correlated with the Phrase Burden Index. This might suggest that if an alert is well-tailored to specific people or contexts, it is more likely to drive an action to the improvement team’s intended outcome even if it interrupts clinicians significantly more. In other words, Increasing bad alerts increases alert fatigue. Increasing well-designed alerts drive intended actions. There are some caveats to this analysis including a limited number of sepsis alerts that make up the data set and the fact that Phrase clients might be more likely to engage in optimization opportunities for their alerts.
Override Rates and Phrase Burden Index Scores for Select Sepsis Alerts
Sepsis and venous thromboembolism (VTE) have the lowest average alert override rates among hospital conditions, while asthma has the highest override rate. This correlation is up for debate, but this is potentially because sepsis and VTE can present more insidiously from a clinical perspective. Additionally, sepsis and VTE are oftentimes reportable metrics that may receive more attention from operational leadership.
Override Rate Comparison Across Selected Conditions
The table below shows a collection of Phrase Burden Index scores for alerts that the Phrase Health team has seen at some point at its partner sites. The Phrase Burden Index can vary widely and requires an understanding of the clinical outcomes of the alert, but it is an interesting metric to discuss in the context of effectiveness and efficiency.
Based on a more in depth analysis of Phrase Burden Index scores, there doesn’t seem to be any overarching patterns of why organizations might build certain alerts with a high burden. However, there may be some evidence that even across different health systems, some specialties are consistent in their approaches to build, deploy, and refine alerts. For example, certain specialties seem to devote less time in tailoring order sets which might be similar across alerts and other types of clinical decision support. This could reflect minimal engagement of some specialties with the design, monitoring, and optimization of alerts.
Alerts with High Phrase Burden Index Scores
As mentioned above, we’ve found health systems leverage override metrics in a variety of ways. An obvious use case for override rates in quality improvement is to look for high values for your initiatives. Although the override rate does not account for all the aspects of a clinician’s decision to override an alert, it can suggest a potential opportunity for improvement.
One organization we’ve worked with improved their ordering of nonviolent restraints by decreasing the alert override by about 67%. They did so by including discrete action buttons that allowed the team to iterate and hone the alert’s target audience. Aside from collecting free text feedback from end-users, the use of discrete action buttons like this serves as an alternative, albeit more limited, approach to capturing user feedback.
One client reviewed their Phrase Burden Index and noticed that a particular ambulatory oncology alert had a burden index of about 16. Upon further inspection through Phrase Health, they identified that one provider was receiving the alert over 16 times per day. Unsurprisingly, the provider didn’t take the intended action of reviewing the problem list even once over the course of more than 4,500 firings.
A different client looked at interruptivity at a more global level. Presented with data from our benchmarking study, they noticed that their General Surgery group had a higher burden of interruptive alerts than their peers. This helped the team contextualize and validate a disgruntled surgeon’s feedback that he was alerted more frequently at their health system than his previous one.
As with all analyses, there are some pitfalls to consider when interpreting these metrics.
For the override rate, quality improvement teams must not necessarily link it directly to quality. Further analysis beyond the override rate data would be needed to draw downstream conclusions. When thinking with a Plan-Do-Study-Act (PDSA) cycle mentality, sometimes override rates are useful to spot improvement opportunities. Override rates also could suggest good feedback opportunities, guiding the team on useful initiatives to engage providers about.
Considerations for the Phrase Burden Index include remembering that there is no magic number regarding a good number. It can be particularly useful when looking at the relative impact compared to other deployed alerts when prioritizing where to focus improvement efforts related to clinician burnout. The input should encourage you to ask more questions. Is a 10 necessarily bad? Ultimately, it depends on if you're able to drive the outcome you’re interested in.
A high Phrase Burden Index can actually be a good thing. One Phrase client asked for feedback on its kidney injury risk alert to the pharmacists who were receiving it, as it had a high Phrase Burden Index. The alert reads simply, “Patient has renal impairment FYI flag. Please review.” The pharmacists actually commented that it was one of the more useful alerts they received, despite the relatively high burden score.
We recommend that informatics and quality improvement teams interested in improving alerting look at two metrics: override rates and the Phrase Burden Index. Override rates show interruptive firings divided by non-neutral interruptive firings. The Phrase Burden Index is interruptive firings per potentially affected end-user per day.
These two metrics point out improvement opportunities around utility, frequency, and concentration of alerts. This can lead to downstream benefits for improving patient outcomes and also decreasing end-user alert fatigue. In our analysis, we were able to pinpoint alerts with both high override rates and high Phrase Burden Index scores. While these results indicated potential issues, we recognized that each situation could benefit from further iterative analysis. Going beyond the override rate, for example, requires knowing what a provider actually did after discarding the alert. That data must also be collected and analyzed.
The override rate and Phrase Burden Index score help informatics and quality improvement teams maintain an efficient and productive system of alerts for end-users.
ADDITIONAL PHRASE HEALTH REPORTS