Last week, I received an email from a #CMIO partner of ours about how to interpret the alert override rate percentage in our system. This simple metric is actually surprisingly pretty complex and open for interpretation. Some complexities include: 1) accounting for alerts with no action available when the end-user closes it, 2) counting non-interruptive alert firings, 3) deciphering multiple actions that can be taken from a single alert firing, 4) acknowledgement buttons can sometimes represent overrides, acceptance, or neither, and 5) accounting for actions that aren’t overrides or accepts. I tried to account for all of these nuances when I started building out the Phrase Health platform. For example, our system guesses appropriate definitions of “OVERRIDE”, “ACCEPT”, and “NEUTRAL” based on alert design, while also allowing administrators to define these themselves at the individual action level for each alert. However, how to then turn this granular control into an interpretable alert-level metric? With broad user feedback, we’re working through a variety of “Acceptance Rate” (#BehavioralEconomics) metrics that provide different lenses into this insight.
The CMIO’s question more broadly highlights the importance (and complexity) of distilling raw data into more interpretable and actionable knowledge. This is a significant focus for me and my team because it’s so fundamental to 1) enabling more people to do this incredibly important work and 2) more rapidly iterating through high-yield PDSA cycles of EHR workflow improvement. Both of these ultimately lead to achieving clinician efficiencies and cost-savings faster and at scale.
The complexity of electronic health record (EHR) optimization is nothing short of daunting and is why EHR Workflow Management solutions like Phrase Health exist. Navigating through this process can often feel like a shot in the dark when using data that is too close to the raw form (e.g., alert firings). When you have a small team (sometimes only you!) and are tasked with improving 1000’s of tools in the EHR, it’s important to prioritize the biggest pain points and to tackle them quickly; armed with insights served up to you. During my clinical informatics fellowship, my colleagues and I strove to bring light into this process for EHR alerting to start, focusing on identifying what was working versus what wasn't.
It quickly became evident that counting alert firings alone wasn't sufficient. We had to understand the impact of these alerts on the end-user; for instance, some alerts might not be interruptive and therefore less burdensome (think of inline ads versus pop-up ads on a website). Importantly, it also mattered how concentrated an alert firing is based on the rule restrictions. In other words, did an alert fire 100 times to one person or once to 100 people? What’s the distribution of interruptions and non-interruptions based on this concentration?
The result of our pursuit led to the creation of the Phrase Burden Index (PBI). PBI is a metric that quantifies interruptive alert firings per day to individuals potentially exposed to the alert. It roughly translates into: “for end-users exposed to this alert, how many interruptive alert firings are occurring per day to them?” Most alerts we’ve seen are less than 2, but we’ve seen a PBI as high as 17 at one organization for an alert focused on a small number of ambulatory clinics! A more extensive analysis of the concept of alert burden can be found in Orenstein et al's publication, "Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics." This initial step marked the first example of our journey to demystify EHR optimization.
When faced with cursory data and limited insights, the challenge to improve broken EHR workflows becomes incredibly steep. Traditionally, expert informaticists and data gurus can take the helm, digging deep into the data and user interviews to uncover issues. However, we've discovered that significant value lies in doing this work faster by empowering more users with the appropriate tools.
The Phrase Health team is working to harness this power by distilling automated insights in a user-friendly manner. This approach allows operational teams at health systems with varying levels or EHR technical expertise to navigate EHR performance data, empowering them to embark on swift PDSA (Plan, Do, Study, Act) cycles.
I’ll make one additional note here. The importance of usability and user experience in the interpretability of information cannot be understated. This is a topic for another day, but I’ll simply say that design impacts action. With that in mind, one of my first hires was a UX Designer and, to this day, user experience remains top-of-mind at Phrase Health. I’m pleased to see this reflected in our positive end-user feedback (“this is an informaticist’s dream”) and NPS score which hovers around 70.
Starting this quarter, we’re deploying our Actionable Insights dashboards to our insight-hungry partner organizations. The aim is to distill data around clinical EHR workflows and get our end-users closer to understanding how to fix an improvement opportunity.
These insights will aim to cover all the EHR workflow components covered in our platform and, when possible, will be normalized into an index for benchmarking. For example, with alerts, we're deploying the "Rage Index," a measure of alert flooding that suggests backoff periods, and "The Cranky Index," a weighted measure of “Cranky Comments” sentiment that convey actionability. In order sets, we're implementing "The Workflow Deviation Index" and "The Friction Index," measures of deviation of ordering after order bundle usage and variation from from defaulted orders, respectively. In the realm of medications, we've developed "The Redesign Reorder Index" and "The Safety Reorder Index." And so on down the list
Our pursuit of optimizing EHR workflows requires an unwavering focus on actionability. Beyond each metric conveying an interpretable potential insight, we are committed to normalizing these metrics, when feasible, for internal comparisons and also for anonymously benchmark organizations to understand “what is normal” and “where are my biggest opportunities.” An ever-growing library of actionable metrics across EHR workflows will serve operational teams as they aim to do more with less.
With a usable interface, we see significant engagement in EHR workflow improvement when it’s made usable and interpretable. For example, one of our partners has had 94 unique users log in within the past 30 days. These users span informatics, quality, safety, IT, and clinical domains. This reflects active engagement in improvement, exploration, and governance of EHR workflows.
Is your EHR Workflow Management solution unleashing the full potential of your workforce? Contact us if you want to learn how we help with EHR workflow insights and governance via an extremely light IT implementation.