What exactly is "Alert Fatigue"?

Clinical Decision Support (CDS) - It's the mythical creature that every healthcare administrator and informaticist is hunting, to help reduce costs and improve care. Loosely, it can be broken into a few different areas :

1. Electronic decision support (e.g. CPOE Alerts to help prevent errors)
2. Order / order set design (e.g. to help prevent errors / guide docs to evidence-based care)
3. Workflow/documentation redesign (e.g. tools used to standardize high-risk decisions, e.g. procedure checklists)
4. Workflow/protocol design (e.g. tools used to automate high-risk procedures)

One of the hardest to tackle is #1 - CPOE Alerts. Are there too many, or too few? Everyone I know seems to be struggling with the same issue :

• Wanting to provide CPOE alerts to avoid errors, but
• Providing "too many alerts" could cause docs to ignore the "important alerts".

This phenomenon is loosely called alert fatigue, and has been fairly well-documented in literature as, paradoxically, a potential risk. 

When you hear Informatics professionals discuss alert fatigue, the challenging part is actually knowing when alert fatigue exists. Docs sometimes complain about it, but the response docs get to this is often skepticism - After all, how can an alert be bad? Maybe the doc just complains too much? And who is going to turn off the alert? Is it safe to turn off the alert? What if this opens up other problems? When is it too much? When is it too little?

So when you ask docs to define alert fatigue, they typically use general, loose definitions, like :

• "It's when the system gives me too much information and I miss the important stuff."
• "It's when the system tells me about the Tylenol interacting with Colace, but I miss out on the Coumadin/Bactrim interaction."
• "It's when I can't read all of the alerts."
• "It's when I just keep clicking 'Bypass' without actually reading the alert."
• "It's when I just keep clicking 'Acknowledge' without actually reading the alert."
• "It's when I click 'bypass' within 3 seconds, so I know I didn't read the alert."

And recently, when I asked some informatics colleagues for their definition of alert fatigue, I again got a myriad of responses, followed by the same sort of response Supreme Court Justice Potter Stewart gave in 1964, when defining "obscenity" in the Jacobellis v. Ohio case : "I know it when I see it."

Unfortunately, this doesn't help much for those of us who are really working to combat alert fatigue. 

The problem with all of these definitions is that they are fairly loose and subjective, and don't make a good litmus test to answer the question : Do you have alert fatigue?

So I'm going to use some reason and inference, to try to develop a better definition of alert fatigue that is quantifiable. (I used to be a mathematician/statistician, so please forgive the quasi-mathematical approach.)

Since it seems the "undesired scenario" nobody wants is made up of two steps :

• An EMR providing a confusing alert environment, and
• A doc displaying signs of poor response to that environment

So I'd like to submit two proofs, for two conditions which then go into a third proof. Here they are :

PROOF1 : "AlertOverload"

1. [AlertOverload] = [Bad] > [Good]

2. [AlertOverload] = [Noise] > [Signal] 

3. [AlertOverload] = [Low-value alerts] > [High-value alerts]

4. [AlertOverload] = [Low-risk alerts] > [High-risk alerts] 

5. [AlertOverload] = [# of low-risk alerts] > [# of high-risk alerts] 

6. [AlertOverload] = [Number of low-risk alerts in a time period] > [Number of high-risk alerts in a time period]

7. [AlertOverload] = When the number of low-risk alerts exceeds the number of high-risk alerts for a given physician in a given time period

PROOF2 : "AlertLoss"

1. [AlertLoss] = [Bad] > [Good]

2. [AlertLoss] = [BypassedAlert] > [AcknowledgedAlert] 

3. [AlertLoss] = [Number of bypassed alerts in a given time period] > [Number of acknowledged alerts in a given time period] 

4. [AlertLoss] = When the number of bypassed alerts exceeds the number of acknowledged alerts in a given time period

If one were to accept proof #1 and #2 as true, then I would propose this final proof/definition of AlertFatigue :

PROOF3 : "AlertFatigue"

1. [Bad] = [Bad] 

2. [AlertFatigue] = [Bad]

3. [AlertFatigue] = [AlertOverload] + [AlertLoss] 

4. [AlertFatigue] = Exists when a given physician experiences [AlertOverload] and displays [AlertLoss] in a given time period

So voila - My proposed definitions :

1. Alert Overload = When the number of low-risk alerts exceeds the number of high-risk alerts for a given physician in a given time period
2. Alert Loss = When the number of bypassed alerts exceeds the number of acknowledged alerts in a given time period
3. Alert Fatigue = "When a given physician experiences alert overload and displays evidence of alert loss in a given time period."

It's certainly not a universally-recognized definition, and I'm curious if other people are aware of any other professional, practical, policy-grade definitions that exist out there. Obviously, this definition now needs to be peer reviewed, tested, validated, and professionally accepted, so please don't use it in your own organization without consulting a legal professional, informatics professional, and your local regulatory agencies first.

Remember : As always, this discussion is for educational purposes only! Remember, your mileage may vary! Always enjoy thoughts, comments, and ideas!

Why not let docs have their own order sets?

Last week, I was asked a question I'd been asked many times before, by someone who was helping another hospital set up their EMR :

"Why shouldn't we let our docs make their own order sets?"

The reason I was asked this is because many EMR packages have this feature, where docs can make their own order sets for their own convenience. 

REASONS TO LET DOCS MAKE THEIR OWN ORDER SETS :

1. Every doc struggles with efficiency, and making your own order sets certainly is tempting. Why not make order sets that accomplish exactly what you want? After all, if I'm a practicing physician, and I know what orders I 'always put in' in certain scenarios, why shouldn't I be able to make my own order sets?
2. We could save so much committee time if the docs could just make their own order sets!
3. If the docs could make their own order sets, they would probably feel "more comfortable" with order sets and CPOE, in general - Wouldn't this help us with our EMR implementation? Wouldn't we get a higher CPOE rate faster?
4. It would save the time and labor of converting their old paper order sets - (Which, as I've discussed in past postings, are often loaded with embedded protocols which are expensive and time-consuming to engineer out!) - Just let the doctors make their own order sets!
5. If docs could make their own order sets, then they probably wouldn't blame some poor informaticist for making a bad order set for them.

REASONS NOT TO LET DOCS MAKE THEIR OWN ORDER SETS :

1. If every doc can make their own order sets, you have no centralized mechanism for clinical decision support. For example, if your pharmacy previously paid a lot for omeprazole, and it suddenly gets a good deal on pantoprazole, you will probably want all of your doctors to take advantage of the cost savings by steering them towards pantoprazole (when backed by good evidence) - If they all have their own order sets, you won't be able to help guide physician behavior and take advantage of this cost savings. 
2. If every doc can make their own order sets, you also have no centralized mechanism for standardizing care. E.g. an appendectomy could get very different care, depending on which physician was using which appendectomy order set. (Most hospital administrators are trying to standardize care to improve quality and reduce costs.)
3. Order sets need to be maintained regularly, to be kept safe, evidence-based, and efficient. If every doc can make their own order sets, you may quickly end up with many, many different order sets which can be a challenge to maintain, from a technical standpoint. What are you going to do when new guidelines suggest you should be using different drugs to treat pneumonia? How will you find which order sets you need to update? Do you have the resources to keep ALL of your order sets updated? 
4. If every doc can make their own order sets, you will miss out on the opportunity to teach your physicians what informatics is, and how they can improve their own care through evidence-based practice and standardization of processes.

I will admit, as a practicing physician, myself, there are times where I wish I could just make my own order sets. But I will also admit that being challenged on my own order sets is a great learning experience, and ultimately, sharing the discussion with my colleagues, and reviewing the literature is the best learning experience of them all. 

(Apparently I'm not the only one who frowns on personal order sets - A final web page, worth reading even though the author is unclear and this looks more like a comment than a legitimate argument : 

http://www.informatics-review.com/wiki/index.php/Personal_Order_Sets) 

Ultimately, every hospital will need to make this decision for themselves, but remember, as I said - With order sets, there are no free lunches. The rule still applies. :)