A few recent events make me want to revisit the concept of measuring efficiency. These events include reading a federal document on testing the usability of electronic health records (EHRs) and recent visits observing doctors and nurses in four settings: two medical offices, an emergency room, and a hospital room.
In the federal document, efficiency is defined as time on task measured in seconds. Two examples from this document are: 1) time to update blood pressure on a particular screen should be operationalized to something like “less than 20 seconds by 90% of first-time users,” and 2) time required to enter data on a prescribed medication with drug-drug interaction should have a goal something like “successful completion in 45 seconds by 95% of users.”
This is the most restrictive definition of efficiency I have ever come across. Most people will state that efficiency can be measured in terms of time, keystrokes, mouse clicks, or screens. Unfortunately, none of these directly measure efficiency. These are, at best, external criteria that may bear on task performance under certain conditions.
Certainly time is important to doctors who have to get patients in and out quickly, but absolute time on task by itself doesn’t matter in most cases. That is, not without an externally defining limit. We humans don’t biologically keep track of time. If we did, we wouldn’t have to wear watches. Sure, if something takes so long we start to notice the time it is taking, it’s an issue. But it’s only an issue in that case because we noticed how long it took.
There are exceptions. Time is the correct measure for task performance when there is an externally defining event that sets the maximum task time. For example, if there is a missile on the way, and you have ten seconds between the time the radar detects its presence and the time of safe intercept, then a task time of less then ten seconds to make a countermeasure launch decision makes sense. Perhaps less dramatically, if you’re buying voting systems, you may want to know the average time to complete a ballot to decide how many machines to buy to ensure lines stay short (though this is not a user issue as much as an administrator issue, and determining all of the variables in this case is rather difficult).
But you’re praying before a false prophet to assume that task time is the correct—-let alone the only—-measure of efficiency. When you’re sitting in your pajamas at 2:00 a.m., looking for a book on Amazon, clock time in seconds is not all that important.
Other things like keystrokes by themselves don’t usually matter, either. As a colleague of mine cleverly pointed out, you can make any system operate with a single keystroke per task if you just develop a large enough keyboard—-one with a single key for each function. It may take ten minutes to find the key, but it’ll only take one keystroke to perform the function, once you do.
For an EHR, even if these task times are based on something like the time it used to take to perform these tasks on paper, time alone is not a complete measure of efficiency. What really matters for efficiency in this case is not time in seconds, or the total number of keystrokes, or mouse clicks, or the number of screens, or some complicated combination of all of these. What matters for an EHR, and for many systems, is the ability to perform the task while monitoring a situation (the patient). This efficiency is measured by the amount of mental effort required to perform the task. And recall that mental effort is at two levels: conscious effort and unconscious effort. What we need to strive for is the least amount of conscious effort.
Some years ago, I sent a briefing to a colleague, an old-school developer who is quite knowledgeable and comfortable in the world of Unix. I happened to run into him the same day and asked if he had looked over the briefing. He said he hadn’t checked for it yet. As I stood there, watching him in front of his Windows-based computer with Outlook clearly visible as an icon, he performed the following tasks to get the briefing:
- He opened a terminal window and used telnet to get to the server.
- He used uuencode to encode the file.
- He transferred the file to his PC using FTP.
- He used uudecode to reassemble the file back into its original format.
- He closed the terminal window, went to his desktop to find the file, and opened it.
After witnessing this process, I asked why he didn’t just open Outlook and click on the attachment in his mail. His response was clear, and correct: “Why? That was easier.” And, to him, it really was. And I know it was because he did all of this while he and I held a conversation!
Based on the levels of concentration of the doctors and nurses using their EHRs that we observed in our visits, you might think their patients could have asked questions, uttered sounds of pain (or sadness at being ignored), or started to turn blue, and the doctor or nurse may not have noticed, let alone been able to talk to them. It is unlikely they would have been able to continue making EHR updates and been able to pay attention to the patient at the same time. The level of concentration required by the EHR was far too high for that. (Recall the moonwalking bear from the February 2014 newsletter and what happens to our ability to process information as our cognitive load increases.) Focusing on task time as the criteria for an EHR encourages designers to make short task times a priority. In the real world, entering data is actually a second task while the doctor or nurse simultaneously tries to maintain awareness of the patient. Designers should be striving to make that as efficient as possible.
It’s true that measuring time is easy, as is measuring keystrokes or mouse clicks or screens. But measuring the easy stuff is not good enough. Designers need to look at the real measure of efficiency in interaction design. It’s not impossible, but it won’t be easy to measure. However, measured or not, it will be possible to design for the least amount of conscious attention if we understand that it’s the goal.
Testers and authors of standards should stop praying to the false prophet of time, keystrokes, or screens as a measure of efficiency. Forget about setting 20-second or 45-second time limits for tasks in EHRs. Forget about attempting to follow the “three clicks” rule for websites. (Yes, there is such a rule. It’s even described in Wikipedia.) For an EHR, it would be better to define a task in terms of task success while maintaining awareness of the patient. Once that’s achieved, then try to optimize the time. But aiming for 90% of first-time users being able to update blood pressure on an EHR screen in less than 20 seconds is not real efficiency. It’s likely to get you a simple thing to measure but a bad system to use.
