For years, O&P professionals have been debating the benefits of outcome measures while struggling to get a grip on how best to implement evidence-based practice. Today, it appears that although most O&P professionals have been convinced of the need, barriers continue to prevent wide-scale implementation of a regularly practiced, structured set of outcome measures.
A University of Washington, Seattle-based team has stepped into the breach, however, applying National Institutes of Health (NIH) grant funds to a five-year program dedicated to identifying and overcoming barriers to outcomes measurement. The plan is to create novel measurement tools and methods that are easier and more convenient to use for clinicians and patients alike.
Initiated in March 2010, the project is under the auspices of the University of Washington Center on Outcomes Research in Rehabilitation (UWCORR), Seattle. Brian J. Hafner, PhD, assistant professor, Department of Rehabilitation Medicine, University of Washington, heads the project.* Hafner and members of his research team are hosting a symposium about the project at this month's American Academy of Orthotists and Prosthetists (the Academy) Annual Meeting and Scientific Symposium. Titled "Advances in Outcome Measures: Computer Adaptive Testing," the symposium describes the project as having developed in response to an NIH request for applications to create and test systems to measure the success and effectiveness of O&P interventions. Louis Quatrano, PhD, at the Eunice Kennedy Shriver National Institute of Child Health & Human Development (NICHD) is the project's scientific officer.
"NIH's intent was for investigators to capitalize on and build upon the existing research and advances in measurement that have been available in other fields—and possibly to use some of these resources to design, develop, and test outcome measures that would work in O&P," Hafner says. "We don't have a lot of them."
In practice, these new outcome measures could be used to gauge the success of an intervention, make decisions about changes that might be made to benefit a patient, provide objective evidence about the intervention's effectiveness, and ultimately improve treatment for those who require O&P products and services.
Most O&P professionals recognize that it's their responsibility to understand how and why patients respond to the interventions they provide and to ensure that patients benefit from such applications, Hafner observes. So why aren't more clinicians using measurements that help to determine the effectiveness of their treatment protocols?
"Is it too challenging to find and select the right outcome measure? Are they too time-consuming to administer? Is the information they provide not valuable to the practitioner or to the patient?" Hafner questions. "I would suspect that the truth probably differs for each practitioner in each setting, but I think these are some of the reasons why measuring outcomes has not gained the foothold in O&P that we might like to see at this point. The availability of valid and reliable measures and proper instruction in how to use them can be a problem, and finding the time to collect, analyze, and interpret the data that you get from using specific instruments can be a challenge.
"It's one thing to administer a test, but then you have to score it, you have to make use of that information, and you have to make a meaningful decision for your patient based on the results—you have to find value in the information that's obtained from the tools," Hafner explains. "These are primary barriers to outcomes assessment."
Team member Robert S. "Bob" Gailey Jr., PhD, PT, associate professor, Department of Physical Therapy, University of Miami Miller School of Medicine, Florida, whose doctorate degree is in outcomes measurement, identifies additional barriers: "We have yet to identify the appropriate outcome measures to differentiate between prosthetic devices. There are measures to differentiate between amputees' functional capabilities, but not between componentry."
Hafner agrees that many existing O&P outcome-measurement instruments are flawed, limited, or both. Some are not sensitive to important changes such as the patient's transition from one type of device to another. "For example, an amputee who has changed from a mechanical knee to a microprocessor knee might report using a measurement instrument that shows no difference—although the patient tells me that they feel a difference."
Gailey adds, "Clinicians must understand and acknowledge the need for outcomes measures, and then participate with every client that comes through their door. Federal law mandates that it's the responsibility of the clinicians, not researchers, to demonstrate the value of the treatment they apply. Third-party payers will expect [practitioners] to produce some level of outcome measure toward their patient's treatment, and without having that data behind them, prosthetists run a risk of lower reimbursement—or no reimbursement at all.
"For a long time, people believed that participating in evidence-based amputee care would take away the clinician's choice, and that's not true," Gailey continues. "Evidence-based rehabilitation allows you take your clinical experience, the patient's condition, and the amputee's desires—as we have traditionally done—to hone in on the best form of treatment for the patient. If we then take the evidence and lay it on top of those three areas, at the end of the day we will have arrived at the best-known method of treatment for that patient.
"Clinicians have to find the time…to participate," he concludes. "It's our goal to try to develop new and more efficient measures that overcome some of these limitations—measures that can be conveniently used to inform clinical practice, research, and even policy decisions regarding O&P interventions and treatments."
To that end, the team is currently developing a patient-reported measure called the Assessment of Personal Mobility (APM).
"You can't design a patient-reported outcome measure without asking the patients how they feel about it," Hafner points out. "We certainly believe that the patient's perspective is under-represented in the assessment of outcomes, and we are trying to design tools that can carefully and accurately incorporate their input into the clinical decision-making process. Thus, our work involves the development and validation of what's called an item bank, or a collection of questions focused on the mobility of users with prosthetic limbs. Our intention is to carefully develop the item bank, using input from consumers or users of prosthetic limbs, as well as those experts that we have assembled for the grant team."
The measure will undergo a lot of testing and multiple revisions to ensure that it clearly and concisely provides accurate and meaningful information on patients' perception of their mobility. Focus groups, cognitive interviews, a large-scale cross-sectional survey, and ultimately a longitudinal trial will be conducted over the five-year period in order to assess and confirm the validity of the measure.
Computer Adaptive Testing
Another novel method the team is incorporating into its project is Computer Adaptive Testing (CAT), a computerized administration process that presents questions in an adaptive sequence. The underlying algorithms use knowledge of the previous response to select the next appropriate question for the respondent, skipping over those questions that can be eliminated as inapplicable to the current case.
The use of CAT can produce a final score far more quickly than asking the respondent to answer the complete set of questions, Hafner notes.
"We believe that tools like this can reduce the burden not only to the patient, but also to the practitioner, and ultimately allow for the easier collection of outcomes in that busy clinical environment. We're trying to think outside the box in ways to make the tools accessible and easy to use, including different ways to administer measures, whether it's on a short form or a CAT on a tablet, kiosk, or computer terminal."
By using novel, web-based programs, Gailey says, "We'll be providing clinicians with better insights into the performance of the amputee through more clinically friendly measures. We're also looking at ways they might be able to partner with other clinicians; i.e., working with physical therapists to help with the assessment of the amputee and the outcome for the amputee."
From the beginning, the team realized that it was impractical to try to develop a universal outcome measure that would address the needs of the O&P industry across the board. It chose instead to focus on a measurement area (mobility) and a patient group (persons with lower-limb loss or limb difference). "This is an area of outcomes assessment we felt was particularly important," Hafner says, "as well as one in which we had some expertise and felt we could address in a project this size. In the United States alone, there are more than one million people living with lower-limb loss, and we felt that there's a real need for better ways to assess outcomes and treatments provided to these individuals."
Since achieving and improving mobility are key goals for many O&P interventions and are especially important for lower-limb prostheses users—determining how readily they can return to their homes, communities, and vocations—mobility was selected as the team's research focus.
"In O&P," Hafner notes, "mobility is also used as a key metric for making clinical decisions about the type of intervention or the device selected; reimbursement agencies also often use the individual's potential for mobility as one way to determine which devices are made available to that patient."
Making It Practical—and Practicable
As a researcher collecting outcomes in a controlled laboratory setting, Hafner recognizes the difference—and the difficulty—of finding outcome measures that work in a clinical environment and contribute to improvements in the health and care of patients. Therein lies the challenge, and the basis, for the project.
"We wanted to be sure we had input from stakeholders all across O&P. Obviously, if you're going to create an outcome measure, you don't want it to be just another one that ends up in a textbook. You really want it to be one that can be adopted and respected," Hafner says.
So the group assembled an advisory panel of recognized experts in O&P and rehabilitation research, clinical care, and measurement theory, including representatives from key organizations like the U.S. military, the Department of Veterans Affairs (VA), large O&P manufacturers, and large O&P clinical service providers.
"The goal in creating this advisory panel is to be sure we really solicited the expertise, experiences, and opinions of those who have been involved in similar efforts and in developing outcome measures, as well as those who would have the potential to use the tools that we develop—and to make sure that we did it right."
As part of the process, Hafner's team started reviewing some of the other measurement tools used by other allied healthcare providers.
"We've been reviewing other existing outcome measures that include items or questions related to mobility. To date we've reviewed more than 40 other self-report measures and have pulled nearly 1,000 different items or questions from them to review and consider for inclusion in our measure. Obviously, as you pull items from different measures, they all ask the questions in different ways; the challenge is to make sure we have all the right concepts in our measure and that we can really span this continuum of mobility from the low end to the high end. We'll systematically evaluate all these questions with the help of our advisory panel and the results of our focus groups and cognitive interviews to choose the items that we believe will function the best."
What Have They Discovered so Far?
Mobility is complex and the users' experiences present some real challenges, Hafner notes. "Different factors influence different people. Ask somebody about mobility, and they might think that pain is what drives how well they can be mobile or not mobile. For some users it's all about the economy of planning out the activities in their life. Some might say, ‘I can walk as far as I want to today, but I'm going to pay for it tomorrow.'?"
Because patient priorities are so variable, the research team is trying to capture as many of those individual preferences as possible so that the measure, "asks about all the right areas," Hafner explains.
"Our hope is that by creating this measure and sharing the results of the research in this way, we'll allow clinicians and researchers to better understand O&P interventions and the impact they have on the lives of the users."
Hafner shares his long-term vision of a world where everyone contributes their findings to a database or central repository of information where patients can see how they compare to other patients, not just locally but on a national level, and clinicians can access outcomes on a larger scale to which many practices with small patient populations have contributed. But that's a topic for the next grant project.
Judith Philipps Otto is a freelance writer who has assisted with marketing and public relations for various clients in the O&P profession. She has been a newspaper writer and editor and has won national and international awards as a broadcast writer-producer.
Author's note: UWCORR is one of six NIH-funded centers in a national network of research centers called the Patient-Reported Outcome Measurement Information System (PROMIS). The measure that Hafner's UWCORR group develops will be made available through the PROMIS Assessment Center (www.nihpromis.org) at no cost to participants.
*The described research is supported by the National Institute of Child and Human Development (NIH grant number 1U01HD065340-01, "A Client-Based Outcome System for Individuals with Lower-Limb Amputation").
