In the US, thousands of people die each year waiting for a kidney, yet thousands of donated kidneys are thrown away. Why? There are many reasons, but one of them is problematic measurement systems. Kidney transplant centers are evaluated on their success rate. They are required to have a 98% success rate. Anything less will lead to serious negative consequences. What happens when they get a donated kidney that isn’t perfect, but it’s pretty good? Let’s say the estimate is a 90% chance of saving the patient’s life. What should the treatment center do? If they want to stay open, they should probably throw it in the garbage. Having a 90% success rate would be devastating to the center’s ratings. There is no penalty for not trying.
It is my opinion that this type of problem occurs all the time in applied behavior analysis programs for children with autism. Most programs set goals and objectives that have a very high chance of success. That’s because we need to report the learner’s progress on those goals and objectives. Having several goals that weren’t successful will look bad on the report, even if there is no formal evaluation system. So, if we aren’t sure we can meet a goal, we often do not include it even though reaching that goal would greatly improve the child’s quality of life. There is no penalty for not having the goal on the report and a possible penalty for including the goal and failing to meet it. This can shape the behavior of whole agencies. Contingencies matter.
This problem can be quite subtle. For example, let’s say a learner at a school or clinic is having fantastic success. The supervisors, parents, and staff are celebrating everything the learner has accomplished. But how do the staff who work with a difficult-to-teach learner sitting just a few chairs away feel? Their learner has made some progress, but may never be able to match the accomplishments of the highly successful learner. Even if the supervisors provide staff with lots of positive feedback, staff will compare their learner’s progress to that of the highly successful learner. There are a variety of management strategies that may help staff morale in a case like this, but it sure is a challenge. This is why I believe many programs struggle to find staff willing to work with the most difficult-to-teach learners, even without formal evaluation criteria. Formal systems can sometimes exacerbate the problem.
These are just a few examples. If you look for this problem, you will see it everywhere. We should evaluate these types of phenomena in applied behavior analysis. It is a bit harder to do the research in ABA than in medicine as we don’t usually have easy-to-measure binary outcomes like death. But just because it is difficult to measure doesn’t mean it isn’t important.
Behavior analysts attempt to measure the quality of ABA services in a variety of ways. That’s extremely important, but we need to focus on how we implement measurement systems. I’m willing to bet on many negative outcomes as a result of well-intentioned attempts at measuring success.