Several times, I have seen programs that are very ineffective, yet seem to have high parent satisfaction. Other times, I have seen programs that seem very effective, yet have low parent satisfaction ratings. How does that happen?
I’ve heard you are not a real consultant until you make your own 2 X 2 matrix explaining a phenomenon. So here goes:
We may think that parents, teachers, and other stakeholders base their satisfaction ratings of applied behavior analysis programs solely on how well they work; as if they were buying a toaster. But people don’t even judge toasters solely on how well they work. How easy was it to navigate the toaster company’s website? Did it come when promised? How friendly was the customer support? Was it easy to return a damaged product? Etc. etc.
In my view, stakeholders will judge applied behavior analysis programs based on at least two factors. First, how effective is the program? A program that clearly shows no benefit will be unlikely to obtain high satisfaction ratings. Second, the social skills of the BCBA and other members of the team. Of course, these two factors don’t act completely independently, but are synergistic. A highly effective BCBA probably has to respond to upset parents and teachers less frequently. A highly socially skilled BCBA probably has teachers and parents who implement programming better, and thus has higher effectiveness. See this article for an excellent discussion.
If a BCBA falls low on social skills and low on effectiveness, they will fall into a category we might label as incompetent (Lower left panel). Very few stakeholders will be satisfied with services if this is the case.
But if a BCBA is highly effective, yet has poor social skills, this person might still have poor stakeholder satisfaction ratings. This BCBA falls into the category we might refer to as a jerk (right lower panel). Even if a program shows outstanding progress, that doesn’t mean you can act like Barry from the Bronx. Now, Barry the BCBA from Bethel, Connecticut realized that, and has Poogi-ed significantly. In other words, BCBAs need to develop a variety of social skills in addition to clinical skills. This has been studied in the medical profession. For example, problematic social skills seem correlated with medical malpractice claims.
OK, that makes sense. If you act like a jerk, then parents and teachers are not likely to be satisfied regardless of how effective the program. That might explain effective programs with poor satisfaction ratings. But why would low quality programs have high satisfaction just because they like the BCBA? If the program isn’t working for the kid, people are still going to be upset. Right?
Maybe not. Most programs are not completely ineffective and can tell a good story. The child has made some progress. It is just that a higher-quality program would have made much more progress. That isn’t easy for the average stakeholder to determine. If you don’t have the experience to know what’s missing, you might be satisfied with the minimal progress you are seeing. That’s why I refer to this category as delusional (upper left). There might be no one complaining; everyone loves the BCBA, but that doesn’t mean the services are high quality.
Of course, everyone should strive to be in the upper right panel (Awesome BCBA) with high levels of effectiveness and high levels of therapeutic social skills.
To me, the lesson is that while satisfaction ratings are essential data, they are easily biased by (a) How much people like the staff, and (b) The amount of experience the rater has to evaluate whether the program really is high quality or not. They are extremely useful, but caution is warranted when interpreting those data.