Answering Questions When There Are No Scientific Answers

It was my first day of a new job. I walked into a super-confrontational meeting between parents and a school district. The discussion was heated. The district was very unsatisfied with the current behavioral consultation and the parents were extremely happy with the current consultants. During the discussion a school district leader asked me, “How long will it take for you to learn about this student and take over the consultation?”

That was a very tough question. There were about 15-20 people in the room. I had met a few of them earlier in the day. I didn’t know much about the student besides what I heard during this difficult meeting. I don’t even know what the rest of my caseload will be like and how much time I can devote to the case. If I say it will take quite some time, I’m sure to upset school district leaders who are furious with the behavioral consultants. If I say I can do it quickly, I’m sure to upset the parents who I haven’t even met outside of this heated discussion. Everyone is emotional and on edge. What to do?

“Well, this is my first day and I think I can remember where the bathroom is located.”

After that, I was able to talk a bit about how I would get to learn about the student without making any specific commitments about when I would complete the project. I was able to build rapport with the parents and school district leaders in subsequent weeks. Many other times, I haven’t been so lucky.

The problem is that as BCBAs we are supposed to make recommendations based on the scientific literature. Often, however, we are asked to make recommendations where it isn’t easy to point to the literature that would justify specific recommendations.

  • How many hours of consultation per week are required for the student to have an “appropriate” program?
  • Does this student need a 1-1 Registered Behavior Technician in order to have an “appropriate” program?
  • Is this goal “medically necessary”?

We can certainly interpret the scientific literature to make recommendations like this even if there is no study exactly on point. We can collect data to inform these decisions. We can and should do both of those things.

I would suggest that in addition, it is important to realize that in the real-world, decisions like these are often made on the basis of arm wrestling. It is more like buying a used car than a debate on the science. Sure, data and scientific literature can help, but it usually only has a small impact on how decisions are usually made. That’s because many times there is no obvious science-based answer to these types of questions and different BCBAs are likely to disagree. Knowing the science and how to collect data to evaluate decisions is necessary, but not sufficient to handle these situations well. Learning to negotiate the complex social environment is essential too.  

 

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.

Hopefully, Most BCBA’s Aren’t Still Making Mistakes Like This

I did an evaluation where the child had an ABLLS (The Assessment of Basic Language and Learning Skills) completed. I noticed all the sections on answering questions were all checked off as mastered (i.e., who, what, when, where, which, why, and how). But when I met the student, and I started to ask him some questions, the student seemed incapable of communicating anything that I could understand. I thought he might be nervous or feeling uncomfortable, but he sure didn’t look nervous or uncomfortable. He was laughing, seemed to want to socially engage, and generally seemed to be having a good time. 

 

The parent was present and so I took her outside to ask if I was doing something wrong. Why doesn’t he seem to be able to respond to anything I’m asking? The ABLLS seemed to suggest that he has these types of language skills. The parent reported that I was asking questions that aren’t mastered. She then pulled out lists of all the mastered questions. Sure, enough, if you asked him questions from the “mastered list,” he was very successful at answering them. 

 

The ABLLS has items on it like, Answers “Where” questions, which is evaluated on a scale of 1-4. You score a 1 if the child answers 5 questions, a two for 10 questions, a 3 for 25 questions, and a 4 for 50 or more questions. The parent took out a list of 50 questions that were completely “mastered.” The student had completed hundreds, maybe thousands of hours of therapy to learn all of this, yet clearly, a lot of it was not very useful. 

 

I’ve sometimes used this story as an example of how easy it is to completely waste therapy time and still have the data look good. This program had pages and pages of graphs of many, many behaviors that had been taught. So, the evaluation using the ABLLS looked good, as many of the boxes were filled in at the highest level of mastery. 

 

The typical response that I’ve heard is that this isn’t a generic problem, simply an example of bad programming from someone not very well trained. Sure, this particular example is extreme. But I’ve seen similar problems come up over and over again. Are you focused on things that will look good temporarily or things that will make a real difference for the client?  

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.

Important Behaviors Used Rarely

I once visited a program where I had previously worked but hadn’t returned in over five years. One student I worked with frequently was about 12 years old when I had left and was about 17 when I visited. When he saw me, he said, “Hi, Barry,” and continued with his work as if he saw me every day. The staff thought it was hilarious.  

 

Apparently, I had made enough of an impression that he remembered my name after five years, but not so much that he was interested in talking to me after so much time had passed. Running into people you haven’t seen in a very long time is an unusual event. It doesn’t come up very often, so the opportunity to practice appropriate social skills for that event are rare. This is not a particularly important event, but some others might be. 

 

For example, many people with developmental disabilities have been victims of abuse. There are many programs designed to teach individuals the appropriate skills to prevent themselves from becoming victims of abuse. Parents of teenagers with developmental disabilities are often extremely interested in programs to address this issue. These skills can be quite challenging to teach. Sure, you might get acquisition, but teaching and assessing whether these skills are generalized to the point where they would be successful under real-world conditions is an extremely challenging task. Most of the time, when I’ve seen people attempt to teach these skills, they do not even attempt to get the generalization data to know if it is likely effective. It is just so hard to collect. Plus, there is little or no opportunity to practice these skills regularly. Even if you were extremely successful teaching these skills, what are the chances that the student would use those skills well if there was an occasion to use them in say, three years? Most skills will deteriorate without practice. Should we have a constant maintenance schedule for a skill we hope the person never has to use? 

 

In my view, there is a place for teaching these types of skills for the right type of learner, for the right time in their development. Is this one of the most critical items for this learner now? If so, definitely teach these skills. Just don’t do it to check off a box on someone’s agenda. If you are going to teach abuse prevention skills, understand it is a major challenge that will require significant time and attention. Spend the time to do it right. If you don’t have the time or resources to do the generalization and maintenance, it’s probably better to not waste everyone’s time. 

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.