Quality over Quantity

Obviously, when we want to know how well a child is doing, we want to see the data. The catch phrase of a BCBA is often “show me the data.” The problem is that sometimes those data are not summarized in a very useful way. In many cases, data summaries are downright misleading. For example, if we want to know how a particular child is doing, we are probably not going to look through dozens of graphs or hundreds of pages of raw data, we need a summary. Summaries often include things like this:

  • Problem behaviors are down 90%
  • 50 New Words Learned
  • etc.

Now, sometimes when I’m doing an evaluation, and I look closely at these types of data, I find that the data were collected very poorly, or that the staff can’t replicate these changes during an observation. But today, I’m not talking about bad data collection. Today, I’m assuming that these data are 100% accurate and reliable.

The problem is they may or may not be very meaningful. The problem behaviors may be down 90%, but only if the preferred staff is there, only if the team dramatically reduces the math presented, and never says “no” to requests. The child may have learned 50 new words, but only uses them if you practice in a distraction-free area, remind him to “get ready” before asking, and provide tangible rewards for answering.

In order to tell if something is meaningful, you need the generalization data. That’s what is important:

Problem Behavior (Examples)

Does the child still have low levels of problem behavior when:

  • The regular staff person is out sick?
  • When there is a fire drill?
  • When there is a change in schedule?

New Words (Examples)

  • Does the child use the words in spontaneous speech? When novel pictures are used?
  • Do novel people understand the new words?

As a general rule, it is much better to make fewer behavior changes, but choose behavior changes that truly make a difference. Make sure that each behavior change is really useful to the learner. Don’t get carried away with absurd numbers of goals or graphs.

It is easier to measure the quantity rather than quality. It can look impressive–50 new words! But in the long run, the quality of the behavior changes is what makes the difference in the learner’s life.

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.

A Hidden Problem in Finding Natural Contingencies

A common complaint from parents and teachers is that the child is doing “great” in therapy, but they don’t use the skills they learn in therapy “in real life.” Now, there might be a lot of reasons why a particular skill occurs in therapy, but fails to generalize to natural environments. I won’t try to cover them all here as there are too many possibilities, and most of them are well known by BCBAs. Today, I’m just talking about one of those possibilities because it is not well known, easy to overlook, and often the data hides the problem. The problem may be that the child is just not very good at the skill, despite what the data seem to show.

For example, remember when you were first learning to drive. At the beginning, you had to think about everything. Put pressure on the gas, where to hold the steering wheel, and when to push the break. Of course, we all realize when you are driving with someone at that level of skill, you should probably be afraid. Now, you don’t do any of that. You just get in the car and drive; you aren’t thinking about it any longer. When you reach a high level of skill at a task so that you can perform it beautifully without thinking, you have attained what is called fluency.

Many times, I’ve seen children who have “mastered” a particular skill, but actually aren’t very good at the skill yet. A common example involves children who have learned a skill through discrete trial teaching. This might be anything from labeling pictures, pronouns, prepositions, or a wide range of other possibilities. Often, a skill would be considered “mastered” if the child achieves 100% accuracy for a few sessions. We should realize, though, that this is the equivalent to student drivers who are still thinking about everything they are doing. Although they can drive, they aren’t likely to do well under difficult “real-world” driving conditions. Likewise, not reaching fluency is often a major reason why a child can perform a skill while in therapy, but does not generalize the skill to “real life.” This is true for a wide variety of skills in all the relevant domains–social skills, language skills, motor skills, play skills, and more.

The subset of Applied Behavior Analysis practice that emphasizes this type of work is Precision Teaching. For more information, my friend Rick has written the book on the topic.

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.

When Social Skills Programs Fail

Once, I was at a school observing a program where a child was engaged in a social skills program to learn conversation skills:

Student (reading from a script): “What are you going to do on your vacation?”

Conversation Partner: “I’m going to Miami.”

Student: “What are you going to do in your Ami?”

I think this is a fascinating response. Not just because it is an amusing anecdote. But because it highlights a fundamental problem with the way social skills and language instruction is often conducted.

This student clearly had clearly learned a social “rule.” Something like, “After the initial conversation starts, ask a follow up question.” The problem here was the student didn’t seem to have any interest in having a conversation. He didn’t care at all about where the conversation partner was going on vacation or even talking to that person at that time. If he really wanted to know about his conversation partner’s vacation plans, his follow up question would have been “What is an ‘Ami’?” The student was working with a motivation system where if he completed a series of activities, he would earn a reward. He was just trying to finish his activities to get the reward.

Now, there is a lot of literature with different approaches to attempt to solve this type of problem. As always, more research is needed. But the key lesson for practitioners is that just getting the response to look right doesn’t mean that you have taught anything useful. Probably, the student in the above scenario would have looked great 90% of the time. The school could have shown many of his conversations at conference presentations as examples of how well their social skills program works.

But we know better now. Someone may learn to have a superficial conversation and look appropriate. But surely it won’t generalize to natural situations or friendships unless they are engaging in conversation because they care about the answers, not simply earning the contrived reward at the end of the conversation.

Behavior analytic services should only be delivered in the context of a professional relationship. Nothing written in this blog should be considered advice for any specific individual. The purpose of the blog is to share my experience, not to provide treatment. Please get advice from a professional before making changes to behavior analytic services being delivered. Nothing in this blog including comments or correspondence should be considered an agreement for Dr. Barry D. Morgenstern to provide services or establish a professional relationship outside of a formal agreement to do so. I attempt to write this blog in “plain English” and avoid technical jargon whenever possible. But all statements are meant to be consistent with behavior analytic literature, practice, and the professional code of ethics. If, for whatever reason, you think I’ve failed in the endeavor, let me know and I’ll consider your comments and make revisions, if appropriate. Feedback is always appreciated as I’m always trying to Poogi.