Author: Barry Morgenstern

Often when parents, teachers, or others observe a behavioral assessment, they report that how he behaves in the assessment isn’t how he behaves in real life! There is an understandable concern that the BCBA did not get an accurate picture of the real problem they are facing.

For many years, researchers (and very, very few BCBAs who work with clients for a living) have put kids in a very special situation to assess their challenging behaviors. This situation is called a “functional analysis.” The results were highly useful in designing treatments that dramatically improved outcomes – at least in the same context as the assessment. But how likely is it that the results of the assessment show how the behavior works in real-life situations? Well, we don’t know. That’s just assumed. And we know what happens when you assume. It is quite understandable that no one has researched this question (as far as I’m aware). It would be very difficult to show convincingly that what was found in a highly controlled environment is the same as real-life situations that have so many more variables and are completely uncontrolled.

As usual, I think Don Baer (Pages 304-305) had it right. In 2000, when talking about how likely it would be that the results of a functional analysis would show what happens in real-life situations, he said:

“…. it has a fair chance, I don’t think a high chance, but I think a fair chance of showing how this person’s behavior actually works in the real-life settings where it is a problem. Sometimes it will do that. Sometimes it will fail to do that.”

Certainly, the assessment can capture how behavior works in the real world–sometimes. But looking at the research literature, it seems clear to me that we do have at least some evidence that if often fails to do that. That’s because when tiny adjustments are made to the assessment procedures, we can get substantially different results. In real-life situations, there are an enormous number of things changing all the time. Although it’s very hard to do the research, I believe that Baer’s argument of a “fair chance” is actually optimistic.

On the other hand, Greg Hanley argues that maybe that isn’t such a big problem. We used to think that the mission of assessment was to act like detectives and determine how the behavior works in the real world. Research seems to be suggesting that even thoroughly understanding how the behavior works in at least one important context may be enough to allow us to teach critical skills that will make a socially significant difference outside of therapy. As always, much more research is needed. But I’m optimistic that we can be dramatically more successful with children with severe problem behaviors than we might have been just 5 or 6 years ago before the recent research had been published.

Many years ago, I decided to lose some weight. I was able to lose about 25 pounds and have kept off 20 of those pounds for many years. That’s better than about 95% of people who try to lose weight. Recently, I was able to lose those five pounds again in just two days. How did I do it? Accidentally–with a violent case of food poisoning and lots of vomiting. I was going to post before/after pictures without a blue shirt on, but Cheryl said it wasn’t a good idea. Considering the way that I lost it, by the time you read this I expect I will have gained those five pounds back. Maintaining results does not only depend on whether you achieved the results. How you achieve the results matters a lot, too.

Good behavior analysts know that you have to measure both behavior changes and the results of those behaviors changes. For example, if you want to lose weight you should measure the behavior changes that you want to make (e.g., eat salad, exercise, avoid late-night snacks) and the results (e.g., what the scale says). Likewise, if you want to help a child with autism refrain from severe problem behavior, we have to look at both behavior changes (e.g., how many self-injurious behaviors per day), but also the results of those behaviors (e.g., Does the person have the opportunity to participate in community settings? Do staff or parents have to constantly avoid situations that may make the child upset?). There is a lot of great advice for BCBA’s on this topic. See this book for an excellent discussion.

Although this is well understood in the literature, I believe that often it is implemented very poorly in practice. Why? There is great advice for managing the conflicts between measuring results and measuring behavior changes. But I find the advice on selecting which results to target less satisfying. Yes, the ACORN test is good, just not sufficient to help most of us find the mission of our jobs.

A lot of this blog is dedicated to situations where BCBAs get this wrong in practice. Common examples include attempts to improve training by teaching staff to accept feedback appropriately or improve performance with public posting of data. But if you look for it you will see this problem everywhere. Anyone in management should take care to make sure you don’t have results measures or behavior change measures in place that are creating havoc on the things you truly value.

I once worked at a place that had a strong culture of everyone working many extra hours. Some of these hours were incredibly useful. Sometimes there was work to do for a particular client that required a lot of preparation. Taking the time to really prepare well could potentially make a big difference. But many staff stayed late working on silly projects because you had to look like you were working hard. That’s what management expected. As you might expect, this led to resentment, high turnover rates, and unsatisfied staff.

This problem isn’t as easy to solve as it sounds. Later, when I became a supervisor, I tried to focus on the results of what people were doing, not whether they were staying late or not since I realized what huge negative effects occurred with that other type of approach. That might be fine in some businesses, but it is incredibly difficult to measure what results should be expected from BCBAs working with children with autism. It is very difficult to compare the results of one BCBA working with one set of kids to another BCBA working with a different set of kids.

I ended up with the opposite problem; high performers were angry that some people took advantage of my flexibility. As always, this is on a Poogi–but the closest that I’ve come to a reasonably solution to this problem is to make sure to block out time to have conversations with staff on a regular basis. During these conversations it is possible to set expectations, provide reinforcement, and monitor performance and feedback in a meaningful way that staff usually find reinforcing.

Even if we can’t predict how much progress a particular child should make in the next 6-months to 1-year, we can make very good decisions about what actions are needed in the next week and what reasonable results would be under a short time frame (e.g., 1-week). Of course, you need to actually see it for yourself. Now, with the technology currently available, it is possible to dramatically improve supervision by staff and supervisors watching critical parts of therapy sessions together. Some ABA cultures haven’t caught up to this practice yet, but I suspect that is just a matter of time.

Parents, teachers, and others working with children with autism often want the answer to the question “What should I do when he hits, kicks, screams, bangs his head, etc.?” The answer to that question takes some assessment; but as BCBA’s, we would try to figure out why the child was engaging in those behaviors and tell the parent to just not let him get what he wants when the behavior is happening. For example, if the child tantrums because he doesn’t want to do his homework, then making sure he finishes his homework would be part of the advice. Although that sounds logical and there is certainly a strong research base for that advice, it often doesn’t work in practice, and can potentially make the problem worse–much worse.

We know that virtually all problem behaviors have a logical reason for occurring. In other words, if the child is engaging in problem behaviors, it is likely benefiting the child in some way—usually multiple ways. For example, when the child engages in problem behavior, adults probably do a lot of things to “calm them down.” This might include things like giving them more time in preferred activities, not requiring them to do non-preferred activities, and interacting with the child in preferred ways. Of course, we know in the long run it is likely that some combination of these adult behaviors results in more problem behaviors over time.

Therefore, it seems logical that one of the first things we teach parents, paraprofessionals, teachers, and others is to simply “not give them what they want” or “don’t reinforce the problem behaviors.” When the child is having a tantrum, don’t give them what they want. Simple, right? Now, that’s rarely a strategy that is recommended in isolation. While avoiding reinforcing the problem behaviors is considered a critical component, additional treatment strategies like teaching communication and reinforcement for appropriate behaviors are almost always included in behavior plans.

If the above is done in a laboratory or highly controlled setting, it works. It works extremely well. But in practice, it often doesn’t. That’s because when the problem behavior isn’t effective for the child, the first thing that happens is the intensity increases. The common warning is “It gets worse before it gets better.” In practical settings, this almost always results in at least some compromise on the part of the adults, at least occasionally. That is much worse than just giving the child what they want right away. Now, you have taught him unless you take it to 11, you don’t get what you want. It also can ruin rapport, relationships, and get kids kicked out of schools, daycares, and other settings. And there is now research that suggests maybe that approach isn’t needed.

Early in my practice, I was very skeptical that this was an essential component of treatment. I wrote a paper for publication (which was rejected) arguing that the research didn’t fully support this practice. All the research showed is that when the reinforcement for engaging in problem behavior versus the preferred behavior was equal, treatment usually wasn’t effective. Reinforcing problem behavior less efficiently than preferred behavior might be effective, but it hadn’t been tested. The paper was thoroughly skewered by reviewers. I took that rejection in stride and had long forgotten about the idea after the paper rejection. But many years later, other researchers demonstrated what I had only hypothesized. It turns out my early hunch was exactly correct; you may not have to completely avoid reinforcing problem behavior. I made a mistake giving up on the idea because it is incredibly useful in practice.

This is an exciting finding. One of the most difficult to implement procedures, which causes huge negative side effects and produces negative responses from parents, teachers, and other non-professionals, may not even be necessary. Of course, more research is needed. We don’t know if this would work in every case. Still, teaching skills that work in the real world is what matters. In most cases, a bit of reinforcement for problem behavior while teaching new skills will not ruin the effects. It might even be helpful if it keeps kids in school, prevents injuries to clients or staff, and maintains rapport with clients and other stakeholders.

We typically want a simple answer to the question, “Is treatment effective?” Ideally, the answer is “Yes” or “No.” But people will accept a graph as the answer to that question. When a school district, attorney, parent, or insurance company wants information on the effectiveness of treatment, BCBAs are likely to respond (appropriately) with graphs of the relevant behavior changes. In most situations, these types of answers with the visual display will be readily acceptable:

Question: “Was the treatment for self-injury effective?”

Answer: “Here is the graph for self-injurious behavior collected over the previous six months showing a 93% reduction in self-injury from baseline levels.”

Question: “Was the teaching program to answer questions starting with “where” effective?

Answer: “Here is the graph showing baseline levels at 0% correct. After treatment, data demonstrate 100% correct on 3 consecutive sessions.”

Even assuming the data are completely accurate (which is a very questionable assumption in many environments), there are still a very large number of ways the data can be completely misleading.

1. The treatment was temporarily effective but doesn’t last over time.
2. The treatment was effective, but staff or parents are using interventions that make it look better than it is in reality.
3. Data was collected in a way that while accurate, did not practically measure the skill.
4. The treatment was effective but caused other negatives to occur.

There are other possible ways for data to be misleading, but an exhaustive list is beyond my scope for today. The key point is that having graphs of your behavior changes is essential, but it is not sufficient. We need to make sure that we have really made a socially significant difference. Sometimes the data on behavior changes will look great and the program is still a failure.