Author: Barry Morgenstern

In 2016, Behavior Analysis and Practice published an excellent article on the measurement of problem behavior called “A Proposed Model for Selecting Measurement Procedures for the Assessment and Treatment of Problem Behavior.” Blanc, Raetz, Sellers, and Carr present a model that behavior analysts can use to decide on the appropriate measurement procedure during the assessment and treatment of problem behaviors. The model captures many of the practical problems that can occur when selecting a measurement procedure, and is a useful contribution to the field. The article goes through all the basics on how to measure whether problem behavior is happening and how much it is happening. That’s obviously essential data to have, and the authors do an excellent job of going through a variety of problems that can occur when selecting a measurement procedure.

When I read this a couple of years ago, the first thing I thought about is how useful this model might be for training new behavior analysts. But second, I thought about how much more is needed to be successful in any practical situation. A good measurement procedure is necessary, but not nearly sufficient for practitioners. Obviously, including everything needed for a successful outcome would go way beyond the scope of this brief article. I’m not talking about successfully treating problem behaviors; just measuring the effects of treatment is a difficult undertaking.

So, what else is needed to measure successful treatment of problem behaviors besides knowing if the problem behavior is happening or not? In my view, at least five other measures are needed:

What else is he or she doing? –The Sit Down and Be Quiet Problem

Often, one of the primary outcomes that people want in a program based on the principles of applied behavior analysis is the absence of problem behaviors. Of course, that can be an important outcome, but it certainly matters what the person is doing instead of engaging in problem behaviors. Everyone is happy when significant self-injurious behavior or aggression is eliminated, but if the person is sitting around doing nothing much of interest, it likely isn’t a very valuable outcome. It is also essential that the person is doing something else that is beneficial to their well-being. Are they engaging in valuable communication, social skills, academic, or self-help skills? If not, it is hard to argue that much value was accomplished from the treatment.

What are the staff doing? –The Child Effects Problem

In research, what the staff or parents do during the treatment is very carefully monitored, but this rarely happens in practice in schools or homes. In the real world, staff usually have to make some “judgement calls.” This often leads to staff doing things to prevent problem behaviors that may or may not be in the child’s long-term best interest. Often, the child shapes the adult to avoid things they don’t want to do. The literature refers to this problem as child effects. If you don’t know about these things the staff are doing during treatment, maintenance and generalization of treatment effects is extremely unlikely. Therefore, it is absolutely essential to have some measurement of how well the treatment is being implemented. Not just for quality control, but to help with generalization and maintenance of the treatment effects.

Will this last over time? The Maintenance Problem.

A very common problem in the treatment of problem behavior is that at first, things work great, but over time, the behavior doesn’t last in the long run. From a measurement perspective, the problem comes when people say things like, “Since we haven’t seen any problem behavior in six months, we don’t need to measure problem behaviors anymore.” In about 99% of cases, people gain the weight back after the diet. We don’t really have data on how often the problem behavior comes back after successful treatment, but I can guarantee you it is a lot. Behavior analysts need to incorporate a maintenance plan with good data collection procedures in their programs.

Will this work in the real world? –The Natural Contingency Problem

There are many potential ways to reduce a problem behavior. Often, behavior plans are implemented over very long periods of time. That is appropriate and necessary most of the time. But eventually, parents and teachers want to live life without having to worry about managing the behavior all the time. If we want to achieve long-term maintenance, the child has to engage in an alternative behavior for a real-world reason that will meet a natural contingency. In other words, the appropriate alternative behaviors will be reinforced even if no one plans it out in advance. Eventually, we need to measure how the real world responds to the child’s alternative behaviors.

Are there hidden reasons treatment will fail? –The Social Validity Problem.

Not everything can be measured with objective data. For example, the teacher doesn’t agree with the treatment procedures; Grandma won’t follow the protocol; it isn’t practical at church; the paraprofessional hates the plan and does her own thing when no one is around. It is essential that we get the honest subjective judgements of all the people involved in the treatment. All the objective data might be showing great results, but if the important people in the child’s life still have significant concerns, it is only a matter of time before those issues start to show up in the objective data too.

In conclusion, the article “A Proposed Model for Selecting Measurement Procedures for the Assessment and Treatment of Problem Behavior” is a great place to start on measuring treatment for problem behavior. Just recognize that there is a lot more needed to be successful in practical settings.

A team I was working on many years ago was teaching a young child with autism who had minimal language how to talk. One of the things he had learned to spontaneously request was “Oh, no!” When the therapist heard, “Oh, no!” he or she would fall down. It was a highly motivating and fun way to get him to use language. A couple of weeks after he learned this, an evaluator was coming to assess the student. The child ran up to the evaluator, and as the person was saying, “Hello I’m doctor so-and-so,” and extending a hand, the child was enthusiastically saying, “Oh, no!” When the evaluator didn’t respond correctly, the child had a major tantrum that disrupted most of the evaluation.

I think this experience can teach us some important lessons. Some obvious, but one not so obvious. Let’s start with the obvious lessons. First, I knew the evaluator was coming to see this student. There are lots of things I could have done to prepare. In particular, warn the evaluator ahead of time so the evaluation time wasn’t ruined. Second, whenever you are teaching requests, part of the procedure must be to teach the student that the requests are not always available, otherwise problem behaviors are inevitable.

But some people who have similar experiences learn what I think is the wrong lesson. Specifically, some will argue that is absolutely essential right from the beginning to ensure that the child can be understood by novel people to avoid these types of problems. I’ve heard this used as an argument against games like my student learned. I’ve heard this used as a rationale to put an immediate and major focus on articulation. I’ve heard this used as an argument against teaching sign language. A common argument against sign language is, “How will he be able to make a purchase at a store?” “How will he be able to talk with peers in class?”

These are real problems, but I don’t think we need to worry about them too much. Our goal should always be to teach skills that will meet a natural contingency, which certainly means we have to reach the point where novel people can understand. The way to do that is to teach the child to be a sophisticated communicator. If you do that, you will overcome these types of issues easily. Sometimes, it may be possible and beneficial to start out immediately with procedures that novel people can understand (e.g., iPads, PECS, articulation training), but I doubt that means we always have to do that immediately. In my view, there isn’t sufficient research to make a definite determination for every case, and currently it requires some clinical judgment.

My student who started talking by learning to say “Oh, no” became a fairly sophisticated communicator. Whether other procedures would have achieved more, better, or faster is hard to say. But always avoiding these types of activities is a big mistake.

There is a tremendous amount of excellent advice published about effective use of Telehealth for BCBAs. I don’t intend to cover that here as there are many great resources to turn to for those looking for advice. Today, I’m interested in discussing some reasons why Telehealth might actually be an improvement over traditional in-person therapy in some cases.

Many BCBA’s I talk to are absolutely astonished by how amazingly well Telehealth services have gone in some cases. As my friend Alan teaches, when you are surprised, that means that there is something that you didn’t understand about reality. I think it is worth taking the time to think this through. Why is it that I (and many other BCBAs) did not expect Telehealth services to go well, yet in some cases things have gone better than we could have imagined?

Probably, many predicted that Telehealth services would not go well because of the obvious limitations of things you can’t do on-line. Things like the BCBA can’t model during training, the short attention span of some children, and difficulty getting a complete picture of what’s happening online. Let me emphasize that those are valid concerns in many cases, and that in-person services are essential much of the time. In some situations, modifications can be made to make Telehealth reasonably effective.

Given all the issues above, I never would have suspected that Telehealth might actually be a tremendous upgrade in the quality of the services. But it can be, particularly when the primary service being delivered is training. This might happen for a few reasons:

1. Increased Frequency and Duration of Services:
   Often, if I’m doing training for a parent or therapist, I might only see that person once per week, or even every other week. But online, it is no problem to schedule sessions several times per week, or even every day. If I’m providing training, and I have to drive to the client’s home or school, I’m going to want to stay awhile to make it worth the time. I’m not likely to drive 45 minutes to an hour to do a 45-minute session. But in a short session online, we can often accomplish a tremendous amount. In summary, if I have three hours per week to work with a client, I’ll get much more done four 45-minute sessions than in one 3-hour session.
2. A Common Training Error Can No Longer Occur:
   Trainers often do too much of the work for the trainees. Those of us who do this type of work usually love to work with the children, and don’t like to give that up. But trainers have to take care not to take up too much of the training time having people “observe.” Sure, observations can be helpful, but too much observation can waste precious training time. With telehealth, the trainer can’t take over.
3. More Responsibility for Trainees:The trainee has to be really motivated to make training successful. If I’m doing training, I’ll probably be responsible for preparing training materials, supplies, and everything we will need to successfully teach the child. But online, that situation is reversed. It is the parent or therapist that has to make sure all the supplies are ready to go. I believe this advance preparation significantly helps trainees learn more.
4. No Modeling:
   One of the major problems with working online is that unless you have great video models of exactly what you want the parent to do, modeling is generally impractical. One potential upside to this, though, is that the parent or therapist can’t just mimic exactly what you are doing. He or she has to really understand what to do and when to do it. I strongly suspect that this problem may actually make the skills more likely to generalize to similar situations.

For me, the lesson is clear. There are many situations where online Telehealth goes beyond just being practical or nearly as effective as in-person services. There are a few situations where it can actually offer a significant amount of Poogi compared to in-person services.

A common concern among parents of children with autism who have significant language deficits is that they won’t be able to express when something is wrong. For example, if they have a sore throat, ear infection, sprained ankle, or other health issue, they may not have the skills needed to communicate that something is wrong. These types of issues can make problem behaviors, social skills, or school difficulties more difficult.

This creates a major teaching challenge. It is much easier to teach children to talk about things happening inside of other people. That’s because the teacher can be sure of what’s happening in pre-planned pictures, videos, or live models. So, it is usually relatively easy to teach children to label pictures or videos as happy, sad, angry, or even hurt. But teaching them to talk about things inside themselves is much harder. The teacher doesn’t really know what is happening inside the child, so it is hard to provide the prompting and feedback needed to teach children to talk about those things.

The standard answer to what to do in this situation is to wait for opportunities when you do know what happened. This is probably how typically developing children learn to talk about these types of internal events. For example, if the child accidentally bumps his or her knee and starts crying the parent can immediately start talking about that. Oh no! You hurt your knee! Pointing to it, talking about it, etc. While I think that advice is fine, it is rather unsatisfying. First, the child is not getting hurt frequently (hopefully). Second, when a rare situation occurs when the child does gets hurt, it doesn’t mean the parent observed it and is able to take the necessary actions. Third, children with delayed language need many opportunities to learn new, difficult language skills. This procedure usually will simply not deliver enough learning opportunities.

As far as I am aware, there is not a published solution to this problem available in the research literature. But having to deal with this frequent concern from parents, I have improvised a solution that seems to work; and even if it doesn’t work, will likely lead to the child learning some useful language skills.

My suggested procedure is based on clinical experience and interpretation of the basic principles, not actual published studies. It is a relatively simple idea. Teach the child to talk about how things feel on different parts of the body so that when a critical situation occurs, they might be able to generalize to talking about an injury.

Here is how to do it:

1. Teach the child the names of body parts without looking (e.g., cheek, elbow, knee, ear, etc). This can often be made into a game that many children enjoy (e.g., cover your eyes, blindfold, etc)
2. Teach the child to label various sensory stimulation without looking (e.g., warm, cold, rough, smooth, tap, squeeze, soft, hard, etc).
3. Teach the child to combine both without looking. The teacher asks, “What do you feel?” and the child makes responses like “Warm ear,” “Rough elbow,” “Squeeze hand,” “Soft knee.”
4. Keep teaching this skill until the child starts making correct responses on novel, untaught examples, and thus demonstrating that they have generalized use of the skill.

I have used this program several times, and on at least two occasions, I have heard from parents that the child spontaneously used the skill when they actually got hurt. The program is generally well liked by children, teachers, and parents. It also has the potential to solve a huge problem for families.

Of course, it would be quite challenging to prove in a controlled research study that the two-component language skill program is why the children were able to respond accurately in the novel injury situation. But it certainly looks plausible.

Several times, I have seen programs that are very ineffective, yet seem to have high parent satisfaction. Other times, I have seen programs that seem very effective, yet have low parent satisfaction ratings. How does that happen?

I’ve heard you are not a real consultant until you make your own 2 X 2 matrix explaining a phenomenon. So here goes:

We may think that parents, teachers, and other stakeholders base their satisfaction ratings of applied behavior analysis programs solely on how well they work; as if they were buying a toaster. But people don’t even judge toasters solely on how well they work. How easy was it to navigate the toaster company’s website? Did it come when promised? How friendly was the customer support? Was it easy to return a damaged product? Etc. etc.

In my view, stakeholders will judge applied behavior analysis programs based on at least two factors. First, how effective is the program? A program that clearly shows no benefit will be unlikely to obtain high satisfaction ratings. Second, the social skills of the BCBA and other members of the team. Of course, these two factors don’t act completely independently, but are synergistic. A highly effective BCBA probably has to respond to upset parents and teachers less frequently. A highly socially skilled BCBA probably has teachers and parents who implement programming better, and thus has higher effectiveness. See this article for an excellent discussion.

If a BCBA falls low on social skills and low on effectiveness, they will fall into a category we might label as incompetent (Lower left panel). Very few stakeholders will be satisfied with services if this is the case.

But if a BCBA is highly effective, yet has poor social skills, this person might still have poor stakeholder satisfaction ratings. This BCBA falls into the category we might refer to as a jerk (right lower panel). Even if a program shows outstanding progress, that doesn’t mean you can act like Barry from the Bronx. Now, Barry the BCBA from Bethel, Connecticut realized that, and has Poogi-ed significantly. In other words, BCBAs need to develop a variety of social skills in addition to clinical skills. This has been studied in the medical profession. For example, problematic social skills seem correlated with medical malpractice claims.

OK, that makes sense. If you act like a jerk, then parents and teachers are not likely to be satisfied regardless of how effective the program. That might explain effective programs with poor satisfaction ratings. But why would low quality programs have high satisfaction just because they like the BCBA? If the program isn’t working for the kid, people are still going to be upset. Right?

Maybe not. Most programs are not completely ineffective and can tell a good story. The child has made some progress. It is just that a higher-quality program would have made much more progress. That isn’t easy for the average stakeholder to determine. If you don’t have the experience to know what’s missing, you might be satisfied with the minimal progress you are seeing. That’s why I refer to this category as delusional (upper left). There might be no one complaining; everyone loves the BCBA, but that doesn’t mean the services are high quality.

Of course, everyone should strive to be in the upper right panel (Awesome BCBA) with high levels of effectiveness and high levels of therapeutic social skills.

To me, the lesson is that while satisfaction ratings are essential data, they are easily biased by (a) How much people like the staff, and (b) The amount of experience the rater has to evaluate whether the program really is high quality or not. They are extremely useful, but caution is warranted when interpreting those data.