Author: Barry Morgenstern

There are many reasons why data collection might go wrong. Someone might have faked the data; mistakes might have been made while collecting the data; or the staff were poorly trained. Today, I’m not talking about any of those types of issues. For this discussion, I’m assuming the data were collected perfectly, the staff never make a mistake, and they didn’t even go to the bathroom as they might miss a minute of data collection. Yet, even under those utopian conditions, the data can lead you to make bad decisions.

In applied behavior analysis research, typically the individual is observed for a relatively short period of time (e.g., 10-15 minutes). But when you and I go to implement programs in the real world, no one wants to see data that were collected over just 10-15 minutes. Imagine the complaints from an administrator:

“He was at school for 6-hours, but you only collected data for 15-minutes?  We need to know how he is doing across the school day.”

Now, collecting data on how many times an individual is having problem behavior across the day might be useful information under certain conditions. Is the program generalizing across the school day? Has he been safe at school? But usually, you can’t look at those data and get any useful information to guide decision making. Why?

Because if the data are collected over long periods of time each day, there is almost never context as to why the data went up or down. This is especially true in school settings. When you start digging in, you will often find important reasons for changes in the data (e.g., the problem behavior was up because there was a sub on Tuesday; the problem behavior was down because there was no math class on Wednesday).

In other words, in most settings the data will go up or down for reasons that have nothing to do with the quality of the intervention plan. If you make decisions about the intervention plan from variables that have nothing to do with the intervention plan, those decisions will likely be poorly made.

In applied behavior analysis, research on this problem typically never comes up. That’s because researchers have to demonstrate with an experimental design the variables that are impacting the behavior. Practitioners in schools aren’t able to do this with the same degree of control. But sometimes, practitioners look at graphs collected under real-world conditions as if they were data collected under tightly controlled conditions in therapy rooms behind 2-way mirrors using laptop computers.

Of course, we are unlikely to be able to match the type of data that gets published in research journals in many practical settings. But, we can Poogi what typically happens. When we are called in to solve a problem, we need to carefully analyze the data in a way that allows you to understand the cause of the problem. That understanding almost never comes from data collected over long periods of time each day. More focused data that helps us understand why the problem is occurring is the essential ingredient.

Many BCBA’s tend to struggle with how to individualize and teach complex skills.  Despite being able to program beautifully for beginning- level skills, some BCBA’s are not able to make the jump to creating programs for high-level skills. A few skills that are frequently hard to teach and generalize are:

• Reading Comprehension
• Answering Wh- questions (who, what, when, where, why)
• Inferencing

Some have argued that this programming is difficult because our science doesn’t have all the tools needed, and recently some have suggested adding new principles and concepts to cover some of the missing gaps. Maybe we need to do this. I’m undecided on this issue. There is a huge on-going debate about this, and I don’t intend to dip into that here.

Today, I’ll just cover one of the basics of good programming that often seems to be missing when we fail to teach these skills. I call this assessment procedure “Find the Breakdown.” That simply means finding the specific point where the learner’s understanding is lacking and teaching specific skills to enable them to perform the task.

For example, let’s say a learner is learning the names of common nouns. After some instruction, the learner can accurately name some pictures, i.e. cat, shoe, juice, hat, and swing. Now, when the teacher introduces a new picture (maybe dog), the learner becomes a bit confused—sometimes they say cat, and sometimes they say dog. Typically, this type of problem can be solved relatively easily by simply providing a variety of examples of dogs and cats until the learner can tell the difference between the two types of animals. In this case, the breakdown is simple; the learner doesn’t understand the difference between dogs and cats. No one is likely to miss it, and solving the problem is relatively simple.

Later, when the learner becomes more sophisticated and has learned hundreds, maybe thousands of words, finding the breakdown isn’t as simple. Now we are trying to teach the learner to answer wh- questions, make inferences, or do reading comprehension, and they aren’t getting it. Finding the breakdown is much more difficult since the issue is usually not as obvious as in the cat vs. dog problem above.

What tends to happen in this situation is a BCBA, speech pathologist, or special education teacher will throw spaghetti and see what sticks. They just try stuff. He is a visual learner, so what if we add some pictures? How about a 2^nd prompter? What if we reduce the length of the passage, and slowly build it up to higher levels? Etc. etc. Of course, sometimes these types of interventions might work, and reinforces just “doing something.”

A much better procedure than throwing spaghetti is to take a small amount of time to do an analysis. If the child is failing to acquire the new skill you are teaching, first figure out why before throwing spaghetti. For example, if you are attempting to teach the learner to answer reading comprehension questions and not succeeding, there might be a variety of reasons. The procedure is simple: Make your best educated guess as to why the learner is failing to acquire the skill, and then briefly test the hypothesis. For example:

• The passage includes vocabulary the learner doesn’t understand. Possible test: Can the learner answer accurately when we carefully control the vocabulary?
• The learner might be able to read accurately, but is so slow that they can’t understand what he or she read. Possible test: Can the learner answer accurately if an adult reads the passage?
• The learner doesn’t understand the difference between certain words in the questions (e.g., who vs. where). Possible test: Can the learner match pictures of people to “who,” and pictures of places to “where”?

Once the cause of the learning problem becomes clear, it is usually a simple manner to design an effective teaching program. If you do this one simple thing, you will get dramatically better at programming.

In Behavior Analysis, we have a saying, “the learner is always right.” We don’t mean this in the sense of giving the correct answer. Of course, learners will make mistakes. We also don’t mean this in the way that businesses mean it as in, “the customer is always right.” We mean it in terms of data. Really, the full saying (perhaps not as catchy) is the “The learner’s data are always right.”

In other words, no matter how well designed the program or how superb the teacher, if the learner didn’t learn, the problem is not with the learner, it is in the instruction. Too often, teachers (and yes, even some BCBAs) will blame the learner: “He is just not getting it;” “He doesn’t generalize;” “He has over-selectivity;” Etc. etc.

Of course, this is in large part the reason why we take data in the first place. These types of data help us make decisions. I think this is one of the reasons we love behavior analysis. We try something that doesn’t work, look at the data, do an analysis, figure out the cause of the problem, make a revision, and finally see the learner succeed based on our efforts.

But… I think we sometimes take this beautiful concept too far. Why?

As one small example, consider the abysmal pass rates of the BCBA exams–approximately 1 out of 3 fail on the first attempt. These people spent years completing all the required courses, devoted enormous amount of time in supervision, and still fail? Is graduate school instruction and supervision so poor that 1 out of 3 people who complete all the requirements fail the exam? Well, probably. No matter how good you are, there is room for Poogi. No doubt, there is room for massive improvement in both graduate school instruction and supervision. I’ve taught graduate school classes in behavior analysis. I’ve supervised many people for their BCBAs. I spent years improving instructional design and supervision, and I see clearly that better teaching leads to better outcomes. At this point, I now feel very confident in my ability to help someone learn the concepts in behavior analysis. But not always. There is another problem besides the quality of instruction and supervision.

If you are in graduate school, there are certain prerequisite skills you should have before you begin. But I’ve seen situations where it looked like people somehow made it into graduate school without possessing the prerequisite skills needed to be successful. For example, if the average high school student skipped algebra and went straight to calculus, we wouldn’t expect them to be successful even if they had the best calculus teacher in the world. In a similar manner, it may not be possible for a graduate school professor or BCBA assigned to provide supervision to help a student or supervisee catch up on the all the prerequisite skills they need. Not that it is the learner’s fault. The learner is still right. Just the prescription might not always be better teaching, it might be building prerequisite skills.

The learner is always right attitude is the way to become a better teacher, regardless of whether you teach children with autism, graduate students, or others. It never makes sense to blame the learner for failing to acquire a skill. The literature is filled with examples of people whom everyone originally thought couldn’t learn a particular concept. But with excellent teaching, they were able to learn. Though if the learner needs to build the prerequisite skills, I think it is reasonable for teachers to ask if they are the right person to do that. This applies to graduate school professors, BCBA’s conducting supervision, and the general education teacher that has a student three years behind grade level included in his or her class.

Of course, our long-term goal for our students is success under realistic conditions, not just success during therapy. But evaluating success during realistic conditions too soon is likely to be extremely misleading.

Say Fred, a young boy with autism, loves pizza. But he has a dairy allergy, so he is not allowed to eat pizza. The school serves pizza for lunch every Friday. When the Fred sees all the other kids eating pizza and he can’t have any, he has severe tantrums, self-injurious behaviors, and dangerous aggression.  There might be a wide variety of solutions to this type of problem. For example, maybe he could have dairy-free pizza every Friday, maybe he joins a lunch bunch on Fridays with kids who aren’t eating pizza, or maybe he has a special lunch with a favorite teacher. Through a combination of these 3 approaches the team eliminates this problem.

Fred’s classroom has art once per week. Due to fine motor issues, the occupational therapist is present during art class to help with his skills. She notices that he wants every project to be green, and problem behavior is extremely likely if not allowed to do so. So, she implements a successful program, “First, you use 2 other colors, then, you can use green,” which data show to be highly successful.

In music, Fred loves to play the drums, which the music teacher doesn’t mind, and allows him to do it for several minutes each class (if she didn’t it would cause a severe problem behavior). But she does mind when he runs into class in the middle of her other classes to play the drums. The team works together to successfully eliminate this problem by keeping the music room door closed. Fred just walks past the music room with his class if the door isn’t open.

After six months of intervention, the data on Fred’s problem behaviors are amazing. They are reduced by 95%. But, there still are some dangerous episodes from time to time. What happened on those days? Well…

• We were walking to lunch bunch and he heard on the loudspeaker that it was pizza day.
• The occupational therapist was sick and the art teacher didn’t remind him he had to use 2 other colors before using green.
• A child went to the bathroom and left the music room door open.

The above examples show how misleading data can be. We want to see practical data of how a plan is working in the “real world,” but obviously, these types of plans are not likely to make a long-term significant impact on the child’s life. They might solve an immediate problem, but, of course, they require constant attention or the problem behavior will come back. Generally, it’s not practical to do this over a whole lifetime.

Now, to be clear, I don’t think there is anything wrong with those types of interventions. They allow us to maintain the safety and dignity of the individual child. But just don’t confuse them with interventions that are likely to make a long-term significant difference in the lives of children with severe problem behaviors. The only thing that has a chance to do that is teaching skills, which can take some time.

So, sure–While we are carefully building the skills needed to be successful in therapy, make whatever modifications are necessary and practical to maintain the safety and dignity of the child when not in therapy. Just be careful not to confuse success under those modified conditions with the likelihood of success after therapy ends.

A critical component of programs based on the principles of applied behavior analysis is the commitment to data-based decision making. Part of this commitment involves deciding what data we should collect. I teach two important criteria for making that decision:

Data must be meaningful.

The data must truly capture how well the client is doing and help guide programming.

Data must be practical.

Teachers and parents should be able to collect the data.

Unfortunately, balancing these criteria can lead to conflict. I have frequently seen schools and even programs designed by BCBAs collect superficial types of data that aren’t very useful in helping to make decisions. Why is this?

It is quite possible to design a data collection system that might work in a scientific laboratory with 2-way mirrors and professionals collecting data on sophisticated computer programs, rewinding videos as needed to get all the relevant data. That doesn’t mean it can be done in a classroom or in a home setting with multiple siblings present.

In practice, when conflict comes between “practical” and “meaningful,” the winner will almost always be practical. After all, not having data is going to get you in major trouble. But rarely do people dig into the details and evaluate how meaningful or useful the data collected are in making decisions.  If you do, you will find that in many cases, poorly thought out data leads to misleading conclusions.

Currently, though many of us could use some Poogi in this area, I fear this problem is instead getting worse. That’s because of a relatively new third criterion:

We want to record the data electronically.

In addition to creating practical and meaningful data collection systems, you have to make sure this data can be collected using an app. Certainly, collecting data electronically has many advantages and is extremely useful in many situations. But this technology is very unlikely to lead to more focus on how useful the data collected are for the child’s program. I see many programs twisting what they want to do so that it fits into the data collection system that has been purchased.

It is only a matter of time before our paper data sheets are as common as typewriters. In general, that’s a good thing. The improvements in efficiency and effectiveness from these systems are apparent. Just be careful to keep in mind the importance of collecting meaningful data as well as creating a practical data collection system. The goal is not to make the child’s programming fit into the technology. The goal is for the technology to assist us in making the child’s program better. Too often it isn’t working that way.