POOGI for ABA

There are many reasons why data collection might go wrong. Someone might have faked the data; mistakes might have been made while collecting the data; or the staff were poorly trained. Today, I’m not talking about any of those types of issues. For this discussion, I’m assuming the data were collected perfectly, the staff never make a mistake, and they didn’t even go to the bathroom as they might miss a minute of data collection. Yet, even under those utopian conditions, the data can lead you to make bad decisions.

In applied behavior analysis research, typically the individual is observed for a relatively short period of time (e.g., 10-15 minutes). But when you and I go to implement programs in the real world, no one wants to see data that were collected over just 10-15 minutes. Imagine the complaints from an administrator:

“He was at school for 6-hours, but you only collected data for 15-minutes?  We need to know how he is doing across the school day.”

Now, collecting data on how many times an individual is having problem behavior across the day might be useful information under certain conditions. Is the program generalizing across the school day? Has he been safe at school? But usually, you can’t look at those data and get any useful information to guide decision making. Why?

Because if the data are collected over long periods of time each day, there is almost never context as to why the data went up or down. This is especially true in school settings. When you start digging in, you will often find important reasons for changes in the data (e.g., the problem behavior was up because there was a sub on Tuesday; the problem behavior was down because there was no math class on Wednesday).

In other words, in most settings the data will go up or down for reasons that have nothing to do with the quality of the intervention plan. If you make decisions about the intervention plan from variables that have nothing to do with the intervention plan, those decisions will likely be poorly made.

In applied behavior analysis, research on this problem typically never comes up. That’s because researchers have to demonstrate with an experimental design the variables that are impacting the behavior. Practitioners in schools aren’t able to do this with the same degree of control. But sometimes, practitioners look at graphs collected under real-world conditions as if they were data collected under tightly controlled conditions in therapy rooms behind 2-way mirrors using laptop computers.

Of course, we are unlikely to be able to match the type of data that gets published in research journals in many practical settings. But, we can Poogi what typically happens. When we are called in to solve a problem, we need to carefully analyze the data in a way that allows you to understand the cause of the problem. That understanding almost never comes from data collected over long periods of time each day. More focused data that helps us understand why the problem is occurring is the essential ingredient.

Many BCBA’s tend to struggle with how to individualize and teach complex skills.  Despite being able to program beautifully for beginning- level skills, some BCBA’s are not able to make the jump to creating programs for high-level skills. A few skills that are frequently hard to teach and generalize are:

• Reading Comprehension
• Answering Wh- questions (who, what, when, where, why)
• Inferencing

Some have argued that this programming is difficult because our science doesn’t have all the tools needed, and recently some have suggested adding new principles and concepts to cover some of the missing gaps. Maybe we need to do this. I’m undecided on this issue. There is a huge on-going debate about this, and I don’t intend to dip into that here.

Today, I’ll just cover one of the basics of good programming that often seems to be missing when we fail to teach these skills. I call this assessment procedure “Find the Breakdown.” That simply means finding the specific point where the learner’s understanding is lacking and teaching specific skills to enable them to perform the task.

For example, let’s say a learner is learning the names of common nouns. After some instruction, the learner can accurately name some pictures, i.e. cat, shoe, juice, hat, and swing. Now, when the teacher introduces a new picture (maybe dog), the learner becomes a bit confused—sometimes they say cat, and sometimes they say dog. Typically, this type of problem can be solved relatively easily by simply providing a variety of examples of dogs and cats until the learner can tell the difference between the two types of animals. In this case, the breakdown is simple; the learner doesn’t understand the difference between dogs and cats. No one is likely to miss it, and solving the problem is relatively simple.

Later, when the learner becomes more sophisticated and has learned hundreds, maybe thousands of words, finding the breakdown isn’t as simple. Now we are trying to teach the learner to answer wh- questions, make inferences, or do reading comprehension, and they aren’t getting it. Finding the breakdown is much more difficult since the issue is usually not as obvious as in the cat vs. dog problem above.

What tends to happen in this situation is a BCBA, speech pathologist, or special education teacher will throw spaghetti and see what sticks. They just try stuff. He is a visual learner, so what if we add some pictures? How about a 2^nd prompter? What if we reduce the length of the passage, and slowly build it up to higher levels? Etc. etc. Of course, sometimes these types of interventions might work, and reinforces just “doing something.”

A much better procedure than throwing spaghetti is to take a small amount of time to do an analysis. If the child is failing to acquire the new skill you are teaching, first figure out why before throwing spaghetti. For example, if you are attempting to teach the learner to answer reading comprehension questions and not succeeding, there might be a variety of reasons. The procedure is simple: Make your best educated guess as to why the learner is failing to acquire the skill, and then briefly test the hypothesis. For example:

• The passage includes vocabulary the learner doesn’t understand. Possible test: Can the learner answer accurately when we carefully control the vocabulary?
• The learner might be able to read accurately, but is so slow that they can’t understand what he or she read. Possible test: Can the learner answer accurately if an adult reads the passage?
• The learner doesn’t understand the difference between certain words in the questions (e.g., who vs. where). Possible test: Can the learner match pictures of people to “who,” and pictures of places to “where”?

Once the cause of the learning problem becomes clear, it is usually a simple manner to design an effective teaching program. If you do this one simple thing, you will get dramatically better at programming.

In Behavior Analysis, we have a saying, “the learner is always right.” We don’t mean this in the sense of giving the correct answer. Of course, learners will make mistakes. We also don’t mean this in the way that businesses mean it as in, “the customer is always right.” We mean it in terms of data. Really, the full saying (perhaps not as catchy) is the “The learner’s data are always right.”

In other words, no matter how well designed the program or how superb the teacher, if the learner didn’t learn, the problem is not with the learner, it is in the instruction. Too often, teachers (and yes, even some BCBAs) will blame the learner: “He is just not getting it;” “He doesn’t generalize;” “He has over-selectivity;” Etc. etc.

Of course, this is in large part the reason why we take data in the first place. These types of data help us make decisions. I think this is one of the reasons we love behavior analysis. We try something that doesn’t work, look at the data, do an analysis, figure out the cause of the problem, make a revision, and finally see the learner succeed based on our efforts.

But… I think we sometimes take this beautiful concept too far. Why?

As one small example, consider the abysmal pass rates of the BCBA exams–approximately 1 out of 3 fail on the first attempt. These people spent years completing all the required courses, devoted enormous amount of time in supervision, and still fail? Is graduate school instruction and supervision so poor that 1 out of 3 people who complete all the requirements fail the exam? Well, probably. No matter how good you are, there is room for Poogi. No doubt, there is room for massive improvement in both graduate school instruction and supervision. I’ve taught graduate school classes in behavior analysis. I’ve supervised many people for their BCBAs. I spent years improving instructional design and supervision, and I see clearly that better teaching leads to better outcomes. At this point, I now feel very confident in my ability to help someone learn the concepts in behavior analysis. But not always. There is another problem besides the quality of instruction and supervision.

If you are in graduate school, there are certain prerequisite skills you should have before you begin. But I’ve seen situations where it looked like people somehow made it into graduate school without possessing the prerequisite skills needed to be successful. For example, if the average high school student skipped algebra and went straight to calculus, we wouldn’t expect them to be successful even if they had the best calculus teacher in the world. In a similar manner, it may not be possible for a graduate school professor or BCBA assigned to provide supervision to help a student or supervisee catch up on the all the prerequisite skills they need. Not that it is the learner’s fault. The learner is still right. Just the prescription might not always be better teaching, it might be building prerequisite skills.

The learner is always right attitude is the way to become a better teacher, regardless of whether you teach children with autism, graduate students, or others. It never makes sense to blame the learner for failing to acquire a skill. The literature is filled with examples of people whom everyone originally thought couldn’t learn a particular concept. But with excellent teaching, they were able to learn. Though if the learner needs to build the prerequisite skills, I think it is reasonable for teachers to ask if they are the right person to do that. This applies to graduate school professors, BCBA’s conducting supervision, and the general education teacher that has a student three years behind grade level included in his or her class.