Author: Barry Morgenstern

Recently, I thought it would be fun and useful to learn Spanish. I signed up for an online app that featured cute characters and gamified lessons with rewards, badges, and lots of fun activities. I did lessons on the app for a few minutes per day, and soon my “streak” was in the hundreds of days. Did I learn Spanish? Well, the data looked good. I had learned lots of words. I could understand much more than when I started. My pronunciation was improving. Was it useful? Hell no. I should have known better. What was the problem? Learning enough to make Spanish a useful skill would take hundreds, probably thousands of hours of work. That’s hard. At the rate of a few minutes a day, by some estimates, it would take around 20 years for Spanish to become a useful skill. That’s assuming there weren’t any lessons with teaching activities that required longer than a few minutes. That’s probably not true.

 

The app was fun and engaging. Maybe even useful as a supplemental learning activity. But I was unlikely to ever learn Spanish only using the app. Am I willing to commit the time and energy required to make Spanish a useful skill? I thought it through, and I decided the answer was no. I have too many other items on my agenda right now.

 

Most people who work in any form of education, from preschool through graduate level, love activities that make learning fun. Teaching concepts using singing characters, gamified apps, problems modeled on real-life examples, or thousands of other creative ideas, that’s much better than slogging through long, boring practice sessions.

 

Popular culture reinforces the idea that this is what great teachers do. I still remember my science teacher trying to replicate this lesson from WKRP in Cincinnati.  Dead Poets Society is filled with Mr. Keating’s memorable lessons about conformity, and, of course, the rip-it-out scene. The movie Stand and Deliver contains lots of examples such as the apple scene and the gigolo problem. Despite the obviously inappropriate nature of some of this, real-world teachers have actually used questionable content, and gotten in trouble for it. The allure of the fun, creative teacher is strong.

 

No one likes this type of teaching more than the people who work with children with autism and other developmental disabilities. I see great, creative examples of the “make it fun” hypothesis almost daily. The “make it fun” approach often leads to faster progress. The child increases their motivation, stays on task longer, and generally prefers fun activities to boring, difficult ones. That sounds great! If we use good judgment and avoid obviously inappropriate content, why not have fun? Including some fun lessons is fine. But you can have too much of a good thing.

 

Fun lessons can backfire. When you try to insert teaching activities into something fun, the child sometimes becomes annoyed–you are spoiling their good time. Even when the learner is having a great time, trying to make everything fun can have negative effects on both the learner and the teacher. The learner believes they never have to be bored or struggle to learn an important skill. And the teacher tends to evaluate how well they are doing based on whether the learner is happy and excited at the moment.

 

Often, it is relatively easy to make some progress at the beginning of learning a skill—but then the hard part comes. Learning the hard part may require struggle and, yes, boring practice. Learners who have only experienced the “make it fun” approach may not make it through the hard part. Teachers may be discouraged that the kids don’t look happy. Observers may think bored-looking kids mean the teaching must be terrible.

 

Usually, the critical thing that predicts how useful a skill will be is how good you are at the skill. If you don’t overcome the hard part, you may never use the skill in real life. Learning to sing along with the cute preschool letter videos is fun. Reading connected text can be hard. Riding a bike with training wheels is easy. Balancing yourself on the bike can be scary and difficult.  

 

Making lessons fun is a wonderful instructional strategy. Just don’t delude yourself that every lesson can be like that. And don’t teach learners that every lesson will be fun and games. Learners sometimes must do difficult things that aren’t fun. Until you’ve mastered the hard part, the skill isn’t useful. Don’t waste time learning new skills if you aren’t going to put in the time and energy to get through the hard part. If you aren’t prepared to do the hard part, it’s better not to start.

 

On the other hand, if you are a teacher and are being evaluated, I’d definitely pull out the fun lessons. The evaluators go to the movies and watch TV too. They know the stereotype of what good teaching looks like.

 

In many fields, science-based professionals have recognized that “the amount of time and effort needed to refute bullshit is an order of magnitude bigger than that needed to produce it.”

It takes almost no time to misinterpret a research study, take things out of context, come up with crazy hypotheses, or even make up “facts” that aren’t true. On the other hand, correcting inaccuracies in the conspiracy theory arguments can take an enormous amount of time. Often, conspiracy theory arguments contain a hint of truth that can make them sound quite convincing. This is why some professionals have been embarrassed in live “debates” when they were obviously on the right side of the science.

In other fields, many scientists have taken the position that it doesn’t make any sense to have live debates with conspiracy theorists. A data-based, carefully thought-through argument takes a lot of time. It is unreasonable to expect anyone to respond in real time when the other side just makes crap up. It’s much better to do your debunking in writing.

This problem will often put BCBAs in a tough spot. We simply can’t do everything in writing. In most cases, decisions on client recommendations are made at meetings. It is easy for the conspiracy theorist to present complete nonsense that can be very hard to debunk on the spot. This can lead to negative outcomes. For example, the client may end up with non-science-based therapies that don’t help their life. Another negative outcome is the BCBA can lose the support of the team, and be thought of as a “BCBA-hole” when coming out hard against the recommendation.

The solution to the problem is difficult. First, recommendations not to be a “BCBA-hole” are appropriate; I wish someone would have told Barry from the Bronx 30 years ago. On the other hand, don’t assume that good social skills or iron-clad reason and logic will solve this problem. To see why good social skills and logic won’t handle this problem, just watch Anderson try that. Those skills are necessary, but not sufficient.

Some BCBAs (including me) have argued for a data-based approach to this problem. “Let’s first collect data with intervention X and then without intervention X and look at the differences in performance.” I have done this many times, and although it sounds simple, it can add many, many hours of work. It can be effective—sometimes–but not usually. Why not? The conspiracy theorist can easily come up with multiple bogus excuses for why intervention X wasn’t effective and what should have been done differently. That doesn’t take barely any time at all. You’ll have them state the requirements in advance? Again, see what Anderson thinks about that. Only rarely should we take on a commitment to “debunking” work. That’s an enormous amount of time and effort and will rarely be effective.Continue reading

In the US, thousands of people die each year waiting for a kidney, yet thousands of donated kidneys are thrown away. Why? There are many reasons, but one of them is problematic measurement systems. Kidney transplant centers are evaluated on their success rate. They are required to have a 98% success rate. Anything less will lead to serious negative consequences. What happens when they get a donated kidney that isn’t perfect, but it’s pretty good? Let’s say the estimate is a 90% chance of saving the patient’s life. What should the treatment center do? If they want to stay open, they should probably throw it in the garbage. Having a 90% success rate would be devastating to the center’s ratings. There is no penalty for not trying. 

 

It is my opinion that this type of problem occurs all the time in applied behavior analysis programs for children with autism. Most programs set goals and objectives that have a very high chance of success. That’s because we need to report the learner’s progress on those goals and objectives. Having several goals that weren’t successful will look bad on the report, even if there is no formal evaluation system. So, if we aren’t sure we can meet a goal, we often do not include it even though reaching that goal would greatly improve the child’s quality of life. There is no penalty for not having the goal on the report and a possible penalty for including the goal and failing to meet it. This can shape the behavior of whole agencies. Contingencies matter. 

 

This problem can be quite subtle. For example, let’s say a learner at a school or clinic is having fantastic success. The supervisors, parents, and staff are celebrating everything the learner has accomplished. But how do the staff who work with a difficult-to-teach learner sitting just a few chairs away feel? Their learner has made some progress, but may never be able to match the accomplishments of the highly successful learner. Even if the supervisors provide staff with lots of positive feedback, staff will compare their learner’s progress to that of the highly successful learner. There are a variety of management strategies that may help staff morale in a case like this, but it sure is a challenge. This is why I believe many programs struggle to find staff willing to work with the most difficult-to-teach learners, even without formal evaluation criteria. Formal systems can sometimes exacerbate the problem. 

 

These are just a few examples. If you look for this problem, you will see it everywhere. We should evaluate these types of phenomena in applied behavior analysis. It is a bit harder to do the research in ABA than in medicine as we don’t usually have easy-to-measure binary outcomes like death. But just because it is difficult to measure doesn’t mean it isn’t important. 

 

Behavior analysts attempt to measure the quality of ABA services in a variety of ways. That’s extremely important, but we need to focus on how we implement measurement systems. I’m willing to bet on many negative outcomes as a result of well-intentioned attempts at measuring success. 

At a small school where I used to work, everyone had to be flexible and move around as needed. We once moved a student to a new room where he hadn’t previously received instruction. He had very minimal problem behaviors, but as soon as the staff attempted to get him to enter the new room, he engaged in extreme levels of problem behavior that we had never seen previously. We had no idea why.

For a couple of days, the staff attempted to shape entering the room. “If you put one foot in the room, you can earn (highly preferred item).” It was a complete failure, and the staff could not get him to cross the threshold.

Eventually, with a lot of detective work, we discovered the problem—the emergency fire exit maps on the wall. On the map, the room was labeled “Eval room,” which sounds like “Evil” when read aloud. He thought the room was the “Evil room.” We simply reprinted the map, changing the name of the room to “Fun room,” and showed it to him. We put signs on the door to the room that said “Fun room,” and left favorite toys in the room. That’s it. We eliminated the problem behavior without further intervention.

Trying to change behaviors without understanding why they are occurring is almost always a poor decision. Taking the time to understand why the person is behaving the way they are leads to huge benefits. It is usually very simple, but often not obvious.

 

 

When I was preparing to go to Brazil, I decided to learn some basic Portuguese before the trip. I did a few hours of online practice using sample conversations in different situations (e.g., making a purchase at a store). Then, in Brazil, I stopped at a store, and it just so happened that I was able to have the exact conversation that I had practiced online involving three or four exchanges between the cashier and me. Of course, I could barely speak any Portuguese; I had just memorized a few conversational exchanges. Yet my friends were astonished at my ability to speak Portuguese.

A similar phenomenon happens when we work with children with autism. We observe an interaction, and it is easy to exaggerate how much progress it represents. Unless you clearly understand factors such as the baseline level of performance, the reason the child can perform well in the current situation, and whether the skill is generalized enough to maintain over time, it is easy to overestimate how much progress a particular observation represents.

It is also easy to underestimate progress too. Steven Pinker argues that the historical data overwhelmingly show that the world has, in general, gotten much better on many fronts, including things like people escaping poverty, violence, and war. But if you ask people, they think the world has gotten much worse. We tend not to notice progress because it occurs incrementally. When a person gets out of poverty, it generally doesn’t make the news. Progress happens bit by bit, and it is hardly noticeable. But violence, wars, and extreme poverty make the news and are dramatic–We notice them.

Again, a similar phenomenon happens when we work with children with autism. The child is making great progress. Step by step, everything is coming together. Then, some unusual event causes the child to have high levels of aggression, self-injury, or other problem behaviors on a particular day. The child might have made great progress, but one dramatic incident can make it feel like you aren’t getting anywhere. People are, of course, much more likely to notice one dramatic incident as compared to the incremental progress over a long time. It is easy to overestimate how problematic one observation is in the overall picture of the child’s progress.

Unless you understand why the problem is occurring (or not occurring), you can’t effectively make judgments. Just as it is ridiculous to conclude that my Portuguese is excellent from an observation in the store, it is foolish to assume a child is likely to make friends because he has some positive social interactions with peers. There are lots of reasons why the child might be successful in a particular situation that likely wouldn’t correlate with long-term success.

In a similar manner, a child may have done wonderfully in programming and learned to handle a wide variety of challenges under increasingly difficult conditions. Then, a challenge that the child hasn’t yet been prepared for occurs, which leads to serious problem behaviors. That might not indicate that things aren’t working, just something we haven’t targeted yet.

The key lesson is don’t be hasty in making positive or negative judgments about progress from an observation. The key to effective evaluation is understanding why the behavior is successful or not successful. Only then can you even hope to predict what is likely to happen in the future.