When I was preparing to go to Brazil, I decided to learn some basic Portuguese before the trip. I did a few hours of online practice using sample conversations in different situations (e.g., making a purchase at a store). Then, in Brazil, I stopped at a store, and it just so happened that I was able to have the exact conversation that I had practiced online involving three or four exchanges between the cashier and me. Of course, I could barely speak any Portuguese; I had just memorized a few conversational exchanges. Yet my friends were astonished at my ability to speak Portuguese.
A similar phenomenon happens when we work with children with autism. We observe an interaction, and it is easy to exaggerate how much progress it represents. Unless you clearly understand factors such as the baseline level of performance, the reason the child can perform well in the current situation, and whether the skill is generalized enough to maintain over time, it is easy to overestimate how much progress a particular observation represents.
It is also easy to underestimate progress too. Steven Pinker argues that the historical data overwhelmingly show that the world has, in general, gotten much better on many fronts, including things like people escaping poverty, violence, and war. But if you ask people, they think the world has gotten much worse. We tend not to notice progress because it occurs incrementally. When a person gets out of poverty, it generally doesn’t make the news. Progress happens bit by bit, and it is hardly noticeable. But violence, wars, and extreme poverty make the news and are dramatic–We notice them.
Again, a similar phenomenon happens when we work with children with autism. The child is making great progress. Step by step, everything is coming together. Then, some unusual event causes the child to have high levels of aggression, self-injury, or other problem behaviors on a particular day. The child might have made great progress, but one dramatic incident can make it feel like you aren’t getting anywhere. People are, of course, much more likely to notice one dramatic incident as compared to the incremental progress over a long time. It is easy to overestimate how problematic one observation is in the overall picture of the child’s progress.
Unless you understand why the problem is occurring (or not occurring), you can’t effectively make judgments. Just as it is ridiculous to conclude that my Portuguese is excellent from an observation in the store, it is foolish to assume a child is likely to make friends because he has some positive social interactions with peers. There are lots of reasons why the child might be successful in a particular situation that likely wouldn’t correlate with long-term success.
In a similar manner, a child may have done wonderfully in programming and learned to handle a wide variety of challenges under increasingly difficult conditions. Then, a challenge that the child hasn’t yet been prepared for occurs, which leads to serious problem behaviors. That might not indicate that things aren’t working, just something we haven’t targeted yet.
The key lesson is don’t be hasty in making positive or negative judgments about progress from an observation. The key to effective evaluation is understanding why the behavior is successful or not successful. Only then can you even hope to predict what is likely to happen in the future.