Author: Barry Morgenstern

The What Am I Doing Wrong Procedure

Most supervisors claim to want feedback from staff, but in my experience it’s very rare for staff to actually provide it. I have written before about how staff sometimes are afraid to tell you when something is wrong. It is a hard problem to solve, as there is no obvious way to know if someone is upset about something and they don’t tell you. In my view, this is a major obstacle that will prevent Poogi. You need the people you supervise to speak up and tell you when something isn’t right. Your programs, your supervision, nothing in your whole organization is perfect. One of the biggest things that we don’t often take advantage of is people’s knowledge about what needs to be improved.

On another front, this is a critical skill that your supervisees will need in their future careers. When people are sitting in meetings, listening to ideas, collaborating with other professionals, and many other activities, the ability to speak up and clearly communicate when something isn’t right — either ethically or scientifically — is a critical skill.

A procedure I’ve used that at least seems effective (it’s hard to know for sure) at increasing the amount of feedback I receive is the “What am I doing wrong here?” procedure. Specifically, I will present something at a staff meeting that sounds as good as I can make it, but doesn’t make sense scientifically or ethically. Then I see if anyone speaks up. If no one does, I might prompt. “Does anyone see anything wrong here?” Then, I enthusiastically reinforce the staff person when they provide feedback. Tell people you are going to do this on a regular basis. Also, if you are the supervisor and you make a natural mistake, it’s great to poke a little fun at it — “Oh, I meant to do that!”

Eventually, people realize that your goal is the POOGI. Once people see giving feedback has both natural reinforcers (we all want to see the program improve) and tangible rewards, you will increase the amount of feedback you receive.

Warning! There will be more wrong than you can imagine. Not only that, it will go on forever as there is no end to the POOGI. It’s not easy, but it’s worth it.

BCBAs tend to have a real love of data and graphs. There are no situations in behavior analysis where we can work without having data. Most of the time, we naturally want to use the best procedures available. We aren’t working with fat grants and usually can’t have multiple people watching sessions behind one-way mirrors recording data on sophisticated laptop computers as often happens in important research projects. In addition to the high cost, there is also practicality — we are in homes, classrooms, and community settings. Still, we try to use the best procedures available under the practical constraints that we have. Often, this is a mistake.

Why? Don’t you believe in Poogi? Shouldn’t we do the best we can do? The problem comes when we compromise on using the best intervention programs available in order to collect more data.

The basic rule on designing intervention programs is to first design the best possible teaching procedures available. Only after you have the determine the best procedures for the individual client should you decide on the data collection. The data collection procedure must give you all the information you need, and the data must be valid and reliable, without requiring compromises in the teaching procedure.

For example, when teaching conversation skills, sometimes behavior analysts ask the RBT or the parent to write down everything the child says. Now, those data might be interesting! We might see if the child is using language learned in speech sessions spontaneously in conversations at lunch. We might detect patterns in conversation topics, etc. etc. But if you watch those sessions, it will usually quickly become apparent that the teacher can’t really do a great job teaching conversation when they are trying to write down every word the child says. You need a different procedure.

Another common example is many instructional programs involve some type of prompting and fading. Now, often it would be interesting to know exactly what type of prompt was used on each instructional opportunity. But I rarely attempt to do this. When I’ve designed teaching sessions this way, frequently it is obvious that this data collection method decreases the ability of the instructor to teach well. Effective teaching depends on careful observation and getting your timing exactly right. Most of the time this isn’t done well when instructors are focused on writing down every prompt used.

POOGI doesn’t mean trying to improve everything all the time. POOGI means trying to improve the right things. The design of the intervention takes priority. POOGI that. Make the intervention the best it can be. Then make sure the data are good enough to make decisions, but don’t go so overboard that you reduce the quality of your program. It happens more than you think.

When working with staff, one of the behaviors that supervisors are frequently interested in improving is “accepting feedback appropriately.” It seems completely reasonable. Anyone who has worked awhile has run into a person they were trying to train who just would not accept any feedback. This can show up in a variety of ways. The person might get belligerent. The person might say they are doing that already. The person might blame others. The person might start crying. The person might “yes” you to death, then keep doing it the way they want to do it. It can be a difficult and frustrating experience. So, of course, supervisors are interested in improving this behavior.

A major problem comes for those of us who are interested in Poogi. One great source of improvement ideas comes from our staff. If you are giving staff a lot of reinforcement and feedback on “accepting feedback appropriately,” how likely is it that they are going to tell you when there is something wrong? Sure, it is easy to say that you also will reinforce staff when they suggest improvements in an appropriate manner. But that isn’t easy. Will staff really feel comfortable telling you about problems? Will staff tell you why the intervention you are suggesting won’t work? What if staff give you suggestions that are terrible?

It is very hard to get data on this potential negative side effect because how will you know when staff don’t tell you what’s bothering them? It will show up in places like high turnover and toxic gossip. But it is very difficult to know in the moment if it is happening.

Most supervisors think that they are approachable and staff will feel comfortable telling them about problems, but in my experience, usually they are completely wrong about that. I don’t claim to have the full answer to this problem, but I have two suggestions that I believe are good steps in the right direction. First, we need to recognize the importance of having conversations with staff. It is easy to get busy and dramatically underestimate how important it is to have a conversation rather than just communicate in brief interactions, texts, and emails. Second, is to have a culture of POOGI. If everyone really believes that you are interested in doing everything possible to improve the system and that you really care about this, you will increase the likelihood that staff will tell you about the problems. Will this completely solve the issue? No, more work is needed. But when I’ve done these two things well, the improvements are obvious.

I love science fiction. One common element of the genre is that someone has to learn to do something — often something that involves some mystical powers. The teacher will often give advice that sounds like an instruction, but really isn’t:

• Use the force.
• Reach out with your feelings.
• Move beyond time and space.

Through a weird quirk in human language, we understand that these statements sound like they are instructions, but obviously they aren’t. In real life, instructions are only useful if the person can understand what they should do differently.

For most people, this doesn’t disrupt the flow of the show. But I’m practically yelling out “that’s not an instruction!”

Parents and teachers can often give “fake” instructions, too. They are simply using language that doesn’t mean much to students in their current situation.

Giving a preschooler a direction like “I need you to focus” is almost never helpful. You might as well tell them to “move beyond time and space.” We can make the same mistake with graduate students when we give instructions like “you need to contrive the motivating operation.”

One of the keys to success in teaching is making sure the instruction sends a crystal-clear communication to the student. If you don’t get this right, the odds of a successful intervention are very low. Sure, you already know that. But it is actually much harder to send a crystal-clear communication then most people think. If you look for this, you will see it happening all the time. It is a common cause of failure. When you see students make “ridiculous” errors like this, you can be sure there was a problem with the communication.

In my view, all BCBAs should take the time to study Engelmann who taught us about “faultless communication.” The classic work is Theory of Instruction. The problem is that book is nearly impossible to read. Engelmann published a much more accessible book here.

I have a dream of a reality TV show for behavior analysis. The idea would be that each week there would be some sort of objective challenge. Maybe week 1 is mand training. In week 2, contestants face the problem behavior challenge. Each challenge has clear measurement procedures and the worst BCBAs get voted out of the clinic each week.

Although I realize the ethical challenges such a show might face, I think it might potentially have a variety of huge benefits:

1. People would clearly see the large differences in quality that currently exist. It would encourage everyone to Poogi.
2. If there were more video of great therapy online it might make it easier for the whole field to POOGI.
3. We might learn a lot about what does and does not work under practical conditions and get great research ideas.

Yes, I know it is never going to happen. Although if you are an interested TV producer, send me a message!

On the other hand, reality-TV-show-like situations happen all the time in practice — The BCBA or the RBT changes, one behavior plan has failed and a new one is tried, an expert comes in to do a consultation, the child moves to a new school or gets a new teacher, or a hundred other natural experiments. This is often a huge opportunity to learn and POOGI.

We probably are not going to get our own reality TV show, but be sure to pay careful attention when these natural experiments occur. It can show you what works under real-world conditions.