There are many reasons why data collection might go wrong. Someone might have faked the data; mistakes might have been made while collecting the data; or the staff were poorly trained. Today, I’m not talking about any of those types of issues. For this discussion, I’m assuming the data were collected perfectly, the staff never make a mistake, and they didn’t even go to the bathroom as they might miss a minute of data collection. Yet, even under those utopian conditions, the data can lead you to make bad decisions.
In applied behavior analysis research, typically the individual is observed for a relatively short period of time (e.g., 10-15 minutes). But when you and I go to implement programs in the real world, no one wants to see data that were collected over just 10-15 minutes. Imagine the complaints from an administrator:
“He was at school for 6-hours, but you only collected data for 15-minutes? We need to know how he is doing across the school day.”
Now, collecting data on how many times an individual is having problem behavior across the day might be useful information under certain conditions. Is the program generalizing across the school day? Has he been safe at school? But usually, you can’t look at those data and get any useful information to guide decision making. Why?
Because if the data are collected over long periods of time each day, there is almost never context as to why the data went up or down. This is especially true in school settings. When you start digging in, you will often find important reasons for changes in the data (e.g., the problem behavior was up because there was a sub on Tuesday; the problem behavior was down because there was no math class on Wednesday).
In other words, in most settings the data will go up or down for reasons that have nothing to do with the quality of the intervention plan. If you make decisions about the intervention plan from variables that have nothing to do with the intervention plan, those decisions will likely be poorly made.
In applied behavior analysis, research on this problem typically never comes up. That’s because researchers have to demonstrate with an experimental design the variables that are impacting the behavior. Practitioners in schools aren’t able to do this with the same degree of control. But sometimes, practitioners look at graphs collected under real-world conditions as if they were data collected under tightly controlled conditions in therapy rooms behind 2-way mirrors using laptop computers.
Of course, we are unlikely to be able to match the type of data that gets published in research journals in many practical settings. But, we can Poogi what typically happens. When we are called in to solve a problem, we need to carefully analyze the data in a way that allows you to understand the cause of the problem. That understanding almost never comes from data collected over long periods of time each day. More focused data that helps us understand why the problem is occurring is the essential ingredient.