External Validity and Believability in Experiments

Imagine for a moment that you are an experiment participant in a dystopian future university thirty years from now. At birth, you were taken from your natural parents and assigned to two robotic parental unit alternatives. The first unit is cold and metal, it has a big frowny face, and all it's good for is dispensing the occasional hot meal through it's midriff. The second unit provides no food, but this unit is fashioned with a luxurious coat of fine fur that feels warm to the touch.

Months pass as you are raised by these two robotic parental units. As you descend further and further into madness, every move you make is video recorded by a pair of enterprising future psychologists who are seeking an answer to one question: Will you spend more time with the cold, metal, food-dispensing robot or the furry one? Surprisingly, though the metal robot fulfills your metabolic needs, the researchers are fascinated to find that you spend most of your time with the furry mother surrogate.

What do results from an experiment such as this (famously conducted by Harry Harlow on monkey's in the 1950's) tells us about the nature of social relationships, love, and survival? Do they tell us anything about the human/monkey experience? Or are the conditions of the experiment so artificial in nature, that they obscure our ability to draw insights about basic psychology? I consider these questions in today's post.

I. The importance of external validity
If the above hypothetical study had been covered by a blog or news outlet, you can pretty easily predict the comments following the summary of the research (which, by the way, you shouldn't read). They would involve critiques of the study that include things like: "This study is nonsense, I can't believe they used taxpayer money to fund this research?" or how about "Another flawed psychology study using university students." or even "Since when do people get raised by robots. Useless!"

Each of these comments points out problems in the external validity of the above experiment, which researchers typically define as the capacity of a given study to generalize to real-world contexts outside the laboratory. Some controlled laboratory experiments suffer from a lack of external validity that arises either from the sample's lack of representation of diverse groups of people (i.e., university student samples) or from the sheer artificiality of the experimental conditions (i.e., robot surrogates).

For researchers interested in studying important social issues, or in using social psychological insights to cause change in social or economic policy, external validity is a huge problem. Take for example, reports that President Obama used a team of psychologists and other behavioral scientists to help him win the 2012 election. High external validity is important if one is trying to win the Presidency, lose weight, reduce discrimination at work, or cure cancer.

II. Defending basic research
Psychology is littered with all sorts of experimental paradigms that fail the external validity test. For instance, have you ever read about a study where participants: pushed a man off a lifeboat to prevent the boat from sinking, posed in an expanded posture for two minutes before gambling, administered electric shocks, made someone else drink hot sauce, or interacted with a stranger in a pitch black room. My grandmother used to point out the lack of external validity in my own research by asking questions like, "So what do you do about it?" There was a time when the question made me stop and think about exactly what I was doing spending all that time in the laboratory?

Of course, for many of my studies I now worry a lot less about external validity. This is because some experiments are artificial by design, and they must be artificial: Usually only a highly controlled artificial setting is sufficient to answer specific questions about basic psychology. Take for instance the robot mother study above: We're interested in answering the question, "What's more important to the individual, food or comfort?" If survival is the only motivation, then all the humans should spend time with the robot who feeds them, but that's not what they would likely do (and it's not what Harlow's monkeys did either). The only experiment that can truly answer this type of question is one that is artificial--that sets up an alternate reality where people are raised by surrogates who offer either food or comfort.

This is the inherent promise of basic research: To answer something fundamental about psychology, we must sometimes depart from reality.

Maybe you are realizing this only now, but there are many psychological studies that have low external validity by design. This, in-and-of-itself is not a problem, because some questions in psychology can only be answered with precise experiments that move beyond the bounds of reality. It's like studying human behavior in a vacuum--other variables (e.g., culture, context, personality) are likely to shift the relationships between constructs that we manipulate in the laboratory. This doesn't diminish the experimental work, rather it highlights the importance of future applied research that brings the ideas tested in the laboratory in contact with real world phenomena. Understanding the importance of social connection (in our robot surrogate example) might lead to applied research that examines, for example, the patient health changes that occur in hospitals with more permissive visitation policies for loved ones.

III. Validity vs. Believability
Let me be clear: I am definitely NOT arguing that low external validity studies lack believability. Just that, it is an assumption that what we learn in experiments will carry over to new samples, situations, and cultures. It is an important and worthwhile empirical endeavor to test the boundaries of what we learn in experiments in real-life settings. Discovering that some cultures value social connection less than others (in our robot example) would be an interesting finding that does not diminish the initial experimental work.

Acknowledging that some experiments have low external validity is important for researchers to remember. I have been asked by journalists and my grandmother if my research calls for policy changes or interventions based on its findings. Basic research calls for additional research to determine if policy changes or interventions will be effective, but I wouldn't recommend sweeping changes to laws based on what is observed in an artificial laboratory setting. Of course, that doesn't stop many researchers (including myself on occasion) from claiming the important applied properties of lab-based experiments. Shame on us for doing this, we should know better.

It is also important to acknowledge that low external validity should not mean that other basic researchers will not be able to replicate what we discover in controlled laboratory settings. Some researchers have claimed that their laboratories are so precise (magically so) that other laboratories that fail to replicate their effects suffer from a lack of precision. This is not the point of low external validity--a basic research finding is only believable to the extent that other researchers can reproduce the experiment and its results. Low external validity studies should still replicate in other controlled laboratory settings.

To conclude, I just want to reiterate the point of this post: Basic research is important because it answers fundamental questions about basic psychological processes, but it is also important for everyone (researchers, research consumers, journalists) to keep in mind that the extent the findings will carry over beyond the vacuum is an empirical question. Yay for basic research!

Mook, D. (1983). In defense of external invalidity American Psychologist DOI: 10.1037//0003-066X.38.4.379