Wednesday, August 21, 2013

External Validity and Believability in Experiments

Imagine for a moment that you are an experiment participant in a dystopian future university thirty years from now. At birth, you were taken from your natural parents and assigned to two robotic parental unit alternatives. The first unit is cold and metal, it has a big frowny face, and all it's good for is dispensing the occasional hot meal through it's midriff. The second unit provides no food, but this unit is fashioned with a luxurious coat of fine fur that feels warm to the touch.

Months pass as you are raised by these two robotic parental units. As you descend further and further into madness, every move you make is video recorded by a pair of enterprising future psychologists who are seeking an answer to one question: Will you spend more time with the cold, metal, food-dispensing robot or the furry one? Surprisingly, though the metal robot fulfills your metabolic needs, the researchers are fascinated to find that you spend most of your time with the furry mother surrogate.

What do results from an experiment such as this (famously conducted by Harry Harlow on monkey's in the 1950's) tells us about the nature of social relationships, love, and survival? Do they tell us anything about the human/monkey experience? Or are the conditions of the experiment so artificial in nature, that they obscure our ability to draw insights about basic psychology? I consider these questions in today's post.

I. The importance of external validity
If the above hypothetical study had been covered by a blog or news outlet, you can pretty easily predict the comments following the summary of the research (which, by the way, you shouldn't read). They would involve critiques of the study that include things like: "This study is nonsense, I can't believe they used taxpayer money to fund this research?" or how about "Another flawed psychology study using university students." or even "Since when do people get raised by robots. Useless!"

Each of these comments points out problems in the external validity of the above experiment, which researchers typically define as the capacity of a given study to generalize to real-world contexts outside the laboratory. Some controlled laboratory experiments suffer from a lack of external validity that arises either from the sample's lack of representation of diverse groups of people (i.e., university student samples) or from the sheer artificiality of the experimental conditions (i.e., robot surrogates).

For researchers interested in studying important social issues, or in using social psychological insights to cause change in social or economic policy, external validity is a huge problem. Take for example, reports that President Obama used a team of psychologists and other behavioral scientists to help him win the 2012 election. High external validity is important if one is trying to win the Presidency, lose weight, reduce discrimination at work, or cure cancer.

II. Defending basic research
Psychology is littered with all sorts of experimental paradigms that fail the external validity test. For instance, have you ever read about a study where participants: pushed a man off a lifeboat to prevent the boat from sinking, posed in an expanded posture for two minutes before gambling, administered electric shocks, made someone else drink hot sauce, or interacted with a stranger in a pitch black room. My grandmother used to point out the lack of external validity in my own research by asking questions like, "So what do you do about it?" There was a time when the question made me stop and think about exactly what I was doing spending all that time in the laboratory?

Of course, for many of my studies I now worry a lot less about external validity. This is because some experiments are artificial by design, and they must be artificial: Usually only a highly controlled artificial setting is sufficient to answer specific questions about basic psychology. Take for instance the robot mother study above: We're interested in answering the question, "What's more important to the individual, food or comfort?" If survival is the only motivation, then all the humans should spend time with the robot who feeds them, but that's not what they would likely do (and it's not what Harlow's monkeys did either). The only experiment that can truly answer this type of question is one that is artificial--that sets up an alternate reality where people are raised by surrogates who offer either food or comfort.

This is the inherent promise of basic research: To answer something fundamental about psychology, we must sometimes depart from reality.

Maybe you are realizing this only now, but there are many psychological studies that have low external validity by design. This, in-and-of-itself is not a problem, because some questions in psychology can only be answered with precise experiments that move beyond the bounds of reality. It's like studying human behavior in a vacuum--other variables (e.g., culture, context, personality) are likely to shift the relationships between constructs that we manipulate in the laboratory. This doesn't diminish the experimental work, rather it highlights the importance of future applied research that brings the ideas tested in the laboratory in contact with real world phenomena. Understanding the importance of social connection (in our robot surrogate example) might lead to applied research that examines, for example, the patient health changes that occur in hospitals with more permissive visitation policies for loved ones.

III. Validity vs. Believability
Let me be clear: I am definitely NOT arguing that low external validity studies lack believability. Just that, it is an assumption that what we learn in experiments will carry over to new samples, situations, and cultures. It is an important and worthwhile empirical endeavor to test the boundaries of what we learn in experiments in real-life settings. Discovering that some cultures value social connection less than others (in our robot example) would be an interesting finding that does not diminish the initial experimental work.

Acknowledging that some experiments have low external validity is important for researchers to remember. I have been asked by journalists and my grandmother if my research calls for policy changes or interventions based on its findings. Basic research calls for additional research to determine if policy changes or interventions will be effective, but I wouldn't recommend sweeping changes to laws based on what is observed in an artificial laboratory setting. Of course, that doesn't stop many researchers (including myself on occasion) from claiming the important applied properties of lab-based experiments. Shame on us for doing this, we should know better.

It is also important to acknowledge that low external validity should not mean that other basic researchers will not be able to replicate what we discover in controlled laboratory settings. Some researchers have claimed that their laboratories are so precise (magically so) that other laboratories that fail to replicate their effects suffer from a lack of precision. This is not the point of low external validity--a basic research finding is only believable to the extent that other researchers can reproduce the experiment and its results. Low external validity studies should still replicate in other controlled laboratory settings.

To conclude, I just want to reiterate the point of this post: Basic research is important because it answers fundamental questions about basic psychological processes, but it is also important for everyone (researchers, research consumers, journalists) to keep in mind that the extent the findings will carry over beyond the vacuum is an empirical question. Yay for basic research!

Mook, D. (1983). In defense of external invalidity American Psychologist DOI: 10.1037//0003-066X.38.4.379

5 comments:

  1. Are you sure your example isn't about ecological validity?
    External validity concerns generalizability of the causal inference of the experiment, ecological validity whether the conditions in the experiment approximate "real life" conditions.

    Validity of any kind is extremely important in psychology, on that I agree.

    ReplyDelete
    Replies
    1. Ecological and external validity are distinct concepts, but very similar and often (though not always) closely related. Michael is saying that people might wonder if any causal inferences drawn from an experiment involving "robot parents" (ha) could really be generalized to another situation (e.g., one in which there are no literal robot parents), and also that people might wonder if any causal claims derived from studies conducted on certain specific, unrepresentative samples (e.g., university students) could really be generalized and assumed to also apply to larger, more diverse samples being targeted by something like a presidential campaign. So, external validity is correct.

      Delete
    2. I see.
      I guess this sentence put me off: "Or are the conditions of the experiment so artificial in nature, that they obscure our ability to draw insights about basic psychology?" + Wouldn't an experiment with "no literal robot parents", increase ecological validity?

      Delete

  2. My name is Madalena and I base in UK...“My life is back!!! After 14 years of Broken marriage, my husband left me and our two twins . I felt like my life was about to end i almost committed suicide, i was emotionally down for a very long time. Thanks to a spell caster called Dr agbakor who i meet online. On one faithful day, as I was browsing through the internet, I was searching for a good spell caster that can solve my problems. I came across alot of testimonies about this particular spell caster. Some people testified that he brought their Ex lover back, some testified that he restores womb,cure cancer,and other sickness, some testified that he can cast a spell to stop divorce and so on. i also come across one particular testimony,it was about a woman called Sarah,she testified about how Dr agbakor brought back her Ex lover in less than 3 days and reverse the effect of their little boys cancer, and at the end of her testimony she dropped Dr agbakor's e-mail address. After reading all these,I decided to give Dr agbakor
    a try. I contacted him via email and explained my problem to him. In just 3 days, my husband came back to me. We solved our issues, and we are even happier than before. Dr agbakor is really a gifted man and i will not stop publishing him because he is a wonderful man... If you have a problem and you are looking for a real and genuine spell caster to solve all your problems for you. Try Dr agbakor anytime, he is the answer to your problems. Here's his contact agbakorspiritualspellcaster@gmail.com. Thank you Dr agbakor.

    ReplyDelete

  3. Hello everybody my name is Louisiana, i really
    want to appreciate a great spell caster who
    helped me when i had problem with my
    Husband,my husband left me after two years
    of our marriage,i endure for so long each day i
    cried,i had seek for solution in many places
    no one could give me a help,i was living in
    sorrow and pains no one could help me. on till i was finally
    introduced to this wonderful spell
    caster,who in my life i will never forget,he is
    honest and kind,who restored my home
    back to peace,you need a right place to solve
    your problems contact
    (agbakorspelltemple@gmail.com) is the right
    choice. he is a great man that have been
    casting spells with years of experience. he
    cast spells for different purposes like

    (1) If you want your ex back.
    (2) if you always have bad dreams.
    (3) You want to be promoted in your office.
    (4) You want women/men to run after you.
    (5) If you want a child.
    (6) You want powers
    (7) You want to tie your husband/wife to be
    yours forever.
    (8) If you need financial assistance.
    (9) Herbal care
    (10)if you want your business to be flourish
    (11)If you are having curt case and you want favor
    email: agbakorspelltemple@gmail.com

    ReplyDelete