Definition of ecological validity

Posted on October 16, 2018

Our quest to define the term ecological validity uncovered more than what we expected. On the one hand, there are researchers who provide details about the meaning of the term. While, on the other hand, there are others who remind their readers that there is no clear consensus on the meaning of ecological validity. Interestingly, the term itself seems to have roused the ire of researchers to the point where the term itself, and its usefulness, have been debated among researchers for years. We share here some interesting discoveries made during own investigation in order to define the term in a semantic manner, but also what it means to researchers.

In broad strokes, ecological validity is more often than not used in opposition to traditional experiments. The controversy pertains to the strengths and weaknesses of ecological research methods that take place in natural settings, as opposed to traditional research methods that takes place in labs. In order to grasp this, let’s take a look at the controversy itself, and then move on to some attempts at defining components of ecological validity.

In their paper Ecological validity and the real-life/laboratory controversy in memory research: A critical (and historical) review (Kvavilashvili & Ellis, 2004), the authors traced back the tension between the two approaches to the 1930s. Most cited as an example of this is Ulric Neisser, who in the 1970s would in his writings and addresses argue strongly against traditional experiments (read: in laboratories). Supporters of this argument would declare that few new discoveries would be made in traditional context, which focuses on broad theories. Laboratories have little in common with real life, they argue, and the results obtained from traditional experiments are too abstract to respond in a relevant manner to important questions. Interestingly, as the debate subsided by the end of the 1980s, Neisser himself attenuated his stance, as contemporary experiments in laboratories were then able to reproduce naturalistic variables more reliably.

In 2001, Mark Schmuckler acknowledged the debates discussed above:

Given that this concept has been debated over the years, it is surprising that there is no clear consensus on exactly what is meant by this phrase.

In his paper What Is Ecological Validity? A Dimensional Analysis (Schmuckler, 2001), he attempted to provide a definition with which researchers could analyze the level of ecological validity of their experiments. Schmuckler provides what he calls three dimensions: setting, which refers to the environment of the experiment; the stimuli, which means the variable manipulated during the experiment; and finally the response, the effect of the experiments on the subject.

As for Kvavilashvili and Ellis, they also provide what they name aspects—representativeness and generalizability—in order to describe the ecological validity of an experiment. In a way, representativeness could be similar to Schmuckler’s setting, in that it means how representative of real-life is the setting of the experiment. Generalizability seems a much more interesting aspect, in that in encompasses every definition we encountered: every researcher intends for their results to be reproducible, no matter if their research was conducted in a real-life setting or in a laboratory.

In The Ecological Validity of Delay Aversion and Response Inhibition as Measures of Impulsivity in AD/HD: A Supplement to the NIMH Multimodal Treatment Study of AD/HD (Solanto et al., 2001), the authors expose the definition of ecological validity from which they worked, which seems to align with the idea of predictability, thus generalizability:

Barkley (1991) has described ecological validity as the extent to which performance on a LM represents the actual behaviors of interest as they occur in natural settings. It is thus a type of predictive validity.

In the same paper as quoted before, Schmuckler describes an interesting experiment, which relies on the “moving room apparatus of Lee,” in which the

[m]ovement of the room simulates the visual input produced via a loss of balance; observers situated in the room compensate for this perceived imbalance with postural sway.

In such a case, the stimuli and the response are both ecologically valid: given that humans rely on visual input as one source of information for their balance, they compensate for what they see, and end up putting themselves in imbalance.

By the same token, such an event is not exactly possible to study in a real-life situation, except for the rare earthquake, and even in such a case, planning for an experiment is akin to placing a bet in a game of roulette. The question is then whether an experiment is ecologically invalid if parts of it are not completely possible to reproduce in a naturalistic environment.

In conclusion, the usefulness of the idea of ecological validity itself has been called into question numerous times, and there is no single agreed-upon definition of ecological validity. However, we still believe that it can be understood overall as a measure of how applicable the results of an experiment are in relation to the real world. In the end, research is looking for knowledge that we can model to predict real-life results accurately.

Credits

Written in collaboration with Natasha Tylosky.

References

Kvavilashvili, L., & Ellis, J. (2004). Ecological validity and the real-life/laboratory controversy in memory research: A critical (and historical) review. History & Philosophy of Psychology, 6(1), 59–80.

Schmuckler, M. (2001). What is ecological validity? A dimensional analysis. Infancy, 2(April 1999), 419–436. https://doi.org/10.1207/S15327078IN0204_02

Solanto, M. V., Abikoff, H., Sonuga-Barke, E., Schachar, R., Logan, G. D., Wigal, T., … Turkel, E. (2001). The ecological validity of delay aversion and response inhibition as measures of impulsivity in AD/HD: A supplement to the NIMH multimodal treatment study of AD/HD. Journal of Abnormal Child Psychology, 29(3), 215–228. https://doi.org/10.1023/A:1010329714819