Threats to Validity of your Design

As discussed in this module it is very important to check the validity of your survey. In addition, it is important that your evaluation or research design have validity. Certain situations can threaten the internal or external validity of your evaluation design.

Threats to internal validity of your study design might mean that factors outside of the program or treatment could account for the results obtained from the evaluation. Ensuring internal validity means you can be more certain that your intervention or program did cause the effect observed and the effect is not due to other causes.

If you have a threat to external validity, you might be wrong in making a generalization about your findings. It deals with the approximate truth of the conclusions.

Below are common threats to internal and external validity that you should consider during your data collection. Keep in mind that this list is not exhaustive to the many threats to validity that may exist.

Expand all

Threats to Internal Validity

Threats to Internal Validity

Selection Bias Control and program participants are selected from populations with different characteristics
Attrition or Mortality Different proportions of participants or different kinds of participants drop out from the control or program groups
History External or unanticipated events occur between administration of evaluation surveys
Maturation Aging or development of participants occurs
Instrumentation Aspects of the evaluation survey itself change between pre/posttest

Threats to External Validity

Threats to External Validity

Situational/Contextual Factors Specific conditions under which the research was conducted limits its generalizability
Pretest/Posttest Effects Results that can only be found after pretests or posttests
Hawthorne Effects Participants’ reactions to being studied alters their behavior and therefore the study results
Experimenter Effects Results are influenced by actions of the researcher