Threats to Validity of your Design

As discussed in this module it is very important to check the validity of your survey. In addition, it is important that your evaluation or research design have validity. Certain situations can threaten the internal or external validity of your evaluation design.

Threats to internal validity of your study design might mean that factors outside of the program or treatment could account for the results obtained from the evaluation. Ensuring internal validity means you can be more certain that your intervention or program did cause the effect observed and the effect is not due to other causes.

If you have a threat to external validity, you might be wrong in making a generalization about your findings. It deals with the approximate truth of the conclusions.

Below are common threats to internal and external validity that you should consider during your data collection. Keep in mind that this list is not exhaustive to the many threats to validity that may exist.

Threats to Internal Validity

Threats to External Validity

Selection Bias

Control and program participants are selected from populations with different characteristics

Situational/Contextual Factors

Specific conditions under which the research was conducted limits its generalizability

Attrition or Mortality

Different proportions of participants or different kinds of participants drop out from the control or program groups

Pretest/Posttest Effects

Results that can only be found after pretests or posttests

History

External or unanticipated events occur between administration of evaluation surveys

Hawthorne Effects

Participants’ reactions to being studied  alters their behavior and therefore the study results

Maturation

Aging or development of participants occurs

Experimenter Effects

Results are influenced by actions of the researcher

Instrumentation

Aspects of the evaluation survey itself change between pre/posttest