As discussed in this module it is very important to check the validity of your survey. In addition, it is important that your evaluation or research design have validity. Certain situations can threaten the internal or external validity of your evaluation design.
Threats to internal validity of your study design might mean that factors outside of the program or treatment could account for the results obtained from the evaluation. Ensuring internal validity means you can be more certain that your intervention or program did cause the effect observed and the effect is not due to other causes.
If you have a threat to external validity, you might be wrong in making a generalization about your findings. It deals with the approximate truth of the conclusions.
Below are common threats to internal and external validity that you should consider during your data collection. Keep in mind that this list is not exhaustive to the many threats to validity that may exist.
| Selection Bias | Control and program participants are selected from populations with different characteristics |
| Attrition or Mortality | Different proportions of participants or different kinds of participants drop out from the control or program groups |
| History | External or unanticipated events occur between administration of evaluation surveys |
| Maturation | Aging or development of participants occurs |
| Instrumentation | Aspects of the evaluation survey itself change between pre/posttest |
| Situational/Contextual Factors | Specific conditions under which the research was conducted limits its generalizability |
| Pretest/Posttest Effects | Results that can only be found after pretests or posttests |
| Hawthorne Effects | Participants’ reactions to being studied alters their behavior and therefore the study results |
| Experimenter Effects | Results are influenced by actions of the researcher |