Psychometric Properties of Evaluation Surveys

What is reliability?

Reliability refers to a survey’s ability to produce consistent results when repeatedly measuring the same outcome.

Understanding reliability can help verify that a survey will generate the same or similar results across time and social contexts. The reliability of a survey can be assessed through several procedures; the most common ways to ensure reliability are listed below.

 

Checks for Reliability

Test-Retest

  • Involves testing and then retesting the same set of people on the same set of test items to examine the consistency of their responses

Inter-rater

  • Looks at the extent to which trained raters agree when using the same survey

Internal Consistency

  • Involves the cohesiveness of a scale’s items – do all the items in the scale measure the same characteristic or concept. Can they be logically grouped together
 
 

Quick facts about reliability

  • Checks if a measure produces the same results, time and again
  • Uses repeated tests with the same survey to determine the consistency of the survey
 

What is validity?

Validity lets you know whether the survey has correctly captured the concept of interest.

In applied research and evaluation, using a valid survey is particularly important because results from data can be used to inform programming decisions. Listed below are two of the major types of validity to look for in your survey, internal and external validity. It is important to be familiar with both types of validity to choose the best survey possible and understand the limitations of your evaluation or research.

 

Measures of Validity

 

Description

Example

Internal Validity

Degree to which a survey correctly measures what it intended to measure

Let’s say you run an after-school program and you want to use a survey to measure students’ perceptions about school. To ensure internal validity you want to make sure the survey questions specifically are measuring their feelings about school, not how they feel about the after-school program, or just how they feel that day. You want to make sure the question asked measures what it is intended to measure, and that the survey as a whole measures what it is intended to measure.

External Validity

Ability of the survey to create results that can be generalizable

For example, asking youth in a large after-school program how many other youth in their program they talk to each day might be an accurate measure of social networks in that program, but the findings may not be an accurate measure of social networks in a very small program.

 

Quick facts about validity

  • Addresses whether a survey actually reflects the idea it is trying to measure
  • May look at the approximation of results to actual situation
  • Highlights the importance of gathering the right information with your survey

  


Ideally you want to create a survey that is both reliable and valid.

Imagine that you have collected survey data on civic engagement in an afterschool program.  In the illustration below, each dot represents an observed or measured score and the bulls-eye represents actual scores of that population. In the first figure, we see that it is possible to have reliability but have low validity – where participants’ scores consistently miss the mark. It is also possible to have validity but low reliability – where participants’ scores are inconsistent; however a total average of the scores reflects the mark. In the last figure, we see that participants’ scores are both consistent and reflect the mark.

 

Reliable surveys consistently get the same results, but aren't necessarily correct. Valid surveys get the right answer on average but can vary greatly.

 

   Test your knowledge of validity and reliability.                             Explore Additional Resources