5. Develop Surveys for Different Purposes
5.2 Sample surveys used during evaluation
When conducting program evaluations, you often need to demonstrate improvements in outcomes. These improvements can be presented as differences in the outcomes before and after the program. To achieve this, you may plan in advance to collect both pre- and post-program data, or you may choose to retrospectively assess pre-program data at the time of post-program data collection.
5.2.1 Pre-post surveys
Suppose you are evaluating a caregiver training program, designed for family members and friends who care for individuals with dementia and other conditions involving cognitive decline. The primary goal of the program is to help caregivers improve resilience. You intend to investigate short-term and long-term outcomes of the program by conducting surveys along with other data collection methods. The survey questionnaire could contain items such as those shown in Exhibit 6.
Exhibit 6 Sample Survey Items to Evaluate a Caregiver Training Program (Adapted from Wagnild & Young, 1993 [1])
Please indicate your level of agreement with each of the following statements about the program by selecting one of the numbers from 1 to 7.
| Question | Strongly disagree 1 |
2 | 3 | 4 | 5 | 6 | Strongly agree 7 |
|---|---|---|---|---|---|---|---|
| I follow through with the plans I make. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I am able to depend on myself. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I have enough energy for daily activities. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I can find my way out of a difficult situation. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I can handle many things. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I can depend on myself more than anyone else. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I am determined. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| I keep interested in things. | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
You may plan to use repeated measures with the same survey questionnaire to indicate changes in caregiver resilience over time. You conduct a pre-survey at the beginning of the program and a post-survey at the end of the program. The changes between the pre- and post-survey data would indicate short-term outcomes. You may also follow up with program completers several weeks or months after the program ends and conduct a follow-up post survey. This may show long-term outcomes. In these types of repeated measures, you have to collect identifiable data (e.g., names, IDs, or other unique identifiers) with each survey administration, to match each participant’s pre-, post-, and follow-up post-data for analysis.
| What | Pre-survey | Post-survey | Follow-up post survey |
|---|---|---|---|
| When | Before or in the beginning of the program | At the end of the program | Several weeks or months after the program |
With the average scores collected from the repeated measures, you can present changes in caregiver resilience from the pre-survey to the post-survey, and the follow-up-post-survey using a line graph, as illustrated in Figure 24.
Figure 24 A Sample Line Graph Illustrating the Changed Levels among Repeated Measures
5.2.2 Retrospective surveys
Suppose you are evaluating the quality of a leadership development program that has been in operation for the last couple of years. The program targets first-level managers, aiming to enhance their leadership behaviors. You intend to investigate both short-term and long-term outcomes:
- Short-term outcomes: Whether the first-level managers are satisfied with the program and have acquired sufficient leadership knowledge and skills
- Long-term outcomes: How the managers’ leadership styles have impacted their employees
Because the evaluation was initiated after the program had already begun, you are unable to collect pre-survey data. Therefore, you decide to use a retrospective survey.
To measure short-term outcomes, you’ve decided to investigate two dimensions:
- Participants’ satisfaction with the program
- Participants’ knowledge and skills in leadership
Thus, the survey questionnaire may contain survey items in two sections, as shown in Exhibit 7, and it can be administered with the participants (first-level managers) at the end of the program. Section 1 measures satisfaction, and Section 2 asks participants to rate their knowledge and skills before and after the program. Unlike pre-post surveys, which require you to collect identifiable data such as names or IDs to match pre- and post-data, retrospective surveys collect both sets of data in one administration, allowing for anonymous responses, since no matching of pre- and post-responses is required. However, if you plan to conduct a follow-up post survey, you will still need to collect identifiable information.
To measure long-term outcomes of how managers’ leadership styles have impacted their employees, several weeks or months after the program, you should survey the employees supervised by the program completers (first-level managers). For consistency, you may want to use survey items similar to those used in Section 2 of the managers’ post-survey (Exhibit 7) that was administered with the program completers (first-level managers). The employee survey is shown in Exhibit 8.
Although you may use a retrospective survey asking employees to rate their manager’s leadership skills before and after the manager completed the program, it can be difficult for employees to recall their managers’ program participation timeframe accurately. In such cases, it may be better to ask employees to rate their manager’s current leadership skills only.
Section 1. Please indicate your level of agreement with each of the following statements about the program by selecting one of the options.
| Question | Strongly disagree | Somewhat disagree | Neutral | Somewhat agree | Strongly agree | Don’t know |
|---|---|---|---|---|---|---|
| The program provided information useful to me. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program content was interesting to me. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program was designed well. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program helped me become more knowledgeable about the impact of different leadership styles. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program was worth the time I spent. | ○ | ○ | ○ | ○ | ○ | ○ |
Section 2. Please indicate the levels of your leadership skills before and after you completed the program by selecting one of the numbers: 1 = Poor, 2 = Mediocre, 3 = Average, 4 = Good, 5 = Excellent.
| Question | Before you completed the program | After you completed the program |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| How to: | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 |
| Delegate tasks | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Motivate others | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Provide feedback | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Resolve conflicts | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Be a good listener | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Gain trust | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Be fair | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Communicate clearly | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Be decisive when needed | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
| Be both confident and humble | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ |
Please rate your manager’s current leadership skills by selecting one of the numbers: 1 = Poor, 2 = Mediocre, 3 = Average, 4 = Good, 5 = Excellent.
| How to: | Poor 1 |
Mediocre 2 |
Average 3 |
Good 4 |
Excellent 5 |
|---|---|---|---|---|---|
| Delegate tasks | ○ | ○ | ○ | ○ | ○ |
| Motivate others | ○ | ○ | ○ | ○ | ○ |
| Provide feedback | ○ | ○ | ○ | ○ | ○ |
| Resolve conflicts | ○ | ○ | ○ | ○ | ○ |
| Be a good listener | ○ | ○ | ○ | ○ | ○ |
| Gain trust | ○ | ○ | ○ | ○ | ○ |
| Be fair | ○ | ○ | ○ | ○ | ○ |
| Communicate clearly | ○ | ○ | ○ | ○ | ○ |
| Be decisive when needed | ○ | ○ | ○ | ○ | ○ |
| Be both confident and humble | ○ | ○ | ○ | ○ | ○ |
Then, the managers’ pre-, post-, and follow-up-post-data can be compared with each other, as shown in Figure 25.
Figure 25 A Bar Graph Comparing Before, After, and 6 Months After Program
A similar follow-up-post survey can be administered to both program completers (first-level managers) and their employees. Then, their data can be compared to determine whether the managers’ and employees’ assessments are similar or different (Figure 26). Also see Appendix A for conducting a t-test to compare independent groups.
Figure 26 A Bar Graph Comparing Managers’ and Employees’ Assessments
5.2.3 Surveys to be used in four-level training evaluations
If you are conducting a training evaluation, you may apply the four-level training evaluation framework (Kirkpatrick, 2004[2]) to assess the following four outcomes:
- Reaction—learners’ reaction to the training program at the end of the program
- Learning—the knowledge, skills, or attitudes gained as a result of the training program
- Behavior—their changed behavior on the job after the training program
- Results—the training program’s contribution to overall organizational results
The four-level training evaluation framework is commonly associated with Donald Kirkpatrick as the developer. However, in his early publication in 1956, Kirkpatrick cited the four steps of conducting training evaluation listed by Raymond Katzell, an industrial-organizational psychologist (Kirkpatrick, 1956[3]), which indicates that the originator of the four steps of training evaluation was Raymond Katzell (Smith, 2008[4]; Thalheimer, 2018[5]). Therefore, in this book, we will recognize the four levels as the Katzell-based Kirkpatrick’s four-level evaluation framework (Smith, 2008).
When conducting evaluations using this framework, you can employ various data collection methods (see Table 8). Surveys can be used in all four levels of training evaluation—however, it is important to note that surveys are not a direct measure for assessing learning, behavior, and results.
Let’s explore how surveys can be used within each of the four levels of training evaluation.
Table 8 Examples of Various Data Collection Methods for Four Levels of Training Evaluation (Table 38, pp, 122-123, Chyung, 2019[6])
| Method | 1. Reaction | 2. Learning | 3. Behavioral change | 4. Results |
|---|---|---|---|---|
| Self-administered surveys | Survey trainees’ reaction (a.k.a. smiley sheet) at the end of a program
Survey trainers about their experience with trainees |
Survey trainees to self-assess their learning
Survey trainers about their experience with trainees |
Survey trainees to self-assess their behavioral change
Survey trainees’ co-workers or supervisors about their experience with trainees |
Survey trainees and other stakeholders, and have them self-assess organizational results
Survey customers about quality of organizational service provided to them |
| Interviews and
focus groups |
Interview trainees individually or as a group
Interview trainers about their experience with trainees |
Interview trainees to self-assess their learning
Interview trainers, co-workers, or supervisors about their experience with trainees |
Interview trainees to self-assess their behavioral change
Interview trainees’ co-workers or supervisors about their experience with trainees |
Interview trainees and other stakeholders, and have them self-assess organizational results
Interview customers about quality of organizational service provided to them |
| Observations (including performance tests via observations) | Observe trainees’ reaction during the program (participant observation) | Observe trainees’ new knowledge and skills during the learning process | Observe trainees’ on-the-job behaviors (pre vs. post)
Compare trainees’ on-the-job behaviors to those of people who did not complete the training |
Observe organizational results such as customers’ reaction |
| Extant data reviews | Review existing data that documented trainees’ thoughts about the program quality | Review existing data that documented trainees’ learning outcomes | Review existing data that documented trainees’ job performance | Review existing data that documented organizational results
Compare organizational results obtained from the group that participated in training and those obtained from a group that did not |
| Written tests | N/A | ·Test trainees’ new knowledge at the end of training, compared to pre-training knowledge
Compare test results obtained from trainees to the test results obtained from people who did not complete the training |
N/A | N/A |
Level 1 Reaction
Surveys are most commonly used to measure trainees’ satisfaction with a training program, capturing participants’ immediate impressions and emotional responses to the training experience. Depending on the type of the training program and its context, the survey questionnaires would be designed to measure different aspects of the training program, such as:
- The training enrollment process
- The instructional content covered
- The instructional strategies used
- The usefulness of the learned information
- The trainer’s knowledge
- The trainer’s demeanor
- The training environment (location, classroom, duration, etc.)
Level 2 Learning
A direct way of measuring learning outcomes is through written or performance tests. Surveys are an indirect measure of learning outcomes. Surveys measure learning outcomes only in terms of:
- the learners’ confidence in using the knowledge and skills learned
- their willingness or intention to apply the learned knowledge and skills in the workplace
This type of self-assessed information does not necessarily reflect the learners’ actual competence or performance capacity. However, when actual knowledge testing it is not practical or possible, surveys can still be used to estimate perceived learning outcomes. To improve the reliability of such data, consider triangulating it with other sources such as their supervisors’ assessments or peer feedback.
For example, suppose you are evaluating an e-learning program on lab safety. You plan to measure level 1 reaction and level 2 learning (confidence in learned knowledge and skills). Exhibit 9 is a sample survey questionnaire to be used, in which Questions 1 through 6 measure level 1 reaction and Questions 7 through 12 measure level 2 confidence in learned knowledge and skills.
Exhibit 9 A Sample Survey Questionnaire That Measures Level 1 Reaction and Level 2 Confidence in Learned Knowledge and Skills
Please indicate your level of agreement with each of the following statements about the program and your learning by selecting one of the options.
| About the program | Very untrue 1 |
2 | 3 | 4 | Very true 5 |
Not sure |
|---|---|---|---|---|---|---|
| It was easy to log in and start the program. | ○ | ○ | ○ | ○ | ○ | ○ |
| It was easy to navigate through the program. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program covered the topic of lab safety sufficiently. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program kept me engaged. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program presented safety information in interesting ways. | ○ | ○ | ○ | ○ | ○ | ○ |
| The program used safety examples relevant to my work situation. | ○ | ○ | ○ | ○ | ○ | ○ |
| About your learning | Very untrue 1 |
2 | 3 | 4 | Very true 5 |
Not sure |
|---|---|---|---|---|---|---|
| I am fully aware of safety policy and guidelines. | ○ | ○ | ○ | ○ | ○ | ○ |
| I know how to use safety gear when in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| I know how to use the safety checklist. | ○ | ○ | ○ | ○ | ○ | ○ |
| I know how to operate the fume hood. | ○ | ○ | ○ | ○ | ○ | ○ |
| I know how to dispose of biohazardous waste. | ○ | ○ | ○ | ○ | ○ | ○ |
| I know how to complete proper hand-washing upon contact with chemicals and before leaving the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
Level 3 Behavior
The most direct way to measure trainees’ changed behaviors on the job is through direction observation of performance or by reviewing concrete records of performance (e.g., computer log files, recorded customer service calls).
When direct observation is not feasible, an alternative method is to use surveys asking trainees to self-assess their behavioral changes. This survey data can then be triangulated with another source such as their supervisors’ or customers’ assessments.
Level 4 Results
Training programs intend to contribute to organizational results, but their actual contribution may vary. This includes improved customer retention, increased profits, increased membership, and improved customer satisfaction. Recall that you conduct surveys when the information cannot be directly observed and recorded. Some of the organizational results such as customer retention, increased profits, and increased membership can be recorded and found in documents. However, perceptions of impact—such as improved customer satisfaction may or improved team safety culture—may not be directly observable and must be measured via surveys.
Using the same lab safety e-learning program scenario earlier, you may survey the employees a few weeks or months after they completed training to measure how well they are following the safety guidelines and what impact they think the training has made on the overall safety-related outcomes. You may also conduct a similar survey with the supervisors of the employees.
Exhibit 10 is a sample survey questionnaire, in which Questions 1 through 6 measure level 3 behavior and Questions 7 through 10 measure level 4 results. Exhibit 11 is a sample survey questionnaire to be administered with the supervisors of the employees who completed the program. The supervisors may complete the survey for each employee or complete one survey to reflect the supervisor’s perceptions of the group results. Then, you can compare the employees’ self-assessments and their supervisors’ assessments to see if their assessments are similar or different. If different, you can investigate the discrepancy further with other data collection methods such as interviews and/or observations.
When measuring level 3 and level 4 outcomes with surveys, you can also use retrospective surveys as shown in Exhibit 7 earlier.
Exhibit 10 A Sample Survey Questionnaire That Measures Self-Assessed Level 3 Behavior and Level 4 Results
Please indicate your level of lab safety compliance and how much you think your safety compliance has impacted the overall organizational results, by selecting one of the options.
| About your work performance | Very untrue 1 |
2 | 3 | 4 | Very true 5 |
Not sure |
|---|---|---|---|---|---|---|
| I am following safety guidelines. | ○ | ○ | ○ | ○ | ○ | ○ |
| I use safety gear when in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| I use the fume hood when handling chemicals. | ○ | ○ | ○ | ○ | ○ | ○ |
| I use the safety checklist when in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| I appropriately dispose of biohazardous waste. | ○ | ○ | ○ | ○ | ○ | ○ |
| I complete proper hand-washing upon contact with chemicals and before leaving the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| About the overall results | Very untrue 1 |
2 | 3 | 4 | Very true 5 |
Not sure |
|---|---|---|---|---|---|---|
| I think the level of lab safety is high. | ○ | ○ | ○ | ○ | ○ | ○ |
| I feel safe while in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| I think the safety training helped eliminate safety-related accidents. | ○ | ○ | ○ | ○ | ○ | ○ |
| I feel a culture of safety consciousness among employees. | ○ | ○ | ○ | ○ | ○ | ○ |
Exhibit 11 A Sample Survey Questionnaire That Measures Supervisors’ Assessments on Level 3 Behavior
Please observe your employees’ lab safety compliance for a few days, and indicate their level of lab safety compliance by selecting one of the options.
| My employees: | Very untrue 1 |
2 | 3 | 4 | Very true 5 |
Not sure |
|---|---|---|---|---|---|---|
| Are following safety guidelines. | ○ | ○ | ○ | ○ | ○ | ○ |
| Use safety gear when in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| Use the fume hood when handling chemicals. | ○ | ○ | ○ | ○ | ○ | ○ |
| Use the safety checklist when in the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
| Appropriately dispose of biohazardous waste. | ○ | ○ | ○ | ○ | ○ | ○ |
| Complete proper hand-washing upon contact with chemicals and before leaving the lab. | ○ | ○ | ○ | ○ | ○ | ○ |
These 4-level surveys may allow you to run correlation analyses to explore relationships between the levels and understand how different aspects of training outcomes are connected. For example, you may answer the following questions:
- Is there a correlation between level 1 and level 2 data? In other words, if trainees are more satisfied with the program, do they tend to feel more confident in their new knowledge and skills, and vice versa?
- Is there a correlation between level 2 and level 3 data? In other words, do employees with higher levels of confidence in their new knowledge and skills report higher levels of commitment to using their new knowledge and skills on the job?
Appendix A explains how to perform this type of correlationship analyses with Excel.
- Wagnild, G. M., & Young, H. M. (1993). Development and psychometric evaluation of the resilience scale. Journal of Nursing Measurement, 1(2), 165-178. ↵
- Kirkpatrick, D. (2004). How to start an objective evaluation of your training program. T&D, 58(5), 1-3. ↵
- Kirkpatrick, D. L. (1956). How to start an objective evaluation of your training program. Journal of the American Society of Training Directors, 10-11(May-June), 18-22. ↵
- Smith, S. (2008). Why follow levels when you can build bridges? Training Development, 62(9), 58-62. ↵
- Thalheimer, W. (2018). Donald Kirkpatrick was not the originator of the four-level model of learning evaluation. Work-Learning Research. ↵
- Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. SAGE. ↵
