"

1. Plan to Conduct Self-Administered Surveys

1.1 Various data collection methods

When collecting data, you take on the role of a researcher. As a researcher, you can use a variety of data collection methods such as the following:

  • Self-administered survey: Individual informants provide information to you by completing your survey questionnaire on their own. The survey questionnaire may consist of closed-ended questions, open-ended questions, or a combination of both.
  • One-on-one interview: You ask each informant a series of questions to gather in-depth information. These interviews typically include open-ended questions, but may also incorporate some closed-ended questions.
  • Focus group: You facilitate a discussion with a group of 10-12 informants from diverse backgrounds to learn about their experiences and opinions.
  • Observation: You collect data by using your senses (watching, listening, talking, feeling, and sometimes interacting) in an environment, either as a participant or non-participant.
  • Extant data review: You review and analyze existing data and documents to extract relevant information.
  • Test: You ask informants to complete written or performance tests to demonstrate their knowledge or skills.

No single data collection method is ideal for all types of situations. Each has its own strengths and limitations (see Table 1). Therefore, it is important to select the method, or a combination of methods, that best fits your project’s goals and context.

Unlike interviews, where you ask questions directly, self-administered surveys (hereafter referred to as surveys) are completed by the respondents themselves. For this reason, your survey questionnaire must be designed clearly so respondents can easily understand the purpose of the survey, the meaning of each item, and how to respond appropriately.

Surveys can be administered on paper or online. Paper surveys may be collected on-site or returned by mail. Additionally, survey questionnaires can be integrated into interviews or focus groups as supplementary tools.

This book focuses on conducting surveys within organizational settings. It is specifically written for professionals in Learning and Development (L&D), Human Resource Development (HRD), Organizational Development (OD), and Performance Improvement (PI)—those who act as survey researchers, collecting data from both internal and external stakeholders to inform evidence-based decisions that support performance improvement.

Table 1 Strengths and Limitations of Various Data Collection Methods (adapted from Table 36, pp. 116-117 in Chyung, 2019[1])

Self-administered survey

Method Strengths Limitations
On site
  • Can administer it to a group of people at once
  • Can handle participants’ questions in person immediately
  • Costly to print out copies of the questionnaire
  • Necessary to use a survey administrator for each site
  • Time-consuming to enter data into an electronic file
Via mail
  • Can administer it to a large number of people in different locations
  • Participants can complete it during their preferred time
  • Costly to print out and mail copies of the questionnaire
  • Notoriously low survey return rates
  • Time-consuming to enter data into an electronic file
On line
  • Can administer it to a large number of people in different locations
  • Participants can complete it during their preferred time
  • Data are already stored in an electronic file, ready for statistical analysis
  • Notoriously low survey return rates
  • Survey administrators and participants must have access to technology

One-on-one interview

Method Strengths Limitations
In person
  • Can observe interviewees’ body language
  • Can retrieve detailed information*

(* common to all types of one-on-one interviews)
  • Interviewers and interviewees need to be in the same location
  • Costly and time-consuming to complete multiple interviews*

(* common to all types of one-on-one interviews)
By telephone
  • Interviewers and interviewees do not need to be in the same location
  • Most people have access to telephone
  • Cannot observe interviewees’ body language
  • Interviewers and interviewees must have access to telephone
Via video-conferencing
  • Interviewers and interviewees do not need to be in the same location
  • Can observe interviewees’ body language
  • Interviewers and interviewees must have access to video-conferencing technology

Focus group

Method Strengths Limitations
In person
  • Participants’ body language provides additional data
  • Can retrieve information from a group of people at once*
  • Participants can learn other people’s views and generate group opinions on a chosen topic through discussions*

(* common to all types of focus groups)
  • Discussions can be influenced by the physical environment (e.g., room setup, sitting arrangement)
  • All participants must be in the same location at the same time
  • Time-consuming to monitor and record participants’ responses*

(* common to all types of focus groups)
By telephone-conferencing
  • Participants do not need to be in the same location
  • Cannot observe participants’ body language
Via video-conferencing
  • Participants do not need to be in the same location
  • Difficult to observe participants’ body language or reaction to other participants’ responses
  • Participants must have access to technology

Observation

Method Strengths Limitations
On site (participant observation)
  • Can understand phenomena in natural settings
  • Can retrieve additional information for better contextual understanding when observers interact with participants
  • Can be used to verify self-reported data
  • Participants may behave differently than usual due to the presence of observers (a.k.a. the Hawthorne effect)
  • Time-consuming for the observers to become accepted as part of the natural settings
  • Costly to use a large number of spot observations through random sampling
Video-recorded (non-participant observation)
  • Can obtain information unobtrusively as observers are not present at the setting
  • No/little interruption to the participants and setting
  • Can be used to verify self-reported data
  • Can be replayed for careful analysis
  • Limiting to rely on recorded information
  • Lacking opportunity to interact with participants to obtain contextual information
  • Costly to use a large number of spot observations through random sampling

Extant data review

Method Strengths Limitations
In print
  • Can reveal actual historical information*
  • Can be used to verify self-reported data*

(* common to all types of extant data review)
  • Not easy to share collected data with other members of the evaluation team (compared to electronic files)
  • Time-consuming to enter data into an electronic file when needed
In electronic files
  • Can share and access easily
  • Can readily conduct statistical analysis on some quantitative data
  • Time-consuming to analyze large amounts of information
  • Difficult to detect errors in information without comparing it to other sources*

(* common to all types of extant data review)

Test

Method Strengths Limitations
Written (on site or online)
  • Can reveal actual knowledge
  • Can be used to verify self-assessed knowledge or confidence levels
  • Difficult to measure knowledge
  • Unreliable results potentially caused by participants’ test anxiety
Performance
  • See Observation
  • See Observation

 

  1. Chyung, S. Y. (2019). 10-step evaluation for training and performance improvement. Sage. ↵

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Survey Design and Data Analysis Copyright © 2025 by Seung Youn (Yonnie) Chyung, Ed.D. is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book