Once you’ve identified your area of interest, sorted through and analyzed the literature to identify the problem you’d like to address, and developed both a purpose and a question; the next step is to design your study. This chapter will provide a basic overview of the considerations any researcher must think about as they design a research study.

Understanding the Language of Research

As you work to determine which approach you will consider in order to best answer your question, you’ll need to consider how to address both the conceptual and operational components of your inquiry. As we discussed in Chapter 1; theory often informs practice (deductive approaches). Theory is often discussed in terms of abstract, or immeasurable, constructs. Because of the ambiguous nature of theory, it is important to conceptualize the parameters of your investigation. Conceptualizing is the process of defining what is or is not included in your description of a specific construct.

Operationalization is the process of defining concepts or constructs in a measurable way. As you dive into the ‘HOW’ you will go about your research, you will need to understand the terminology related to study design

Variables

As we discussed in Chapter 1, there are several kinds of Variables. As a reminder, a variable is an objective and measurable representation of a theoretical construct. An independent variable is a variable which causes an effect on the dependent, or outcome variable. Note that there may be more than one independent variable in your study. Therefore, the dependent variable is the variable which you are measuring as an effect of an intervention or influence; you can think of this as the outcome variable. Identifying at least these two variables is an essential first step in designing your study. This is because how you explore the relationship between your effect (independent variable) and outcome (dependent variable) with help guide your methodology. Other variables to consider include mediating variables, which are variables that are explained by both the independent and dependent variables. Moderating variables influence the relationship between the independent and dependent variables and control variables which may have an impact on the dependent variable but does not help to explain the dependent variable.

Identifying and assigning the dependent and independent variable(s) is one of the most important research activities as this will help guide you toward the type of information you’ll be collecting and what you will do with that information. However, as you consider both the outcome (dependent) variable and the impact (independent) variable, it is also important to consider what other variables may influence the relationship between these two primary variables.

There are very few instances wherein you can control EVERY variable. However, it is your job as a researcher to plan for, acknowledge, and attempt to address anything that may influence the results you present.

levels of measurement can be thought of as values within each variable. For example, traditionally, the variable ‘Gender’ had two values: male or female. The modern variable of ‘Gender’ may have several values which are used to delineate each potential designation within the variable. Each value represents a specific designation of measure.

Values of measures may be considered quantitative (numeric); in our example of traditional gender you may assign a numeric (quantitative) value to male and female as either ‘1’ and ‘2’, respectively. Values may also be assigned non-numerically; meaning they are qualitative. It is important to note that if you want to analyze non-numeric data, it must be coded first.

Levels of measurement

One of the most important aspects of operationalizing a theoretical construct is to determine the level(s) of measurement. This is done by assessing the types of variables and values:

• Nominal: also called categorical. This level of measurement is used to describe a variable with two or more values BUT there is no intrinsic ordering to the categories

• Ordinal: This level of measurement is used to describe variable values that have a specific rank order. BUT that order does not indicate a specificity between ranks.

• Interval: You’ll use this level of measurement for variable values which are rank ordered AND have specified intervals between ranks and can tell you ‘how much more’.

• Ratio: Ratio values have all of the qualities of a nominal, ordinal, and/or interval scale BUT ALSO have a ‘true zero’. In this case true zero indicates a lack of the underlying construct (i.e. it does not exist). Additionally, there is a ratio between points on this particular scale. That is, in this case, 10 IS twice that of 5.

Identification of variable and values is essential to a successful project. Not only will doing this early in the process allow you to predict factors that may affect your research question, but it will also guide you toward the type of data you will collect and determine what kind of statistical analyses you will likely be performing in order to understand and present the results of your work.

Scales

Scales are used to glean insight into a situation or phenomenon and can be used to help quantify information that would otherwise be difficult to understand or convey. Although there are several types of scales used by researchers, we’ll focus on the two of the most common:

• Binary scale: Nominal scale that offers two possible outcomes, or values. Questions that force a respondent to answer either ‘yes’ or ‘no’ utilize a binary scale. IF you offer more than two options, your scale is no longer binary, but is still a nominal scaled item

• Likert scales: Likert scales are popular for measuring ordinal data and include indications from respondents. Data can be quantified using codes assigned to responses and an overall summation for each attribute can be associated with each respondent

Sampling

Sampling is the statistical process of selecting a subset (called a “sample”) of a population of interest for purposes of making observations and statistical inferences about that population. We cannot study entire populations because of feasibility and cost constraints, and hence, we must select a representative sample from the population of interest for observation and analysis. It is extremely important to choose a sample that is truly representative of the population so that the inferences derived from the sample can be generalized back to the population of interest. Probability sampling is a technique in which every unit in the population has a chance (non-zero probability) of being selected in the sample, and this chance can be accurately determined. An example of probability sampling is simple random sampling wherein you include ALL possible participants in a population and utilize a method to randomly select a sample that is representative of that population. Nonprobability Sampling is a sampling technique in which some units of the population have zero chance of selection or where the probability of selection cannot be accurately determined. Typically, units are selected based on certain non-random criteria, such as quota or convenience. Because selection is non-random, nonprobability sampling does not allow the estimation of sampling errors, and may be subjected to a sampling bias. Therefore, information from a sample cannot be generalized back to the population. An example of nonprobability sampling is utilizing a convenience sample of participants due to your close proximity or access to them.

Why does sampling matter?

When you measure a certain observation from a given unit, such as a person’s response to a Likert-scaled item, that observation is called a response. In other words, a response is a measurement value provided by a sampled unit. Each respondent will give you different responses to different items in an instrument. Responses from different respondents to the same item or observation can be graphed into a frequency distribution based on their frequency of occurrences. For a large number of responses in a sample, this frequency distribution tends to resemble a bell-shaped curve called a normal distribution, which can be used to estimate overall characteristics of the entire sample, such as sample mean (average of all observations in a sample) or standard deviation (variability or spread of observations in a sample). These sample estimates are called sample statistics (a “statistic” is a value that is estimated from observed data). Populations also have means and standard deviations that could be obtained if we could sample the entire population. However, since the entire population can never be sampled, population characteristics are always unknown, and are called population parameters (and not “statistic” because they are not statistically estimated from data). Sample statistics may differ from population parameters if the sample is not perfectly representative of the population; the difference between the two is called sampling error. Theoretically, if we could gradually increase the sample size so that the sample approaches closer and closer to the population, then sampling error will decrease and a sample statistic will increasingly approximate the corresponding population parameter.

If a sample is truly representative of the population, then the estimated sample statistics should be identical to corresponding theoretical population parameters. How do we know if the sample statistics are at least reasonably close to the population parameters? Here, we need to understand the concept of a sampling distribution. Imagine that you took three different random samples from a given population, as shown below, and for each sample, you derived sample statistics such as sample mean and standard deviation. If each random sample was truly representative of the population, then your three sample means from the three random samples will be identical (and equal to the population parameter), and the variability in sample means will be zero. But this is extremely unlikely, given that each random sample will likely constitute a different subset of the population, and hence, their means may be slightly different from each other. However, you can take these three sample means and plot a frequency histogram of sample means. If the number of such samples increases from three to 10 to 100, the frequency histogram becomes a sampling distribution. Hence, a sampling distribution is a frequency distribution of a sample statistic (like sample mean) from a set of samples, while the commonly referenced frequency distribution is the distribution of a response (observation) from a single sample. Just like a frequency distribution, the sampling distribution will also tend to have more sample statistics clustered around the mean (which presumably is an estimate of a population parameter), with fewer values scattered around the mean. With an infinitely large number of samples, this distribution will approach a normal distribution. The variability or spread of a sample statistic in a sampling distribution (i.e., the standard deviation of a sampling statistic) is called its standard error. In contrast, the term standard deviation is reserved for variability of an observed response from a single sample.

The mean value of a sample statistic in a sampling distribution is presumed to be an estimate of the unknown population parameter. Based on the spread of this sampling distribution (i.e., based on standard error), it is also possible to estimate confidence intervals for that prediction population parameter. Confidence interval is the estimated probability that a population parameter lies within a specific interval of sample statistic values. All normal distributions tend to follow a 68-95-99 percent rule (see below), which says that over 68% of the cases in the distribution lie within one standard deviation of the mean value (μ 1σ), over 95% of the cases in the distribution lie within two standard deviations of the mean (μ 2σ), and over 99% of the cases in the distribution lie within three standard deviations of the mean value (μ 3σ). Since a sampling distribution with an infinite number of samples will approach a normal distribution, the same 68-95-99 rule applies, and it can be said that:

• (Sample statistic one standard error) represents a 68% confidence interval for the population parameter.
• (Sample statistic two standard errors) represents a 95% confidence interval for the population parameter.
• (Sample statistic three standard errors) represents a 99% confidence interval for the population parameter.

A sample is “biased” (i.e., not representative of the population) if its sampling distribution cannot be estimated or if the sampling distribution violates the 68-95-99 percent rule. As an aside, note that in most regression analysis where we examine the significance of regression coefficients with p<0.05, we are attempting to see if the sampling statistic (regression coefficient) predicts the corresponding population parameter (true effect size) with a 95% confidence interval. Interestingly, the “six sigma” standard attempts to identify manufacturing defects outside the 99% confidence interval or six standard deviations (standard deviation is represented using the Greek letter sigma), representing significance testing at p<0.01.

^[1]

Types of Research Designs

There are many different approaches to research. The list provided here is not exhaustive by any means; rather, this is a brief list of the most common approaches you may identify as you review the literature related to your interest.

Experimental

Experimental research is typically performed in a controlled environment so that the researcher can manipulate an independent variable and measure the outcome (dependent variable) between a group of subjects who received the manipulated variable (intervention) and a group of subjects who did not receive the intervention. This type of design typically adheres to the scientific method in order to test a hypothesis. A hypothesis is a proposed explanation for a phenomenon and serves as the starting point for the investigation.  You may see a hypothesis indicated as (H_O), also called the null hypothesis. This is to differentiate it from an alternative hypothesis (H₁ or H_A), which is any hypothesis other than the null.

Randomized Experimental: Participants are randomly assigned to either a treatment (intervention) or a control group. Typically, the treatment group receives an intervention (independent variable) and the outcome of each group is considered dependent variables and compared for effect. Independent variables in this case are considered active in that this variable can be manipulated.

Quasi Experimental: Similar to the randomized experimental approach aside from the random assignment. In quasi-experimental approaches, participants are NOT randomly assigned; however, one group does receive an intervention while the control group does not and outcomes are still compared. The independent variable is also active.

Non Experimental

Non-experimental approaches include a wide variety of approaches; therefore, it is difficult to list them all in a succinct way here. However, it is safe to say that a study approach is considered non-experimental when there lacks intentional manipulation of the independent variable.

Comparative approach: Groups are compared to reveal differences in outcome (dependent variable). Groups are typically formed based on independent variables that cannot be manipulated but are important to the study. This type of independent variable is known as an attribute independent variable. In this approach there are a few categories (2-4 levels) of attribute independent variables that are then compared.

Associational or Correlational approach: Two or more variables for the same group of participants are explored for relationships. Independent variables are also attributive in this approach; meaning, they can be manipulated to impact the dependent variable. Variables included in this approach are typically continuous or have at least five ordered categories.

Descriptive research: Projects which only gather data which can be described, not inferred. That is, results and data collected cannot be inferred back to the population nor can comparisons or associations be made. Many qualitative studies are considered descriptive. This is done by considering only one variable at a time and there is no independent variable.

It is important to note that it is possible to have more than one approach in one research project. This is because the approach selected is specific to the question that has been asked. If there is more than one question asked, it is reasonable to assume that more than one approach may be used.

There are a few areas of research that although fit under the category of non-experimental; do not quite fit within the classifications presented here. Two of these areas are quality improvement (QI) projects and protocol development.

Quality improvement (QI) projects: The purpose of a QI project is to evaluate the performance of systems, processes, or practices to determine whether either function or operational improvements are needed. Using tools such as the SQUIRE explanation and elaboration guidelines, is extremely helpful in developing, conducting, and analyzing a thorough and impactful QI project.

The SQUIRE guidelines focus on the following four questions:

1. Why did you start?
2. What did you do?
3. What did you find?
4. What does it mean?

These four questions are then expanded upon to help develop the systematic approach to your inquiry and presentation of your findings. An extended investigation of this method is covered in Chapter 6.

Protocol Development

Before we dig too deep into the development of protocols, a clarification needs to be made regarding vocabulary relating to projects of this nature. Although frequently used interchangeably, the terms protocol and guideline are not synonymous. A protocol is described as an official procedure or system of rules governing a process. A guideline is a suggested course of action, policy, or conduct. In healthcare, this is an important distinction; a protocol is a course of action to which treatment must follow without deviation whereas a guideline, although firmly rooted in evidence, allows for deviation based on best judgment of a clinician or presentation of a specific case. Through a research lens, this distinction is important because the process by which these two objectives are realized are very different. The complete process for the development of guidelines which are generalizable beyond a specific situation is best outlined by the World Health Organization Handbook

The development of both guidelines often involves a team of people who are charged with first evaluating the existing evidence and then contributing an interpretation of that evidence toward the consensus of best practice. This is why guidelines are typically issued by federal or state agencies or professional organization. Protocols are generally less generalizable due to contextual constraints. However, even organizational protocols are not developed by a single individual. This does not mean, however, that you cannot begin the process of developing a guideline or protocol for your organization on you own; rather, it is important to frame the work you contribute as the foundation upon which a group can work toward the consensus of best practice. Typically, this initial work is referred to as a narrative review. A narrative review can be described as a broad perspective on a topic which may or may not be impacted by bias. This type of review differs from a systematic review in that it is understood that a narrative review may not encompass all relevant literature on a relevant topic as might a systematic review. Another note; the development of both guidelines and protocols is often an iterative process requiring several cycles of evaluation and revision.  A systematic review is described as exhaustive review of the literature relevant to a specific topic. In addition to being exhaustive, a systematic review includes methodology which is both explicit and reproducible to select, evaluate, and synthesize ALL available evidence. A meta-analysis is a systematic approach to evaluating the data from independent studies of the same subject to evaluate overall trends. Often, a meta-analysis is part of a systematic review.

Selecting your approach

As we’ve discussed, there are several factors which will guide your approach selection. Emphasis should be placed on the development of your purpose and problem statements as well as your research question. Ambiguity in these areas may cause some confusion as you begin to consider what approach you will take to answer your question.  Here we will work to narrow the scope of your approach using a systematic process and answering a few specific questions:

Step 1: Outlining your general purpose

Understanding the overarching goal of your study will help direct the rest of your approach. Here, you will ask yourself “What am I trying to do?”.

Step 2: Identifying your general approach

Earlier we discussed the difference between experimental and non-experimental approaches. As we mentioned, these are two broad categories of approaches. Your general purpose will determine which of these two general approaches you take. The determination here will point you toward a more focused, or specific, approach.

• Experimental: Experimental research is typically performed in a controlled environment so that the researcher can manipulate an independent variable and measure the outcome (dependent variable) between a group of subjects who received the manipulated variable (intervention) and a group of subjects who did not receive the intervention. A true experimental approach means that you have random selection or assignment of participants. All other elements aside, if you do NOT have randomization incorporated into your approach, your approach becomes quasi-experimental.
• Non-experimental: Nonexperimental research is an extremely broad category of approaches. Therefore, the simplest way to explain non-experimental research is to simply state that this approach lacks the manipulation of an independent variable. That is, you are not imposing an intervention on one group and comparing the outcome with a control group. Rather, you may have attribute independent variables which influence, or impact, the dependent variable, but the purpose of the research is not the direct manipulation of that variable. There are several different types of non-experimental research approaches, as we will soon see; however, it is important to understand that descriptive research is always classified as nonexperimental.

Step 3: Narrowing down your specific purpose

Now that you’ve decided what the general purpose and approach, you can begin to really narrow down the ‘how’ of your research. I find that this is best done by again asking yourself what you are really trying to do. Now that you understand the boundaries of your purpose and approach, you can work to understand the fine points about what types of interactions between variables you’re looking to explore and determine.

Step 4: Selecting your specific approach

As you can see, there are specific words you should pay attention to when you’re describing your purpose. Given these key words, like ‘determine causality’, or ‘compare groups’, you’ll have a bit more direction as to what approach is most appropriate to identify the best answer to your question. Once we know what it is we really want to do with the information we’re planning to gather (variables), we can select an approach.
Selecting your specific approach

Resources

Crawford, L.M., Burkholder, G.J., Cox, K.A. (2020). Writing the Research Proposal. In G.J. Burkholder, K.A Cox, L.M. Crawford, and J.H. Hitchcock (Eds.), Research design and methods: An applied guide for the scholar-practitioner (pp. 309-334). Sage Publications

Gliner, J.A., Morgan, G.A., & Leech, N.L. (2017). Research methods in applied settings: An integrated approach to design and analysis. Routledge