Required sample size to detect mediation in 3-level implementation studies

The goal of implementation science is to improve the quality and effectiveness of health services by developing strategies that promote the adoption, implementation, and sustainment of empirically supported interventions in routine care [1]. Understanding the causal processes that influence healthcare professionals’ and participants’ behavior greatly facilitates this aim [2, 3]; however, knowledge regarding these processes is in its infancy [4, 5]. One popular approach to understanding causal processes is to conduct mediation studies in which the relationship between an independent variable (X) and a dependent variable (Y) is decomposed into two relationships—an indirect effect that occurs through an intervening or mediator variable (M) and a direct effect that does not occur through an intervening variable [6, 7]. Figure 1 shows a mediation model in which the effect of X on Y is decomposed into direct (c’) and indirect effects (the product of the a and b paths). Estimates of the a, b, and c’ paths shown in Fig. 1 can be obtained from regression analyses or structural equation modeling. Under certain assumptions, these estimates allow for inference regarding the extent to which the effect of X on Y is mediated, or transmitted, through the intervening variable M [8‐10]. Interpreted appropriately, mediation analysis enables investigators to test hypotheses about how X contributes to change in Y and thereby to elucidate the mechanisms of change that influence implementation [5, 9, 10]. Recently, several major research funders, including the National Institutes of Health in the USA, have emphasized the importance of an experimental therapeutics approach to translational and implementation research in which mechanisms of action are clearly specified and tested [11‐13]. Mediation analysis offers an important method for such tests.

Mediation analysis has long been of importance in implementation science, with recent studies emphasizing the need to increase the frequency and rigor with which this method is used [5, 14]. Guided by theoretical work on implementation mechanisms [15, 16], emerging methods-focused guidance for implementation research calls for the use of mediation analyses in randomized implementation trials to better understand how implementation strategies influence healthcare processes and outcomes [5, 17]. A systematic review of studies examining implementation mechanisms indicated mediation analysis was the dominant method for testing mechanisms in the field, used by 30 of 46 studies [4]. Other systematic reviews highlight deficits in the quality of published mediation analyses in implementation science to date and have called for increased and improved use of the method [5, 18]. Reflecting its growing importance within the field, mediation analyses feature prominently in several implementation research protocols published in the field’s leading journal, Implementation Science, during the last year [19‐22]. Chashin et al. [23] recently published guidance for reporting mediation analyses in implementation studies, including the importance of determining required sample sizes for mediation tests a priori.

Designing mediation studies requires estimates of the sample size needed to detect the indirect effect. This seemingly simple issue takes on special nuance and heightened importance in implementation research because of the complexity of statistical power analysis for multilevel research designs—which are the norm in implementation research [17, 24]—and the constraints on sample size posed by the practical realities of conducting implementation research in healthcare systems. While statistical power analysis methods and tools for single-level mediation are well-developed and widely available [8, 25‐29], these approaches are inappropriate for testing mediation in studies with two or more hierarchical levels, such as patients nested within providers nested within organizations [9, 30, 31]. Generating correct inferences about mediation from multilevel research designs requires multilevel analytic approaches and associated power analyses to determine the required sample size [32‐36].

While some tools have begun to emerge to estimate required sample sizes for 2- and 3-level mediation designs [37, 38], findings from this preliminary research indicate that calculation of statistical power for multilevel mediation is complex and depends on the anticipated range and configuration of study design input values—such as effect sizes and sample sizes—at each level (e.g., organization, clinician, patient). As a result, the feasibility of obtaining adequate sample sizes to test multilevel mediation is highly field-dependent; which mediation hypotheses can be realistically tested in implementation science depends on the anticipated range and configuration of realistic study design input values for the field. In implementation research, resource and practical constraints often limit the sample sizes that are feasible to recruit and enroll at the highest level of the design—for example, the number of geographical areas, organizations, or clinics that can be studied—thus potentially restricting the mediation hypotheses that can be realistically tested. Furthermore, the structure of healthcare systems and natural constraints on healthcare processes (e.g., patient flow) often limit the number of providers available within higher-level units over a project period as well as the number of patients each provider serves. These field-specific constraints on sample sizes at each level create a more specific and high-stakes question for implementation scientists interested in using mediation analysis: what are the minimum sample sizes required—at each level—to detect mediation in 3-level designs, given what is realistic for implementation settings?

Our method for estimating statistical power was based on empirical Monte Carlo simulations [46, 47]. Under this approach, many samples of a specified size are generated from a hypothetical population and the model of interest is estimated in each sample. Statistical power is computed as the proportion of samples (e.g., 400 out of 500) in which the parameter of interest is statistically significant. Monte Carlo simulation methods are well-established as a general approach to determining statistical power; they make similar assumptions as formula-based approaches but have greater flexibility for estimating power in complex models derived from hierarchically selected samples [46, 47]. We followed guidelines for reporting Monte Carlo simulation studies as suggested by Boomsma [48].

We used simulations to estimate statistical power for designs that incorporated a continuous outcome and mediator and varied systematically with regard to the population design parameters shown in Fig. 2. Values for each of the nine design parameters were fully crossed, resulting in 17,496 designs (3⁷*4*2). Following prior work [29, 49], values of the two standardized paths that make up the indirect effect (i.e., a₃ and b₃) were set at 0.14, 0.39, and 0.59, which represent small (~2% of the variance), medium (~13% of the variance), and large (~26% of the variance) effect sizes, respectively¹, as suggested by Cohen [50]. Based on the same logic, values of the standardized c’₃ path, which represents the direct effect, were set at 0.14 (small) and 0.39 (medium). Values of the standardized b₂ path, which is not typically of substantive interest in implementation studies, were fixed at 0.39 (medium). Values of ICC for the mediator and outcome were set at 0.05, 0.10, and 0.20 at each relevant level of the design. These correspond to small, medium, and large ICCs based on research describing ranges of ICC for process and endpoint variables in implementation research and healthcare settings [51‐53].

We studied a range of sample sizes relevant to implementation research. As is shown in Table 1, the level-3 sample size (N₃) represents the number of highest-level clusters (e.g., organizations), the level-2 sample size (N₂) represents the number of intermediate-level units per cluster (e.g., providers), and the level-1 sample size (N₁) represents the number of lowest-level units per intermediate unit (e.g., patients). Guided by the range of sample sizes observed in systematic reviews of implementation studies [5, 54‐57], level-3 sample sizes were set at 10, 20, 40, and 60. We chose 10 because it was the expected lower limit on the number of level-3 units (e.g., organizations) necessary to achieve adequate power and 60 because reviews of implementation studies suggest 60 is often the largest feasible sample size. Level-2 sample sizes were set at 5, 10, and 20, reflecting a minimum number of intermediate-level units (e.g., providers) expected to achieve adequate power and an upper limit expected to reflect larger samples in healthcare settings. Level-1 sample sizes were set at 3, 6, and 12, reflecting a minimum number of lowest-level units (e.g., patients) to justify clustering and an anticipated upper limit feasible to recruit during a time-limited period. The code in Additional file 1 can be modified to calculate power for designs not studied here.

For each design, 500 simulated datasets were generated using the MONTECARLO command in Mplus 8 [58]. These were analyzed using the TYPE=THREELEVEL option of the ANALYSIS command with the default maximum likelihood estimator with robust standard errors (MLR). Simulations were conducted on multi-processor computing platforms which allowed for simultaneous estimation of models.

We generated statistical power estimates for each of the 17,496 designs under four different conditions shown in Fig. 3. Cells A and C in Fig. 3 represent statistical power estimates generated for traditional multilevel models with manifest variables (MVM). Cells B and D represent statistical power estimates generated for MSEM. Indirect effects for MVM models were calculated using the “centered within context with means reintroduced” approach described by Zhang et al. [32]. MSEM indirect effects were based on latent partitioning of variables [35].

Cells A and B in Fig. 3 represent statistical power estimates for both MVM and MSEM using a 2-sided null hypothesis test (H₀: a₃*b₃ = 0) with an alpha set at α=0.05. For these tests, we used the first-order delta method which is sometimes called the Sobel test [59]. The Sobel test is widely used for mediation analyses across multiple disciplines [6] and is slightly more conservative than computationally intensive bootstrapping methods [60] or the Monte Carlo confidence interval approach [26, 61].

Cells C and D in Fig. 3 represent statistical power estimates for MVM and MSEM using a 1-sided hypothesis test. Many mediation hypotheses could reasonably be specified as directional (i.e., 1-sided) because the implementation strategy is anticipated to have a positive (or negative) effect on the mediator and outcome. The use of a 1-sided test should reduce the sample size needed to detect mediation. Estimates of statistical power for 1-sided tests were generated using an algebraic transformation of the results from the 2-sided simulations and thus did not require additional computational time (details available upon request).

Completion of the simulations required 591 days of computational time. Completion rates, defined as the number of replications within a simulation that successfully converged (e.g., 500 out of 500), were high: 97.8% (n=17,114) of the MVM simulations exhibited complete convergence (i.e., 500 of 500 replications were successfully estimated) and 79.4% (n=13,889) of the MSEM simulations exhibited complete convergence. The lowest number of completed replications for any design was 493 (out of 500). The high rate at which the replications were completed increases confidence in the resulting simulation-based estimates of statistical power.

Thought leaders and funders in the field of implementation science have increasingly called for a stronger focus on understanding implementation mechanisms [13‐16], with methodologists pointing to mediation analysis as a recommended tool in this effort [5, 17]. Because statistical power to test mediation in multilevel designs depends on the specific range of input values that are feasible within a given research area, we estimated what sample sizes, effect sizes, and ICCs are required to detect mediation in 3-level implementation research designs. We estimated statistical power and sample size required to detect mediation using a range of input values feasible for implementation research. Designs were tested under four different conditions representing two statistical models (MVM vs. MSEM) and 1- versus 2-sided hypothesis tests (see Fig. 3). Fewer than 5% of the designs studied had adequate statistical power to detect mediation. In almost all cases, the smallest number of level-3 clusters necessary to achieve adequate power was 40, the upper limit of what is possible in many implementation studies. This raises important questions about the feasibility of mediation analyses in implementation research as it is currently practiced. Enrolling 40 organizations usually requires substantial resources and may not be feasible within a limited geographic area or timeframe [24, 55]. In many settings, it also may not be possible to enroll enough level-2 units per setting (e.g., nurses on a ward, primary care physicians in a practice, specialty mental health clinicians in a clinic) or level-1 units (e.g., patients per provider). Below, we discuss the implications of these findings for researchers, funders of research, and the field.

This study assesses the sample sizes needed to test mediation in 3-level designs that are typical and plausible in implementation science in healthcare. Results suggest large effect sizes coupled with 40 or more highest-level units are needed to test mediation. Innovations in research design are likely needed to increase the feasibility of studying mediation within the multilevel contexts common to implementation science.