Mediators of measurement-based care implementation in community mental health settings: results from a mixed-methods evaluation

We set out to address pressing implementation questions in an initiative to bring measurement-based care (MBC) into community mental health settings. MBC is the systematic assessment of patient symptoms prior to clinical encounters to collaboratively inform care [7]. MBC can be conceptualized as having three components of fidelity. The foundational MBC component, per its evidence base, requires patient-reported outcome completion, using measures like the Patient Health Questionnaire (PHQ-9), prior to every single encounter [8, 9]. This component alone—often referred to as “routine outcome monitoring” or “progress monitoring”—has been demonstrated to improve clinical outcomes [10, 11], and yet it is met with numerous implementation challenges in community mental health settings where fewer than 20% of clinicians engage in MBC [7]. MBC fidelity is optimized when score trajectories are discussed with patients (second component) and treatment is adjusted according to the data (third component). In collaboration with Centerstone—one of the US largest not-for-profit community behavioral health providers—we designed a study to compare the effectiveness of standardized versus tailored approaches to implementation (details about the activities in each group are found in the “Method” section) to support MBC integration.

Our active implementation phase outcomes of this cluster randomized trial are published elsewhere [12]. We found that tailored implementation outperformed the standardized approach in terms of PHQ-9 measure completion (foundational fidelity component) but overall fidelity remained suboptimal in that score review, and collaborative discussion did not significantly increase in either group and MBC fidelity was not associated with improvements in patient depressive severity outcomes.

In the current study, we explored patient-reported PHQ-9 completion beyond this initial implementation phase and examined the degree to which contextual factors served as determinants of this implementation outcome over time. To do so, we used a mixed-methodological approach in which we first conducted a quantitative mediation³ analysis, then analyzed qualitative data from four exemplar sites to explore possible novel determinants, and then integrated the two datasets by merging them for analysis and comparison [14]. We hypothesized that tailored implementation would remain superior and that contextual factors (e.g., structure, norms) would emerge as mediators of patient-reported PHQ-9 completion in the tailored group, but not in the standardized group. This evaluation was largely exploratory given the dearth of implementation theory to inform hypotheses about which specific contextual factors might causally lead to the institutionalization of MBC.

MBC has been shown to improve treatment outcomes compared with treatment as usual and may be particularly beneficial for identifying and adjusting treatment when a patient has experienced a lack of improvement or deterioration [9]. In the current study, the PHQ-9 was used as the measure for routine administration to adult patients with a depression diagnosis [15, 16]. The PHQ-9 is a nine-item self-report measure that assesses depressive symptoms that align with the DSM-5 diagnostic criteria for a major depressive episode; scores reflect the severity of depressive symptoms. The PHQ-9 is a widely used measure with strong evidence of reliability and validity, including sensitivity to change [15, 17]. The extant literature indicates that routine administration of the PHQ-9 alone can lead to benefits over treatment as usual as it reveals patient deterioration or lack of progress [9, 15]. Results from our active implementation phase of this trial indicated that the tailored approach led to improvements in PHQ-9 completion, over that of the standardized approach. Accordingly, PHQ-9 completion was the primary outcome of the sustainment phase of this trial.

This mixed-methods study was conducted in 12 community-based mental health clinics in two Midwestern states [18]. Participating clinics were assigned to the implementation group using a dynamic cluster randomized design similar to Chamberlain et al. [19]. Clinics were matched on urbanicity, number of clinicians, and number of patients seeking care for depression. Participants were not blind to the study group. All eligible clinicians were offered a 4-h MBC training followed by 5 months of active implementation support. The type of active implementation support was determined by the group. Details of the study protocol and methods have been described previously [20, 21]. This study was reviewed and approved by Indiana University’s IRB; clinician participants provided informed consent.

In the tailored group, implementation teams of 5–8 members representing different clinic roles (i.e., clinician, office professionals, directors) were formed and met approximately every 3 weeks for 5 months of active support. Individuals who were identified as influential in the organization using a sociometric network analysis [22] and/or as having positive attitudes toward MBC via a survey [23] or self-nomination were invited to join. Meetings focused on reviewing data on clinic MBC use and strategies to further MBC implementation. Implementation teams could tailor the MBC guideline or subscribe to the standardized recommendation. Several teams opted to expand their guidelines to include individuals 12 years and older given that the PHQ-9 is validated for use with youth. Several teams opted to expand their guideline to include all diagnoses given high rates of comorbid depression in community-based populations. Five of six teams opted to tailor their guideline with details offered in Table 1.

In the standardized group, consultation teams were formed and open to all participating clinicians. However, the schedules of individuals who were identified as influential and/or as having positive attitudes were prioritized. Meetings were led by an external consultant and focused on supporting clinician use of MBC as recommended by the study guideline.

Clinicians were included if they completed the baseline and 5-month assessment of the contextual measures and had one or more appointments with an eligible patient during the sustainment phase of the study. Of the 155 clinicians with sustainment phase data, 122 had sufficient data on one or more of the candidate measures to be included in the study. See Table 2 for the demographic characteristics of clinicians.

Patient data pertaining to MBC fidelity were included in the sustainment phase if patients met clinic-specific eligibility criteria in the tailored group, or if in the standardized group, they were diagnosed with depression and were at least 21 years old at the time of their session. Data from a total of 6709 patients were included. These patients accounted for 33,354 sessions that served as the target for our primary implementation outcome (i.e., PHQ-9 completion).

There are numerous determinant frameworks that can guide implementation and evaluation. Across all, it is clear that contextual factors ought to be considered at varying levels of the organization from the beliefs of individuals involved to the broader socio-political conditions in which the organization is embedded. The Framework for Dissemination was used as a guiding framework for the identification and evaluation of a relatively parsimonious set of contextual factors as candidate mediators because it grew from an academic-community partnership, offered a three-phased evaluation process, and integrated the best available evidence [24]. This framework outlines six contextual domains theorized to be associated with successful implementation of a new practice: (i) norms and attitudes: knowledge, beliefs, and values of stakeholders about the intervention as well as perceptions of organizational support for the intervention (i.e., organizational climate); (ii) structure and process: organizational operations and attributes such as decision making processes, types of services, and organization size; (iii) resources: financial, technological, human, social, and political capital that supports implementation; (iv) policies and incentives: the costs and benefits of use of the new intervention as determined by regulatory policies and funding mechanisms; (v) networks and linkages: connections between organizations and between individuals that facilitate knowledge sharing and social support related to the new intervention; and (vi) media and change agents: internal and external sources of credible information and influence such as external trainers, consultants, and internal champions of the intervention.

The battery of measures used to capture candidate mediators was comprised of six, self-report instruments completed by clinicians at baseline, after 5 months of active implementation, and again 10 months later (i.e., end of sustainment monitoring). See Table 3 for a description of the measure, sample items for each subscale, and how subscales were mapped onto the Framework for Dissemination. Scale reliability estimates in the current sample are provided in Table 4.

We assessed group main effects and group-by-time interactions across the sustainment phase using generalized linear mixed models. Following recommendations from Singer and Willett [25], we evaluated three unconditional growth models (i.e., models with only temporal variables) to determine the trajectory that best fits the data prior to adding other fixed effects. We compared three unconditional growth models, linear, quadratic, and log-transformed, and selected the best-fitting model on the basis of the Bayesian Information Criterion (BIC). As a final step, we allowed time slopes to vary by each of the random effects and select the model with the lowest BIC. After establishing the unconditional growth model, we fit a model in which the implementation group was added as the main effect and a model in which a group-by-time interaction was added. All data analysis was conducted using R 4.0.3 in RStudio 1.4.1103 [26]. All models were fit using the glmer function in the R lme4 package. PHQ-9 completion was regressed on the mediator in a model that contained intercepts random on the patient, clinician, and clinic, which reflected multiple observations (i.e., sessions) within patients, multiple patients within clinicians, and multiple clinicians within organizations. Models assumed a binary distribution with a logistic link function. Each session represented an observation in which PHQ-9 completion was recorded.

We examined mediation effects in which contextual factors mediate the impact of the implementation group on PHQ-9 completion. Mediators were obtained at the 5-month assessment (end of active implementation), and patient outcomes were data points collected between the 5- and 15-month assessments (during sustainment monitoring). This approach allowed us to establish temporal precedence in the path of the indirect effects in which exposure to the implementation group preceded the assessment of mediators and mediator assessment preceded outcomes. To assess mediation, we tested whether Path a: implementation group predicted differences in the mediator; Path b: the mediator predicted change in PHQ-9 completion; and Path c: group predicted change in the outcome. The first two conditions enumerated above are the core relationships for establishing mediation [27]. Prior to assessing indirect effects, we assessed each of the a and b paths in individual models. All data analysis was conducted using R 4.0.3 in RStudio 1.4.1103 [26]. All a path models were fit with the lmer function in the R lme4 package, version 1.1.26 [28], and the degrees of freedom were estimated using Satterthwaite’s degrees of freedom method implemented using the R merTools package, version 0.5.2 [29]. In each a path model, the mediator was regressed on the implementation group in a model that contained an intercept random on the clinician’s clinic. Each b path model was fit using generalized linear mixed models with the same structure described above for PHQ-9 completion. If significant effects in the a and b paths were observed, indirect effects (i.e., the product of paths a and b) were tested using biased-corrected bootstrapped confidence intervals.

Each null hypothesis significance test is treated individually, and thus, alpha levels are adjusted, which effectively treats each test individually, as opposed to a disjunctive test in which the rejection of any of multiple null hypothesis significance tests would result in the rejection of a joint intersection null hypothesis. Alpha-level adjustments are not required for individual tests but are required for disjunctive tests [30]. Although alpha levels are not adjusted, the possibility of a type I error that can occasionally occur with a multiplicity of tests is an acceptable risk when balanced with type II errors. In accepting this risk [31], we acknowledge that the analyses are exploratory in nature.

The comparison of the linear, quadratic, and log-transformed unconditional growth models that were fit as a subsequent step to evaluating the implementation group on PHQ-9 completion indicated that the linear time effect with random slopes for patient and clinician was the best fitting model, see Table 5. The implementation group effects model indicated that levels of PHQ-9 completion did not differ across the sustainment phase (z = 0.30, p = 0.765). The group-by-time interaction model revealed a significant interaction across the sustainment phase (z =  − 2.44, p = 0.015). We probed the interactions with simple effects for treatment conditions at the beginning and end of the sustainment window and with simple slopes for the standardized and tailored groups. PHQ-9 completion was higher at the beginning of the sustainment window for the tailored group and was higher at the end of the sustainment window for the standardized group; however, there were no significant differences between the groups at either time point. Similarly, the tailored group exhibited a decreasing simple slope and the standardized condition exhibited an increasing slope, though neither simple slope was significant. Thus, the interaction represents a crossover interaction whereby the tailored group decreased across the sustainment window and the standardized group increased.

Means (SD) and Cronbach’s alpha for each mediator are displayed in Table 4. Prior to fitting inferential models, we evaluated each of the reliability constructs using Cronbach’s α. We excluded scales with α < 0.60, which we set as the lower limit for exploratory research [36]. Excluded scales included several from the SOF.

Results for the mediation paths are displayed in Table 6. Regarding Path a, controlling for baseline levels of the outcome, the tailored group exhibited higher ratings for program-level structures on the B&F than did the standardized group at 5 months (t[6] = 3.84, p = 0.005), suggesting greater levels of facilitation of implementation at the program or clinic level. Similarly, the tailored group exhibited greater perceptions that change was possible and encouraged, and training utilization typically occurred at the program level on the SOF than did the standardized group at five months (t[114] = 2.29, p = 0.024; t[113] = 1.99, p = 0.049). Regarding Path b, only the 5-month “practicality” assessed on the ASA was a significant predictor of PHQ-9 completion across the sustainment phase of the study (z = 1.96 p < 0.050).

Thematic analysis of the 5-month FG with the four exemplar clinics illuminates additional clinic-level characteristics that may help to explain the quantitative results. Three themes emerged from qualitative analysis of these clinics that suggest the central importance to implementation outcomes (i) of compelling communication from leadership, (ii) supportive supervision, and (iii) clinical consultation opportunities. A fourth potential theme, the presence of strong support staff, also emerged from the analysis, although it was only explicitly discussed at one clinic. Each of these themes is discussed further below with quotes provided in Table 7.

This study is one of few longitudinal implementation trials with rich quantitative data to engage in rigorous mediation analyses. The experimental design of this study also fulfilled several key requirements for identifying mediators including variable selection guided by implementation theory, longitudinal design in which changes in predictors preceded changes in outcomes across time, random assignment to a group, and experimental manipulation of groups [13]. Although no quantitative mediators emerged, three scales demonstrated significant improvement from baseline to 5 months in the tailored group. These were program-level training utilization and organizational climate for change from the SOF [37] and program-level structures from the B&F [38]. These constructs assess clinician perception of the extent to which new interventions are supported and used at the organizational level. More specifically, the program-level training utilization scale assesses clinician perception of the frequency at which new interventions are integrated into the organization following training (e.g., “How often do new ideas learned from workshops get discussed or presented at your staff meetings?”). The organizational climate change scale assesses organizational expectations and encouragement of change (e.g., “You are encouraged here to try new and different techniques”), and the program-level structure scale assesses organizational infrastructure to support MBC (e.g., “We use fidelity reports to improve our MBC practice”).

A deeper look at how the tailored group unfolded over time provides possible explanations for why these scales were not associated with PHQ-9 completion from 5 to 15 months. During the 5-month active implementation period, clinics in the tailored group held approximately triweekly implementation team meetings. Implementation teams were tasked with selecting and executing strategies to facilitate MBC implementation. As reported in a previous study of these implementation teams’ strategy use, quality management and communication were among the most common categories of strategy use [39]. Quality management took the form of review and discussion of reports of MBC use at the clinic (i.e., “fidelity reports”) and communication strategies often took the form of encouragement to use the PHQ-9 in clinic team meetings, the break room, and via email. When strategies used by the implementation team are compared with individual items from the above scales, it seems possible that clinicians at 5 months were reflecting on the efforts of their clinic implementation team. However, following the 5-month active implementation phase, the study team withdrew active facilitation support. Although some clinics opted to continue meeting, it was typically with reduced frequency and intensity. It is possible that across the next 10 months, these strategies were not consistently used undermining lasting changes in MBC use. Community mental health clinicians are faced with competing and constantly changing demands. For example, as is common in many community mental health organizations, clinicians expressed concern about achieving productivity requirements for billing in focus groups and informally to study staff during training visits and meetings [40]. Continuing to devote unbillable time to implementation team meetings and to-dos generated during meetings might have felt unfeasible. If these program-level strategies were contingent upon the active support of the study team, then its discontinuation may well have undermined MBC sustainment [41].

This raises the question: does tailoring change the context or does tailoring align strategies to be responsive to context and accommodate the barriers to implementation? As an analogy, if a person has the intention to go for a walk but has a broken leg, crutches can be used as a strategy to support the person to go for a walk. However, when this aid is removed, the person can no longer go for a walk. The context (broken leg) remains unchanged, but strategy (crutches) supports movement toward the goal (going for a walk). In parallel, once active study team support was removed, clinics that previously struggled with unclear communication from leadership, administratively focused supervision, and competing demands were faced with these barriers again without the “crutch” of the implementation team. Successful repair of a broken leg that enables a person to walk freely requires setting and bracing the leg as well as taking weight off the leg for a sufficient period of time to heal. In parallel, it could be that strategies to permanently alter (setting and bracing the leg) and/or lengthier involvement by the study team (using the crutches for an extended period of time) are needed to sustain the use of measurement-based care. In the tailored group, the clinic team members were empowered by the study team to direct implementation strategies given their site-specific expertise. However, teams resorted to the use of a limited number of all possible implementation strategies that may not have supported sustainment optimally [39]. Although clinic implementation team members were experts in their own work context, understandably, they were not experts in the science (or practice) of implementation and thus not knowledgeable about more intensive strategies geared toward leadership and managerial change [42].

There are specific packages of implementation strategies such as the Leadership and Organizational Change for Implementation (LOCI) intervention that have been shown to result in alterations in clinician perceptions of leadership style and leadership support for implementation efforts involving repeated training sessions, coaching calls, feedback reports about leadership, and individualized development plans to improve leadership [43, 44]. Although clinic leadership and informal leaders (i.e., opinion leaders) were included on implementation teams, the present study did not make explicit attempts to alter individuals’ approaches to leadership or supervision. And, in fact, candidate mediators reflecting aspects of leadership (i.e., Director Leadership subscale from the SOF, ILS, and Agency Leadership Support subscale from the BFS) did not change as a result of tailoring during the 5-month active implementation period.

In addition to the three scales that demonstrated significant improvement from baseline to 5 months in the tailored group, the practicality subscale of the ASA measure at 5 months was associated with PHQ-9 completion at 15 months. The practicality subscale captures perceptions of the ease of using measures like the PHQ-9 (e.g., “standardized progress measures can efficiently gather information”), meaning that clinicians who found the PHQ-9 easy to use were more likely to sustain administration over time. As a reminder, the standardized group was not an inert control, rather it represented certain implementation “best practices” which included support for the use of MBC via triweekly consultation meetings. Consultation has been shown to be a promising strategy for supporting clinician skill acquisition and implementation of new practices [45]. It is therefore possible that clinicians across groups who used MBC during the 5-month active implementation period gained direct exposure to the ease of use and continued to incorporate it in their practice through the sustainment period. It is somewhat surprising that the experimental group (i.e., tailored or standardized group) did not alter perceptions of practicality across the 5-month active implementation period given the concerted effort by several implementation teams in the tailored group to incorporate PHQ-9 administration into the front-office workflow. However, when office professionals administered the PHQ-9 at patient check-in, clinicians may have not been directly exposed to the speed and efficiency of measure completion.

Given the variability in the trajectories of PHQ-9 completion during the sustainment phase that could not be explained by the study group, we undertook exploratory qualitative analyses of the focus groups conducted at the end of the 5-month active implementation period to characterize two exemplar clinics that demonstrated improvement in PHQ-9 completion versus two clinics that deteriorated regardless of the study group (see Table 8 for characteristics of exemplar clinics). A comparison of improving and deteriorating clinics revealed the role of leadership, supportive supervision, and clinical consultation with peers in the implementation of MBC. Specifically, clinicians at clinics that experienced increases in PHQ-9 completion across both the standard and tailored groups described clear communication from leadership regarding expectations and rationale for the implementation of MBC, incorporation of MBC discussion during clinical supervision, and opportunities to consult about MBC or receive general support from colleagues as facilitative. Conversely, clinicians at clinics that experienced decreases in PHQ-9 completion described unclear top-down (or no) communication about MBC from leadership, confusion about MBC from supervisors, and isolation from colleagues. These findings map onto extant implementation research that demonstrates the importance of strong leadership and the role of supervisors as leaders in supporting implementation efforts and skill acquisition [46‐48]. The role of peer support is also reflected in the literature regarding the influence of social norms on behavior [49].

Although attempts were made to capture some of these domains, particularly leadership, via the quantitative data, baseline levels of these measures were controlled for when quantitatively assessing the association between the experimental group and candidate mediators, meaning it was not possible to determine if pre-study strong/weak leadership and supervision played a role in PHQ-9 completion. In addition, measures that assessed leadership or clinician perceptions of the organization (i.e., ILS and certain subscales of the SOF) did not orient the survey-taker to the level of analysis to which they should respond. For this reason, it is possible that some clinicians responded to survey questions with their direct supervisor in mind while others may have responded with upper-level organizational leadership in mind. This is especially possible given that some clinics housed regional leadership making them more visible to clinicians compared with clinics who were more disconnected from upper-level leadership. Future studies should, therefore, carefully define the level of analysis in the survey directions and, when appropriate, in the body of the survey [50].

There were several limitations to the current study that surface implications for future implementation strategy mediation analyses. First, there are a host of measurement issues, such as the one described above in which measures that assessed leadership or clinician perceptions of the organization did not orient the respondent to the level of analysis to which they should respond. Thinking carefully about the theory of change and the target social unit for an implementation strategy will be important for devising measure instructions because no single study could assess perceptions of all stakeholder leader-like roles. Moreover, prior research has demonstrated that manipulation of the EBPAS language, for example, to refer to general interventions versus specific evidence-based treatments changes clinician response suggesting the importance of assessing intervention-specific attitudes rather than general attitudes [51]. Although we deployed surveys multiple times, it is possible that waiting some period of time post-active implementation support is crucial when measuring candidate mediators; guidance is needed. Moreover, even though our survey battery of relevant contextual factors was quite extensive, measures related to patients, their perspective, and the clinical encounter itself (e.g., length of time, purpose) [52] were altogether missed.

Although qualitative inquiry presents a way in which to surface novel contextual factors where quantitative measures do not exist, the focus group interviewing guide used in this study queried the same domains outlined in the Framework for Dissemination [24] and, as such, may have missed other relevant contextual factors not included in this framework. We did offer clinicians an opportunity at the end of the focus group to share about areas not queried in the main portion, but relatively little new information emerged at that time. In addition, although attempts were made to include clinicians who had varying use and opinions about MBC, it is possible that capacity and willingness to attend resulted in a biased sample of respondents. Rapid ethnographic methods might offer a way to overcome some of the limitations of the traditional focus group methodology, which might be important for the study of factors influencing sustainment [53].

Finally, this approach to tailoring the implementation of measurement-based care, although informed by a framework, was largely atheoretical. The field is at a point of readiness for centering theory to inform practical implementation [54, 55]. In the case of tailoring implementation, causal theory can be leveraged in at least two ways. One, determinants rarely act in isolation yet commonly used approaches to prioritizing barriers for tailoring strategies invite stakeholders to rate each individual factor for its importance and/or feasibility, among other parameters [1]. Bringing theory to bear on this step, to generate a theory of the problem, could result in a more focused set of determinants. This could be done by leveraging the notably sparse implementation theory or by theorizing the temporal interrelationships among local contextual factors and implementation outcomes [56, 57]. Two, strategies could be matched or tailored to determinants by articulating their mechanisms of action and aligning those with target barriers. This could be done by diagramming causal pathways of strategies that center determinants, although more empirical work, is needed to bring confidence to this endeavor [58].