Nationwide implementation of a multifaceted tailored strategy to improve uptake of standardized structured reporting in pathology: an effect and process evaluation

The effectiveness study consisted of monitoring the use of SSR templates (for gastrointestinal, gynecologic, and urologic oncology) in all Dutch pathology laboratories by conducting an interrupted time-series-analysis. As this study was conducted on a national level, no specific inclusion criteria were applied. The dissemination process is shown on the timeline (Fig. 2). The pre-introduction period included data from June 2020 to November 2020, the strategy introduction period ranged from December 2020 to May 2021, and the post-introduction period included data from June 2021 to November 2021, using data on SSR template use from the pathology database [31]. The unit of analysis for the effect analysis is the number of pathology reports. The selection criteria are shown in Additional file 5. In short, we included all reports, both biopsies and resections, for all suspicious malignancies (irrespective of final diagnosis) that can be reported within the available SSR templates for gastrointestinal, gynecological, and urological tumors. The colon biopsy template was excluded, since this is mandatory for the national bowel screening program and thus unaffected by the implementation strategy.

The aggregated data based on continuous input from the pathology departments were evaluated for SSR usage on a weekly basis, 26 time points before and 26 time points after the dissemination period of the toolbox, as shown in Fig. 2. The primary outcome was the proportion of SSR pathology reports compared to all reporting for the specific tumor groups.

We used IBM SPSS Statistics V25.0 (IBM Corp, Armonk, NY, USA) to analyze the primary outcome among all Dutch pathology laboratories, using descriptive statistics. To analyze the effect of the nationwide implementation strategy, we conducted an interrupted time series analysis [32]. To do that, we performed a segmented regression analysis to identify if the level of proportion of SSR usage and the linear trend in SSR usage were significantly different before and after the strategy introduction. We created a data sheet including nationwide aggregated weekly SSR use among all Dutch pathology laboratories. The outcome was measured as the proportion of cases for which SSR was used in pathology reporting. The autoregressive integrated moving average (ARIMA) model including the following covariates was performed:

All data points of the strategy introduction period had been excluded from analysis, as introduction of the strategy was occupied [33]. The impact of the strategy was also analyzed separately for subgroups (1) the reporting of gastrointestinal, gynecological, or urological oncology; (2) biopsies and resections; and (3) non-academic and academic pathology laboratories, since SSR implementation varies between tumor groups, retrieval method, and type of laboratory. We knew from our previously conducted barrier and facilitator analysis among these three tumor groups that specific influencing factors related to these subgroups of pathology reporting exist [19, 20]. Next, we added the variable “group” to test differences between the subgroups. For these tests, the gastrointestinal group, biopsy group, and non-academic laboratories group were used as reference groups. We combined the “group” variable with the other variables (strategy phase, time, and time since strategy introduction) to generate interaction variables. The autoregressive integrated moving average (ARIMA) model including all these covariances was performed. A p value of <0.05 was considered to be statistically significant, based on two sided tests.

We used descriptive statistics to determine the average proportion of SSR usage out of all reporting for the three tumor groups per pathology laboratory for all laboratories in the period before and after the strategy introduction. For this analysis, we also conducted a subgroup analysis per tumor group (gastrointestinal, gynecological, or urological), retrieval method (biopsies and resection), and type of laboratory (non-academic and academic).

Data on exposure rates were determined during the strategy introduction period in combination with a nationwide process evaluation, using questionnaires, to gather evidence on the feasibility of our multifaceted strategy [34, 35]. We included all pathologists and pathology residents operating in the Netherlands, being the primary users of SSR templates, and PALGA liaisons, who exchange information on SSR between pathologists of their laboratory and PALGA, and are responsible for local technical settings of SSR templates.

To determine exposure to our implementation strategy elements, we combined multiple data sources, such as Google Analytics for website usage, Cuble’s statistics for eLearning usage [36], and PALGA statistics for SSR template related strategy elements. For the active strategy elements, the eLearning, the “Feedback button”, and audit and feedback reports only, we were able to collect data on a laboratory level, since due to the General Data Protection Regulation, we were not allowed to collect data such as the Internet Protocol address from website visitors. Furthermore, we performed a national survey to evaluate the process of SSR implementation, with questions about the actual use of the strategy element, user experiences regarding accessibility, content and usability, and self-reported effectiveness of mechanisms of action, more SSR usage, and better SSR usage. In addition, questions were asked on barriers of SSR implementation and their improvements. The link to the eSurvey was distributed using various channels including direct mails to PALGA liaisons and interested pathologists from existing mailing lists. In addition, calls were posted on the PALGA website, on the PALGA LinkedIn page and in eNewsletters from the Dutch Pathology Association (n=6) and the Dutch Pathology Residents Society (n=2). Information on the research question, target groups, and the anonymous data analysis was included in the introduction of the survey. Before starting the eSurvey, respondents provided informed consent for anonymous data analysis. Completing the questionnaire took 10 to 15 min for all target groups. Unanswered questions were not accepted. Recommendations of Fan & Yan [37] were used in eSurvey development and dissemination to improve eSurvey response.

For the active elements, the eLearning, the “Feedback button”, and audit and feedback reports, we tested the differences in effect between pathology laboratories who did use, in case of the eLearning and “Feedback button”, or did receive, in case of the audit and feedback, the strategy element during strategy introduction versus laboratories who did not use or receive it. We calculated the proportion of SSR usage out of all reporting per week among the groups of pathology laboratories who did and who did not use (eLearning, “Feedback button”) or receive (audit & feedback) the specific active strategy element. The pathology laboratory not using SSR (n=2) were excluded from the analysis for the “Feedback button”, since this element is only accessible via the SSR templates. Besides analysis per active element, we also grouped the users of 2 or 3 active elements, 1 active element, or 0 active elements (=reference group) in three groups to test the effect of multiple active elements use on SSR usage, also excluding the two laboratories not using SSR. The autoregressive integrated moving average (ARIMA) model including all covariates (Strategy phase, Time, Time since strategy introduction, group, Strategy phase*group, Time*group, Time since strategy introduction*group) was performed. A p value of <0.05 was considered to be statistically significant, based on two sided tests. A comprehensive overview of the effect and process evaluation, including all outcomes, is provided in Additional file 6.

Figure 3 shows the trend in SSR use for the SSR templates included. The nationwide implementation of the tailored implementation strategy caused a non-significant effect in level of SSR usage (+1.4%, p=.135). The change in trend of SSR use after strategy introduction compared to before was also non-significant (−0.1% per week, p=.277). The results of the subgroup analysis can be found in Table 2. Only the significant results of subgroups and between subgroups are reported on.

Trends in SSR usages per tumor group are shown in Fig. 4. The increase in level of SSR use was significant for gastrointestinal tumors: (+2.9%, p=.018) and urological tumors (+3.0%, p=.003). For gastrointestinal tumors, the slope of the trend was declining (−0.4% per week, p=.000). The increase in level was lower for gynecological tumors (−3.3%, p=.045) compared to gastrointestinal tumors. The change in trend after the strategy introduction was lower for gynecological tumors (−0.3% per week, p=.003) and higher for urological tumors (+0.4% per week, p=.001) compared to gastrointestinal tumors.

The level in SSR usage after introduction for resections only showed a significant increase (+2.5%, p=.007) (Fig. 3). The increase in SSR usage for non-academic laboratories is significant compared to academic laboratories (+4.2%, p=.024).

Figure 5A–D shows the average use of SSR: laboratories (33 out of 42) increased average laboratory use of SSR templates post introduction and overall, average SSR use post-introduction increased (47.3 vs. 42.3%). In the majority of laboratories, the use of SSR increased for gastrointestinal reporting (34 out of 42) for gynecological reporting (32 out of 42) and for urological reporting (28 out of 42). SSR use increased for all tumor groups: gastrointestinal reports (44.1 vs. 36.7%), gynecological reports (44.6 vs. 39.7%), and urological reports (53.5 vs. 49.9%).

Figures 6A, B and 7A, B show the average use of SSR: there was more increase in the reporting of resection (34 out of 42 laboratories) compared to biopsies (30 laboratories). Relatively more non-academic laboratories improved (27 out of 34) compared to academic laboratories (6 out of 8).

Exposure rates are shown in Table 3. We reached 33 out of 40 pathology laboratories with the active strategy elements. The “Feedback button” was the most often used active strategy element; it was used 128 times during the strategy introduction. Except for the main webpage, the webpage providing information on SSR content was the most visited webpage of the website (n=173). Of the 13 different eLearning courses available, the first course on the introduction of SSR templates was chosen most often (n=7). This was followed by the course on using fast keys within SSR templates (n=6), the eLearning course explaining how to properly manage the settings of SSR templates (n=5), and the course on how to combine two different SSR templates (n=5). More than half (n=23) of the pathology laboratories received audit and feedback reports, of which 5 pathologists of 4 different pathology laboratories replied by asking questions and reporting feedback.

The eSurvey evaluating the process of the nationwide implementation strategy yielded 53 responses: 40 pathologists, 5 pathology residents, and 8 PALGA liaisons with an average age of 44 (range 26–66) and average years of experiences of 11 (range 1–37). Respondents were working in non-academic hospitals (59%), academic hospitals (26%), and independent laboratories (15%). The majority of the pathologists (93%) was involved in either gastrointestinal, gynecological, and/or urological oncological pathology. All were aware of SSR templates and used them in clinical practice. Of the 45 pathologists (and residents), 11 stopped using a SSR template, mostly while using the template for reporting prostate biopsies.

More than half (53%, n=24) of the respondents received the toolbox. Most pathologists received the toolbox via the PALGA liaison (21%, n=12). Three-quarter (n=6) of the PALGA liaison respondents received the toolbox via e-mail, and 5 of them forwarded it to pathologists within their laboratory. Half of the 30 respondents receiving the toolbox also entered the toolbox. The mean reasons for not consulting the toolbox were lack of need for assistance (23%, n=7) and lack of time (17%, n =5). Various reasons were mentioned to use the toolbox, corresponding to the widely varying rating of the toolbox with a median score of 7.0 (range 1–10).

In Table 4, results of the eSurvey are shown. The “Feedback button” was used most frequently by the respondents (43%, n=19), compliant with exposure rates. All implementation strategy elements scored sufficient. The updated PALGA website was used most often (64%, n=7). All eLearning users (n=4) agreed that this could be used for resident education. Three out of 9 mentioned that the audit and feedback reports gave them new insights to improve SSR usage locally, resulting in new local agreements. The top three of the elements of respondents included all active strategy elements. Of the respondents using one or more elements, 8 out of 37 (22%) reported an effect on SSR use. Pathologists were mostly encouraged by their colleagues (9 out of 43, 21%) to start using SSR templates. Strategy elements were reviewed as very accessible (range 75–100% agreement) and usable (range 75–100% agreement). Respondents not using elements argued that they had no specific need to use it (range 48.8–61.9%) or that they were not familiar with that particular element. The latter applied less to SSR integrated elements (range 16–17.1%) compared to external elements (range 26.2–58.5%). Including the element within SSR templates was mostly chosen as an alternative method of accessibility, for both the website and communication manual (range 61.0–71.4%). Providing audit and feedback was specifically suggested as being a responsibility for the Dutch Pathology Association by 6 out of 41 respondents (15%).

Two thirds of the respondents (n=30) perceived barriers related to SSR use. Most perceived barriers were rigidity of SSR templates (42%, n=19), time needed to use SSR, and content of templates (24%, n=11) and SSR (20%, n=9), the last mostly being relevant for the reporting of ovarian cancer. Potential solutions to these barriers mentioned by pathologists and pathology residents were to improve user-friendliness, readability of SSR by using less punctuation and duplicate terminology, to integrate speech recognition and automatic transfer of clinical data.

There was no change in SSR usage based on the audit and feedback reports.

Pathology laboratories using the “Feedback button” ≥3 times during strategy introduction (n=18), significantly increased SSR use (+4.4%, p=.004), whereas the pathology laboratories using the “Feedback button” ≤ 3 times or not at all (n=24), SSR usages did not change after strategy introduction (−1.4%, p=.099). This was also significant different between groups (+4.2%, p=.016). This was a lasting effect, with no changes in trends after the introduction period (−0.1% per week, p=.463 vs. −0.03% per week, p=.610).

The use of e-learning by pathologists of specific pathology laboratories (n=9) did not lead to a change in SSR usage (−0.6%, p=.844), whereas the pathology laboratories not using e-learning (n=33) increased SSR usage after strategy introduction (+2.1, p=.019). Both groups showed no changes in trends after implementation (−0.1% per week, p=.827; −0.1% per week, p=.161). Both the change in SSR use (−3.3%, p=.386) as well as the change in trend in SSR use (−.064% per week, p=.791) were not significant between groups.

Of the pathology laboratories, 7 used 0 active strategy elements, 19 used 1 active strategy elements, and 14 used 2 or 3 active strategy elements. The use of multiple active strategy elements (2 or 3) did not lead to more a change in SSR use (+2.4%, p=.166) compared to using 1 or 0 active strategy elements, respectively +0.8%, p=.283 and +0.6%, p=.735), and this was not significant between groups using 1 active strategy element (+0.5%, p=.812) and 2 or 3 active strategy elements (+0.9%, p=.654) compared to the group using 0 active strategy elements. In addition, the trend in SSR per week did not change when multiple or a single active strategy element was used, respectively −0.1% per week (p=.513) and −0.1% per week (p=.306) compared to using 0 active strategy elements (−0.2% per week, p=.079), and this was also not significant between groups using 1 active strategy element (+0.2% per week, p=.250) and 2 or 3 active strategy elements (+0.1% per week, p=.382) compared to the group using 0 active strategy elements.