The Guideline Language and Format Instrument (GLAFI): development process and international needs assessment survey

In order to identify candidate domains for inclusion in a guideline language and format tool, we started by extracting all attributes in the “Communicating content” tactic in the GUIDE-M implementability framework [5]. We further complemented this list with all language and format attributes and sub-attributes presented in Kastner and colleagues’ 2015 realist review [2] (which contained more detailed sub-domains than the GUIDE-M). Our goal was to fashion these attributes as actionable constructs that may facilitate implementability. Guideline experts MK and SG reviewed this comprehensive list independently to identify all constructs that could be included in a language and format tool. Criteria for inclusion were (1) evidence exists that adhering to the practice represented in the construct improves uptake of the content; (2) feasible to explain to non-expert guideline developers through description and/or an example; (3) feasible for non-expert guideline developers to determine whether existing content adheres to the practice recommended in the construct (for assessment of existing guideline content); (4) actionable, either to improve existing content or when being considered during de-novo content production; (5) feasible for non-expert guideline developers to understand and address with minimal or no training or external guidance; and (6) distinct from direction typically provided in the process of journal typesetting (relevant for format-related concepts) (e.g., journals often have established conventions for format issues such as how subtitles are presented). All discrepancies were resolved through discussion, and the final list was vetted by a 3rd guideline expert (IF). Based on this comprehensive list of constructs and in accordance with the hierarchy presented in the referenced documents, we then created the following items: domains (global categories), subdomains (sub-categories within each domain), and action items (individual actionable recommendations with explanatory operational definitions and examples). Referring back to the realist review [2], and to original literature sources where required, we drafted a description, including both the definition and the evidence-based expected benefit of adhering to that practice, for each domain and subdomain. These items were then organized into a prototype tool.

The prototype tool was then presented to a group of 18 guideline experts participating in the annual Canadian Thoracic Society (CTS) Guideline Methodology Workshop (Vancouver, British Columbia, April 2018). This study was approved by the North York General Hospital Research Ethics Board (REB# 18-0008), and all participants provided written informed consent. Two members of our research team (SG and MK) led a 2-h workshop to introduce the prototype tool and to test its face validity. The session began with a didactic presentation on the importance of and evidence for language and format concepts, and an introduction to the prototype tool. Next, participants were organized into 4 small groups (4–6 individuals per group, with diverse guideline development experience, roles, and expertise, and representation from both organizations, where possible). To test the face validity of the prototype, each group was asked to apply the paper-based prototype tool on 4 specific guideline recommendation examples from recent CTS (3) or Chest (1) guidelines (Additional file 1). The goal for each small group was to optimize the language in each guideline recommendation by using the tool to identify and address language concerns. At the conclusion of the small-group work, the moderators reviewed the language issues identified for all recommendations with the entire group of workshop participants, presented a proposed revised version for each, and solicited feedback (recommendations and suggestions for content and usability improvements) on the items and the overall prototype tool.

At the conclusion of the workshop, consenting participants completed an anonymous paper-based evaluation survey capturing demographic information and perceived usefulness of the prototype tool, including a Likert scale ranking the usefulness of each action item (individual actionable recommendations within domains). Any action item with a mean Likert scale usefulness rating of < 4/5 was re-structured in the prototype tool (i.e., the description and/or accompanying example were re-drafted). Lead authors (MK, SG) also assessed all open-ended feedback in the questionnaire and made corresponding improvements to the structure, descriptions, and content of included elements.

Next, we used a document analysis approach to identify guidance pertaining to language and format constructs in existing commonly used guideline tools/approaches (“tools”) [7]. We selected tools that were identified by the GUIDE-M group for comparative analysis, as per the following criteria applied by GUIDE-M: (i) published or unpublished reports freely available in the public domain; referenced in the realist review of guideline implementability domains [2]; (ii) designed to provide practical advice related to guideline development, reporting or appraisal; and (iii) perceived by experts to be in wide use internationally [5]. The original list of guideline tools that met all of these criteria were: AGREE II [8], IOM standards [9], the Guideline International Network (G-I-N) standards [10], Guidelines 2.0 [11], ADAPTE [12], and GRADE [13]. The GUIDE-M group also added the GLIA instrument [3], as it specifically addresses guideline implementability. In our assessment of their eligibility, we eliminated the ADAPTE tool [12], since it focuses on adaptation of existing guidance for a specific context, rather than de novo guidance development; and added the AGREE-REX tool [14], which was developed in response to a gap identified in the GUIDE-M analysis, and which our research team perceived to be an emerging tool of importance in optimizing guideline credibility and implementation.

To perform our document analysis, we identified the most recent and most readily available version of each guideline tool (i.e., the version most likely to be used by guideline developers, as opposed to derivative or explanatory publications). Two reviewers (SG and KP) independently analyzed each tool to identify any language and format guidance, including advice that matched any existing items in our prototype tool. We also sought to identify any new concepts. Using an Excel^TM spreadsheet containing each item of our prototype tool, each reviewer worked independently to identify and match elements of the existing guideline tools with those within the language/format constructs in our tool, adding specific quotes and page references next to the identified item.

For each of our language and format items that were identified within the guideline tool, reviewers further qualified the nature of the guidance by classifying the item as either (i) mentioned in the guideline tool (alluded to without description); (ii) described in the guideline tool (provided a description and /or explanation of the item, with or without a rationale, but without guidance on how a guideline developer would operationalize it in practice); or (iii) operationalized in the guideline tool (provided sufficient detail for a guideline developer to take action and apply the item in their guideline writing/formatting). These independent analyses were reviewed by a 3rd reviewer (MK), and any discrepancies resolved by discussion and review of the original guideline tools. A descriptive summary of findings included the proportion of tools that mentioned, described, and/or operationalized each item (the denominator used for total items was all action items + any domain/subdomain we found addressed in an existing tool), and the proportion of items that were mentioned, described, and/or operationalized in each existing tool. For any newly identified language or format domains, sub-domains, or action items from existing guideline tools, we included these in our prototype if they met any of the following pre-set criteria: evidence for effect on uptake of content; recommendation found in more than one existing guideline tool; or consensus among research team members that the element adds practical value for the target user of the guideline (e.g., improving efficiency of consuming guideline information) without apparent deleterious consequences.

Next, we conducted a survey with an internationally representative group of guideline developers to measure perceptions of the importance of language and format items and the adequacy of existing resources to address these items, and the potential usefulness of a targeted tool addressing these issues. The study was approved by the North York General Hospital Research Ethics Board (REB# 18-0008), and all participants provided written informed consent.

We aimed to recruit a broadly representative sample of both Canadian and international guideline developers. To identify target participants, we searched for English language guidelines indexed in the: Canadian Medical Association’s Joule Clinical Practice Guideline (CPG) Infobase (a database of over 1200 recent (last 2 years) evidence-based, rigorously produced guidelines developed or endorsed by authoritative medical or health organizations in Canada) [15] (January 2017–July 2019); and the Guidelines International Network (G-I-N) International Guideline Library (a library of over 6500 guidelines developed or endorsed by organizations around the world) [16] (January 2014–July 2019). For each unique guideline, we retrieved the original guideline publication and documented the corresponding author’s email address. Where a corresponding author or their contact information could not be identified, we searched for the email address of the first author, last author, committee chair, or committee co-chair (in this order of preference).

We emailed each identified author to provide a brief introductory background and a link to a ̴ 10 min survey (SurveyMonkey^TM) (September 2019). To maximize response rates, we sent non-responders a reminder email 2 weeks after the original email and remaining non-responders another email 1 week after that. For any undeliverable email addresses, we attempted to identify alternative contacts for authors from the same guideline publication, applying the same priority as that noted above.

The survey was developed iteratively by authors SG, MK, and RT, with serial edits based on pilot testing and feedback from 3 external guideline experts on questionnaire content, clarity, and length. The survey described the concept of the tool, prior work, and definitions of the 4 main proposed subdomains under “language” and “format” (definitions available in Fig. 1). It included Likert-scale and open-ended questions, and aimed to capture respondent: (i) demographics; (ii) perceptions of the guidance provided by existing guideline tools (specifically assessing the tools included in phase 2), across each language and format subdomain in our prototype tool; (iii) importance rankings of each subdomain in our prototype tool; (iv) perceptions of the importance of considering language and format items on end-user uptake of guideline recommendations; and (v) likelihood that their guideline development organization would adopt a targeted language and format tool in their guideline production process. We invited respondents to indicate any additional tools/approaches used by their organization that were not among the tools included in our phase 2 analysis, and planned to add any tool used by ≥ 10% of responding organizations to our phase 2 analysis. Quantitative survey responses were analyzed using descriptive statistics (frequencies, means and standard deviations).

Constructs which met inclusion criteria were organized into 3 main language domains with 4 subdomains (21 action items) and 2 main format domains with 4 subdomains (14 action items) (Figs. 1 and 2). An example of a sub-domain definition and corresponding action items in the tool is provided in Fig. 3.

All 18 guideline experts participating in the CTS Guideline Methodology Workshop agreed to participate in the study. The group included 17 clinicians and 1 guideline methodologist involved in guideline production at the CTS (15) and Chest (3). Participants were aged 40–49 (n = 7), 50–59 (n = 8), or over 60 years (n = 2), and the majority were female (n = 10) and worked in an academic setting (n = 15). Among the 17 physicians, 14 had been in practice for at least 15 years. On the anonymous feedback survey, all participants perceived (i.e., indicated a Likert scale score of 5/6/7 on a 7-point scale, with a mean score of 6.1/7) that the tool would improve the implementability of guideline recommendations. Although half perceived that it would significantly slow down the guideline production process (mean Likert score 4.3/7), almost all (14/15, 93%; mean Likert score 5.9/7) indicated that it should be used by all future CTS guideline panels. By the end of the session, 12/17 (71%) participants believed that they had adequate knowledge and expertise to improve the language and format of their guideline recommendations through use of this tool.

Five (14%) of the 35 identified action items (2 language items, 3 format items) received a mean Likert scale importance rating of less than 4 out of 5 and were re-structured (collapsed into existing items or re-worded) after study completion (Likert-scale scores for each action item are provided in Additional file 2). Based on global feedback received during the session, the tool was also divided into more clearly distinct language and format sections, and we provided additional examples explicating action items, where possible.

Of 25 language items (21 pre-defined action items + 4 domains/sub-domains we found mentioned in existing tools), 15 items (60%) were mentioned in at least one tool, 13 items (52%) described in at least one tool, and only 7 items (28%) operationalized in at least one of the seven existing guideline tools in our analysis (Fig. 2). Of 19 format items (14 pre-defined action items + 5 domains/sub-domains we found mentioned in existing tools), 13 items (68%) were mentioned in at least one tool, 12 items (63%) described in at least one tool, and only 7 items (37%) operationalized in at least one tool (Fig. 2). Accordingly, 10 of the 25 language items (40%) and 6 of the 19 format items (32%) were not mentioned (and by extension not described or operationalized) in any of the seven existing guideline tools. The pre-existing guideline tool that addressed (i.e., at least mentioned) the most language items was the GLIA (8/25–32%), the most format items was the AGREE-II (8/19–42%), and the most overall items were the GLIA (11/44–25%) and IOM (11/44–25%).

Based on our analysis of the existing guideline tools, our data allowed us to add 1 new subdomain and 5 net new action items pertaining to “language” (we added 3 new items under the new subdomain and replaced 1 existing item with 3 more detailed items under an existing subdomain) (Table 1). We did not add any new subdomains or action items pertaining to “format” but added to the existing operational definition for 1 action item (Table 1). The final tool is presented in Additional file 3.

We identified 1054 unique clinical practice guidelines from the CPG Infobase (n = 328) and G-I-N International Guidelines Library (n = 726). Among these, 210 (20%) guidelines were duplicates and 120 (11%) guidelines had no available author contacts, leaving 724 (69%) guidelines for which a contact email address was available [corresponding authors (33%); first or senior authors (35%); and guideline chair and/or co-chairs (32%)]. Further removal of 41 duplicate authors resulted in the final sample of 683 unique guideline developers (representing 724 identified guidelines) who were invited to complete the survey via email. Nine email addresses (1.3%) were invalid, and replacements could not be found. Among the remaining 674 unique eligible guideline authors, 18 (2.7%) declined to participate (one of these provided an alternate contact who did complete it), 4 (0.6%) provided no usable data, and 148 responded, for a response rate of 22.0% (Fig. 4).