Review Protocol

This systematic review is an update of an earlier report published in 2013 which evaluated questions related to stroke prevention in patients with atrial fibrillation (AF) and atrial flutter.¹³ Given the uncertainties which remained within the limitations of the available evidence, and the new data which have emerged since that report, an update of the systematic review was commissioned.

The Patient-Centered Outcomes Research Institute (PCORI) convened two multi-stakeholder virtual workshops in December 2016 and January 2017 to gather input from end users of research and clinical, content, and methodological experts on scoping for the updated review, prioritization of Key Questions, a discussion of changes in the evidence base since the prior review, and emerging issues in AF. The protocol for this update was developed based upon findings from the January 2017 workshop, and builds upon Key Questions (KQs) 1-3 from the original report. The finalized protocol for this systematic review update is posted on the Effective Healthcare (EHC) Web site (www.effectivehealthcare.ahrq.gov). The PROSPERO registration is CRD42017069999.

Literature Search Strategy

Inclusion and Exclusion Criteria

The PICOTS criteria used to screen articles for inclusion/exclusion at both the title-and-abstract and full-text screening stages are detailed in Table 2.

Table 2

Major therapeutic options for stroke prevention in atrial fibrillation.

Quality (Risk of Bias) Assessment of Individual Studies

We assessed methodological quality, or risk of bias, for each individual study using tools specific to the study’s characteristics. For all studies, we used the following strategy: (1) classify the study design, (2) apply predefined criteria for appraisal of quality, and (2) arrive at a summary judgement of the study’s quality. For studies assessing diagnostic accuracy, we used the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool, following guidance for use of that tool to arrive at an overall judgement as defined in Table 3.⁸⁸

Table 3

Definitions of overall quality assessment ratings for diagnostic studies.

For nondiagnostic studies, we used the Cochrane Risk of Bias tool for randomized studies^89,90 and the Risk Of Bias In Nonrandomised Studies of Interventions (ROBINS-I) tool for observational studies.^91,92 We rated each study as being of good, fair, or poor quality based on its adherence to well-accepted standard methodologies. For each study, one investigator made an assessment of methodological quality which was then reviewed by a second investigator; disagreements were resolved by consensus or by a third investigator if agreement was not reached.

Quality assessment was outcome-specific, such that a given study that analyzed its primary outcome well but did an incomplete analysis of a secondary outcome could be assigned a different quality grade for each of the two outcomes. We applied this outcome-specific quality assessment to groups of outcomes that have lower risk of detection bias (e.g., mortality) and those at higher risk of detection bias (e.g., quality of life outcomes). Studies of different designs were evaluated within the context of their respective designs.

To indicate the summary judgment of the quality of individual nondiagnostic studies, we used the summary ratings of good, fair, or poor based on the classification scheme presented in Table 4.

Table 4

Definitions of overall quality assessment ratings for nondiagnostic studies.

We did not formally re-evaluate quality ratings for articles considered in this report that were included within the original systematic review. The quality assessments performed in the original review were based on QUADAS-2 for KQs 1 and 2, and for KQ 3, on an approach described in the Methods Guide²⁸ that used a similar strategy of (1) classifying the study design, (2) applying predefined criteria for quality and critical appraisal, and (3) arriving at a summary judgment of the study’s quality. Criteria considered for each study type were derived from core elements described in the Methods Guide (details available in the prior report).¹³ When we identified additional publications describing results from a study that was included within the prior review, we reviewed the new article(s) in the context of the prior quality rating to determine if any adjustment to the prior quality rating was warranted. Quality ratings for individual studies are presented in Appendix F.

Data Synthesis

We began by summarizing key features of the included studies for each KQ. To the degree that data were available, we abstracted information on study design; patient characteristics; clinical settings; diagnostic tools; and intermediate, final, and adverse event outcomes. We ordered our findings by treatment or diagnostic comparison, and then within these comparisons by outcome, with long-term final outcomes emphasized.

We reviewed and highlighted studies using a hierarchy-of-evidence approach. The best evidence available (normally RCTs) was the focus of our synthesis for each KQ. If high quality evidence was not available, we described any lower quality evidence we were able to identify, but we underscored the elements that influenced our assessment of lower quality and the uncertainties in our findings. We assessed whether the inclusion of lower quality studies would change any of our conclusions and performed sensitivity analyses excluding such evidence where appropriate.

We determined the feasibility of completing a quantitative synthesis (i.e., meta-analysis) based on the volume of relevant literature, conceptual homogeneity of the studies in terms of study population and outcomes, and completeness of the reporting of results. We grouped interventions by prediction tool (KQs 1 and 2) and drug class or procedure (KQ 3), when appropriate. We required three appropriate studies to consider meta-analysis of intervention studies and three to consider meta-analysis of observational diagnostic test studies. Given concerns about quality, we did not include observational studies in quantitative synthesis that did not use propensity matching for controls or similar methods.

When at least three comparable studies reported the same outcome, we used the R statistical package (version 3.1.2) (The R Foundation), with the “metafor” meta-analysis library (version 1.9-7) to synthesize the available evidence quantitatively. We used the random-effects DerSimonian and Laird estimator⁹³ to generate summary values. In addition, we used the Knapp–Hartung approach to adjust the standard errors of the estimated coefficients. We explored heterogeneity using graphical displays and test statistics (Q and I² statistics), while recognizing that the ability of statistical methods to detect heterogeneity may be limited. We perform quantitative and qualitative syntheses separately by study type and discuss their consistency qualitatively. When we were able to calculate hazard ratios (HRs), we assumed that a HR between 0.8 and 1.2 with a narrow confidence interval that also crossed 1.0 suggested no clinically significant difference between treatment strategies; in such cases, we describe the treatment strategies being compared as having “comparable efficacy.” For some outcomes, study quality or other factors affected comparability; these exceptions are explained on a case-by-case basis.

For KQ 1 and KQ 2, we synthesized available c-statistics which quantify the prediction/discrimination ability of the studied tools. Since these tools are not binary, summary receiver operating characteristic (ROC) curves were not considered as would have been possible for binary diagnostic tests. The c-statistics were pooled by considering their estimated values (point estimates) and confidence intervals, and the “Generic point estimates” effect specification option in the Comprehensive Meta-Analysis software. For a clinical prediction rule, we assumed that a c-statistic <0.6 had no clinical value, 0.6–0.7 had limited value, 0.7–0.8 had modest value, and >0.8 has prediction adequate for genuine clinical utility.⁹⁴ Of note, a risk score may have a statistically significant association with a clinical outcome, but the relationship may not be discriminated enough to allow clinicians to accurately and reproducibly separate patients who will and will not have the outcome. In addition, the c-statistic value is almost always higher when assessing prediction accuracy in the patient data set used to develop the model than in independent sets of patients; we therefore indicate when studies being discussed were actually used to develop the models they describe.

For KQ 3 we focus on the statistical significance of our findings for the individual outcomes but do not make recommendations on whether specific differences are clinically relevant.

We hypothesized that the methodological quality of individual studies, study type, the characteristics of the comparator, and patients’ underlying clinical presentation would be associated with the intervention effects, causing heterogeneity in the outcomes. Where there were sufficient studies, we performed subgroup analyses and/or meta-regression analyses to examine these hypotheses.

Strength of the Body of Evidence

We identified a set of comparisons and outcomes for strength of evidence grading with the goal of selecting outcomes of greatest importance for decisionmaking. We rated strength of evidence using the approach described in the Methods Guide.^28,95 and Medical Test Guide.²⁹ We graded the strength of evidence for each outcome individually; thus, the strength of evidence for two separate outcomes in a given study may be graded differently. These grades are presented in the strength of evidence tables in the Discussion section of the report.

Briefly, the approach requires assessment of five domains: study limitations (previously named risk of bias), consistency, directness, precision, and reporting bias, which includes publication bias, outcome reporting, and analysis reporting bias. Note that reporting bias was not possible to assess for the diagnostic studies. The five domains were considered qualitatively, and a summary rating of “high,” “moderate,” or “low” strength of evidence was assigned after discussion by two reviewers. In some cases, high, moderate, or low ratings were impossible or imprudent to make—for example, when no evidence was available or when evidence on the outcome was too weak, sparse, or inconsistent to permit any conclusion to be drawn. In these situations, a grade of “insufficient” was assigned. The four-level rating scale is described in Table 5. Outcomes based on evidence from RCTs or observational studies started with a “high” or “low” strength of evidence rating, respectively, and were downgraded for inconsistency, indirectness, or imprecision. Studies of risk prediction outcomes started with moderate strength of evidence.⁹⁶ We assumed that outcomes based on only 1 study should not be downgraded for lack of consistency if the study included more than 1,000 patients. Intention-to-treat (ITT) findings were evaluated when available and form the basis of our strength of evidence ratings. When ITT findings were not available and only on-treatment findings were reported, our confidence in the stability and precision of our findings was reduced, and therefore the related strength-of-evidence rating was lowered. Finally, when outcomes were assessed by RCTs and observational studies, we focused our strength of evidence rating on the findings from the RCTs and then increased or decreased the strength of evidence rating depending on whether findings from the observational studies were consistent or inconsistent with those from the RCTs. We provided greatest weight to findings from large RCTs.

Table 5

Definition of strength of evidence grades.

Applicability

We assessed applicability across the KQs using the method described in the Methods Guide.^28,97 In brief, we used the PICOTS format to organize information relevant to applicability. The most important applicability concern is whether the outcomes observed for any individual study, with its specific patient population and methods of implementing interventions, can be confidently extrapolated to a broader context. Differences in intervention methods or study population characteristics (e.g., age, comorbidities) can affect the rates of events observed in both control and intervention groups, and may limit the generalizability of the findings. Specific criteria considered in applicability assessments are listed in Appendix B. We used these data to evaluate applicability to clinical practice, paying special attention to study eligibility criteria, demographic features of the enrolled population in comparison to the target population, characteristics of the intervention used in comparison with care models currently in use, and clinical relevance and timing of the outcome measures. We summarized issues of applicability qualitatively.

Peer Review and Public Commentary

Experts in the fields of internal medicine, cardiovascular medicine, electrophysiology, hematology, geriatric medicine, clinical trial and systematic review methodology, health services research, and patient advocates were invited to provide external peer review of the draft report. AHRQ, PCORI, and an associate editor also provided comments. In addition, the draft report was posted on the AHRQ EHC Web site from February 5, 2018, to March 22, 2018, to elicit public comment. We have addressed all reviewer comments and have documented our responses in a disposition of comments report that will be made available 3 months after the Agency posts the final systematic review on the EHC Web site. A list of peer reviewers submitting comments on the draft report is provided in the front matter of this report.