Inclusion and Exclusion Criteria

Table B-1 outlines the inclusion and exclusion criteria related to populations, interventions, comparators, outcomes, timing, and settings (PICOTS), and study designs of interest for each Key Question (KQ):

KQ1.

In adults with chronic pain, what are the benefits of cannabinoids for treatment of chronic pain?

KQ2.

In adults with chronic pain, what are the harms of cannabinoids for treatment of chronic pain?

KQ3.

In adults with chronic pain, what are the benefits of kratom or other plant-based substances for treatment of chronic pain?

KQ4.

In adults with chronic pain, what are the harms of kratom or other plant-based substances for treatment of chronic pain?

Important subgroups to consider in evaluating this evidence are:

• Specific types of pain: neuropathic pain (including nociceptive and centralized; patients with multiple sclerosis and painful skin disorders are included in this category), musculoskeletal pain (including low-back pain), visceral pain, fibromyalgia, inflammatory arthritis, headache disorders, sickle cell disease, and cancer pain (non-end of life)
• Degree of nociplasticity/central sensitization
• Patient demographics (e.g., age, race, ethnicity, sex, socioeconomic status)
• Comorbidities, including past or current substance use disorders, mental health disorders, medical comorbidities, and high risk for opioid use disorder)
• Plant-based compound characteristics: route of administration, frequency of administration, potency of product, dose or estimated dose, specific compounds (e.g. tetrahydrocannabinol, cannabidiol, terpenes, flavonoids), and specific formulations used
• Co-use of other interventions for pain: opioids, nonopioids (e.g., nonsteroidal anti-inflammatory drugs, acetaminophen, gabapentin, pregabalin)

Below are additional details on the scope of this project:

• Study Design: For all Key Questions, we included randomized controlled trials (RCTs) of at least 4 weeks duration. Initially, in the base-year of this living systematic review, we included observational studies for both benefits (to address gaps in evidence where RCTs are not available) and harms. Eligible observational studies must have assessed a mean duration of treatment of at least 4 weeks, and have concurrent controls (e.g., cohort and case-control studies). Those controlling for potential confounders were prioritized. As the evidence grows, and more RCTs become available throughout the project, we will reassess the need to include observational studies, specifically to address benefits. A decision to discontinue including them will be made based on the strength of the RCT evidence. When the RCT evidence on a given Key Question and outcome is insufficient, we will include observational studies that meet inclusion criteria. When the strength of evidence is low, moderate, or high based on RCTs, we will update our protocol to exclude observational studies. We do not anticipate excluding observational studies assessing harms. For all Key Questions, we excluded uncontrolled observational studies, case series, and case reports. Systematic reviews were used to supplement searches and identify primary studies.
• Non-English Language Studies: We restricted to English-language articles, but reviewed English-language abstracts of non-English language articles to identify studies that would otherwise meet inclusion criteria in order to help assess for the likelihood of language bias.

Study Selection

Electronic searches for evidence were conducted in Ovid^® MEDLINE^®, PsycINFO^®, Embase^®, the Cochrane Library, and SCOPUS^® databases through February 4, 2021. Searches were initially run in September 2020 with ongoing, automated monthly searches to identify newly published studies. Search strategies are available in Appendix A. Electronic searches were supplemented with review of reference lists of relevant studies and reviewing the two prior AHRQ pain reports^1,2 for studies that met our inclusion criteria. A Federal Register Notice was posted, and a Supplemental Evidence And Data for Systematic review (SEADS) portal was available for submission of unpublished studies. As part of living systematic review methods, the electronic searches were automated to be run on a biweekly basis, with results emailed directly to the EPC librarian and the research team for processing. Citations were uploaded into DistillerSR® software for study selection management.

The pre-established criteria listed above were used to determine eligibility for inclusion and exclusion of abstracts. Using Distiller® SR, the review team conducted manual online assessment of study citations. All citations deemed potentially relevant by at least one of the reviewers were retrieved for full-text review. To ensure accuracy, any citation deemed not relevant for full-text review were reviewed by a second researcher. We initially planned to explore using the Distiller® AI feature to automate exclusion of abstracts that are clearly not relevant. Briefly, Distiller®SR AI is training in the background, learning from the human decisions on abstract eligibility. When the Distiller® AI decisions reach a level of 95 percent accuracy, we will deploy the system to assist with dual review (this typically takes 2000 citations, but varies by topic).³ To date, the biweekly citation counts have been low, and the AI feature has not been utilized.

Data Extraction

After studies were selected for inclusion, data were abstracted into categories that included but are not limited to: study design, year, setting, country, sample size, eligibility criteria, population and clinical characteristics, intervention characteristics, and results relevant to each Key Question as outlined in the previous inclusion and exclusion criteria section. Information that was abstracted that was relevant for assessing applicability included the number of patients randomized relative to the number of patients enrolled, use of run-in or wash-out periods, and characteristics of the population, intervention, and care settings. All study data were verified for accuracy and completeness by a second team member. On a quarterly basis, any newly identified studies were abstracted and evidence tables updated. Quarterly reports were published to the Agency for Healthcare Research and Quality (AHRQ) website, and evidence tables are updated in AHRQ’s Systematic Review Data Repository Plus (SRDR+).

Risk of Bias Assessment of Individual Studies

Predefined criteria were used to assess the risk of bias of individual controlled trials, systematic reviews, and observational studies. RCTs were evaluated using criteria and methods developed by the Cochrane Back Review Group,⁴ and cohort and case-control studies were evaluated using criteria developed by the U.S. Preventive Services Task Force.⁵ These criteria and methods were used in accordance with the approach recommended in the chapter, Assessing the Risk of Bias of Individual Studies When Comparing Medical Interventions in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews developed by AHRQ.⁶ Studies were given an overall rating of “low,” “medium,” or “high” risk of bias. We used DistillerSR^® software to conduct these assessments, using dual review by two independent reviewers. Disagreements identified by DistillerSR^® were resolved through consensus. Assessments and final ratings were converted to evidence tables, and will be uploaded on a quarterly basis to SRDR+.

Data Synthesis and Analysis

We constructed evidence tables showing study characteristics (as discussed above), results, and risk of bias ratings for all included studies, and summary tables to highlight the main findings. Data were qualitatively summarized in tables, using ranges and descriptive analysis and interpretation of the results. Studies identified in prior AHRQ chronic pain reports^1,2 that meet inclusion criteria are included in this review. We evaluated the persistence of benefits or harms by evaluating the three periods identified in prior AHRQ pain reports (3 to <6 months, 6 to 12 months, and ≥12 months).^1,2,7–9

Meta-analyses were conducted to summarize data and obtain more precise estimates on outcomes for which studies were homogeneous enough to provide a meaningful combined estimate.¹⁰ The decision to conduct quantitative synthesis depends on presence of at least two studies, completeness of reported outcomes and a lack of heterogeneity among the reported results. To determine whether meta-analyses were indicated, we considered the risk of bias of the studies and the heterogeneity among studies in design, patient population, interventions, and outcomes. Meta-analyses were conducted using a random effects model based on the profile likelihood method,¹¹ and statistical heterogeneity was assessed using the I² method. Publication bias (small sample size bias) was assessed using funnel plots when there are eight or more studies in meta-analyses. To evaluate subgroup effects, we summarized within-study analyses of subgroup differences and performed study-level analyses on key demographic and clinical factors. Sensitivity analyses were conducted on study risk of bias.

The magnitude of effects for pain and function is classified using the same system used in other recent AHRQ Evidence-based Practice Center (EPC) reviews conducted on chronic pain^1,2,7–9 to provide a consistent benchmark for comparing results of pain interventions across reviews. Table B-2 provides thresholds for determining the magnitude of effect. A small effect is defined for pain as a mean between-group difference following treatment of 5 to 10 points on a 0- to 100-point visual analog scale (VAS), 0.5 to 1.0 points on a 0- to 10-point numeric rating scale, or equivalent; for function as a mean difference of 5 to 10 points on the 0- to 100-point Oswestry Disability Index (ODI) or 1 to 2 points on the 0- to 24-point Roland-Morris Disability Questionnaire (RDQ), or equivalent; and for any outcome as a standardized mean difference (SMD) of 0.2 to 0.5. A moderate effect is defined for pain as a mean difference of 10 to 20 points on a 0- to 100-point VAS, for function as a mean difference of 10 to 20 points on the ODI or 2 to 5 points on the RDQ, and for any outcome as an SMD of 0.5 to 0.8. Large effects are defined as greater than moderate. We apply similar thresholds to other outcomes measures. Small effects using this system may be below published thresholds for clinically meaningful effects; however, there is variability across individual patients regarding what constitutes a clinically meaningful effect, which is influenced by a number of factors such as preferences, duration and type of chronic pain, baseline symptom severity, harms, and costs. For some patients a small improvement in pain or function using a treatment with low cost or no serious harms may be important.

Findings that were not statistically significant were interpreted as follows:

• In determining the strength of evidence (SOE), the precision of evidence was downgraded two levels if inadequate sample size (optimal information size) and the 95% confidence interval includes both potentially meaningful benefit and harm (e.g. for a relative effect, the lower bound is ≤ 0.75 and the upper bound is ≥ 1.25)¹²
• If the magnitude of effect is below the threshold for a small effect, the finding is considered to have “No effect”¹
• If the magnitude of effect is small or greater, and SOE is at least Low, the finding is considered to have a “Potential effect, not statistically significant”
• If the magnitude of effect is small or greater, and SOE is insufficient, the finding is considered to have “failed to demonstrate or exclude a beneficial/detrimental effect.”¹³

Grading the Strength of the Body of Evidence

We assessed the SOE for all primary comparisons and outcomes listed in Table B-1. Regardless of whether evidence is synthesized quantitatively or qualitatively, the strength of evidence for each Key Question/body of evidence is initially assessed by one researcher for each clinical outcome by using the approach described in the AHRQ Methods Guide.⁶ To ensure consistency and validity of the evaluation, the strength of evidence is reviewed by the entire team of investigators prior to assigning a final grade on the following factors:

• Study limitations (low, medium, or high level of study limitations)
• Consistency (consistent, inconsistent, or unknown/not applicable)
• Directness (direct or indirect)
• Precision (precise or imprecise)
• Reporting/publication bias (suspected or undetected)

The SOE was assigned an overall grade of high, moderate, low, or insufficient according to a four-level scale by evaluating and weighing the combined results of the above domains:

• High—We are very confident that the estimate of effect lies close to the true effect for this outcome. The body of evidence has few or no deficiencies. We believe that the findings are stable, i.e., another study would not change the conclusions.
• Moderate—We are moderately confident that the estimate of effect lies close to the true effect for this outcome. The body of evidence has some deficiencies. We believe that the findings are likely to be stable, but some doubt remains.
• Low—We have limited confidence that the estimate of effect lies close to the true effect for this outcome. The body of evidence has major or numerous deficiencies (or both). We believe that additional evidence is needed before concluding either that the findings are stable or that the estimate of effect is close to the true effect.
• Insufficient—We have no evidence, we are unable to estimate an effect, or we have no confidence in the estimate of effect for this outcome. No evidence is available or the body of evidence has unacceptable deficiencies, precluding reaching a conclusion.

Plain-language statements are used in the Main Points, the Evidence Summary and the Discussion to convey the SOE. High SOE is described as “is associated with” or simply “reduces/increases;” moderate SOE is described as “probably;” and low SOE is described as “may be.”¹⁴

Peer Review and Public Commentary

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodological expertise. The EPC considers all peer review comments on the draft report in preparation of the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The final report does not necessarily represent the views of individual reviewers. The EPC will complete a disposition of all peer review comments. The disposition of comments for systematic reviews and technical briefs will be published 3 months after the publication of the evidence report.

Potential Peer Reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Invited Peer Reviewers may not have any financial conflict of interest greater than $5,000. Peer reviewers who disclose potential business or professional conflicts of interest may submit comments on draft reports through the public comment mechanism.

Assessing Applicability

Applicability is assessed in accordance with the AHRQ Methods Guide,¹⁵ which is based on the PICOTS framework. Applicability addresses the extent to which outcomes associated with an intervention are likely to be similar across different patients and settings in clinical practice based on the populations, interventions, comparisons, and outcomes evaluated in the studies. For example, exclusion of chronic pain patients with psychiatric comorbidities reduces applicability to clinical practice since many patients with chronic pain have such comorbidities and may respond more poorly to treatment. Similarly, trials that use active run-in periods evaluate highly selected populations who tolerated and responded well to the study intervention, rather than the general population of chronic pain patients being considered for the intervention. Factors that may affect applicability which we have identified a priori include eligibility criteria and patient factors (e.g., demographic characteristics, duration or severity of pain, underlying pain condition, presence of medical and psychiatric comorbidities, event rates and symptom severity in treatment and control groups), intervention factors (e.g., dose and duration of therapy, intensity and frequency of monitoring, level of adherence, use of co-interventions), comparisons (e.g., type and dosing of comparison), outcomes (e.g., use of unvalidated or nonstandardized outcomes, measurement of short-term or surrogate outcomes), settings (e.g., primary care vs. specialty setting, country), and study design features (e.g., use of run-in periods) relevant to applicability. We use this information to assess the situations in which the evidence is most relevant and to evaluate applicability to real-world clinical practice in typical U.S. settings, summarizing applicability assessments qualitatively.

Appendix B References

1.

Chou R, Hartung D, Turner J, et al. Opioid treatments for chronic pain. Rockville, MD: Agency for Healthcare Research and Quality 2020. PMID: 32338848 [PubMed: 32338848]

2.

McDonagh MS, Selph SS, Buckley DI, et al. Nonopioid Pharmacologic Treatments for Chronic Pain. Rockville, MD: Agency for Healthcare Research and Quality; 2020. PMID: 32338847. [PubMed: 32338847]

3.

Taieb V, Smela-Lipińska B, O’Blenis P, et al. Use of artificial intelligence with DistillerSR software for a systematic literature review of utilities in infectious disease. Value in health: The Journal of the International Society for Pharmacoeconomics and Outcomes Research. 2018 Oct;21(Supplement 3):S387. doi: 10.1016/j.jval.2018.09.2299. [CrossRef]

4.

Furlan AD, Pennick V, Bombardier C, et al. 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine. 2009 Aug 15;34(18):1929–41. doi: 10.1097/BRS.0b013e3181b1c99f. PMID: 19680101. [PubMed: 19680101] [CrossRef]

5.

U.S. Preventive Services Task Force. Methods and processes. 2019. https://www​.uspreventiveservicestaskforce​.org/uspstf/about-uspstf​/methods-and-processes.

6.

Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2018. https:​//effectivehealthcare​.ahrq.gov/topics​/cer-methods-guide/overview. Accessed June 1, 2019. [PubMed: 21433403]

7.

Skelly AC, Chou R, Dettori JR, et al. Noninvasive nonpharmacological treatment for chronic pain: a systematic review update. Rockville, MD: Agency for Healthcare Research and Quality; 2020. PMID: 32338846. [PubMed: 32338846]

8.

Chou R, Deyo R, Friedly J, et al. Noninvasive treatments for low back pain: Agency for Healthcare Research and Quality (US), Rockville (MD); 2016. [PubMed: 26985522]

9.

Skelly AC, Chou R, Dettori JR, et al. Noninvasive nonpharmacological treatment for chronic pain: a systematic review: Agency for Healthcare Research and Quality (US), Rockville (MD); 2018. [PubMed: 30179389]

10.

Morton SC, Murad MH, O’Connor E, et al. Quantitative Synthesis—An Update: Agency for Healthcare Research and Quality (US), Rockville (MD); 2018.

11.

Huizenga HM, Visser I, Dolan CV. Testing overall and moderator effects in random effects meta-regression. Br J Math Stat Psychol. 2011 Feb;64(Pt 1):1–19. doi: 10.1348/000711010x522687. PMID: 21506942. [PubMed: 21506942] [CrossRef]

12.

Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011 Dec;64(12):1283–93. doi: 10.1016/j.jclinepi.2011.01.012. PMID: 21839614. [PubMed: 21839614] [CrossRef]

13.

Guyatt GH, Norris SL, Schulman S, et al. Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012 Feb;141(2 Suppl):53s–70s. doi: 10.1378/chest.11-2288. PMID: 22315256. [PMC free article: PMC3278053] [PubMed: 22315256] [CrossRef]

14.

Gerrity M, Fiordalisi C, Pillay J, et al. AHRQ methods for effective health care. Roadmap for Narratively Describing Effects of Interventions in Systematic Reviews. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020. [PubMed: 33180401]

15.

Atkins D, Chang SM, Gartlehner G, et al. Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol. 2011 Nov;64(11):1198–207. doi: 10.1016/j.jclinepi.2010.11.021. PMID: 21463926. [PubMed: 21463926] [CrossRef]