Topic Refinement

The topic for this report was nominated in a public process. With input from a group of key informants, the topic was refined to assure its relevance to stakeholders, after which the proposed analytic framework and key questions were posted to a public website for comment. The EPC subsequently revised the analytic framework and key questions based on the comments received.

Search Strategy

We conducted a comprehensive search of the scientific literature to identify randomized controlled trials (RCTs), non-randomized comparative studies, case series, and case reports relevant to the key questions. Searches of electronic databases used the National Library of Medicine's Medical Subject Headings (MeSH) keyword nomenclature developed for MEDLINE^®, adapted as needed for other sources. For Key Questions 1-4 we combined search terms for JIA with terms for the interventions of interest; for Key Question 5 we supplemented this basic search with general terms for clinical outcomes measures and specific terms for the measures of interest. Detailed search strategies are provided in Appendix A. We also reviewed selected gray literature identified by the SRC, abstracts presented at relevant meetings (the 2008 and 2009 meetings of the American College of Rheumatology and the 2008 and 2009 meetings of the Pediatric Academic Societies), and the reference lists of relevant review articles and included studies for all key questions.

To identify literature describing the comparative benefits and harms of DMARDs (Key Questions 1-4) and the accuracy of clinical outcome measures (Key Question 5) we searched:

• MEDLINE^® via PubMed (1966 to December 23, 2010);
• EMBASE^® (1947 to December 23, 2010);
• Gray literature identified by the SRC;
• Conference abstracts (as described above);
• Reference lists of review articles and included primary studies.

Our searches identified a total of 4815 citations. We imported all citations into an electronic database (EndNote^® version X13).

Study Selection

We developed criteria for inclusion and exclusion based on the patient populations, interventions, and outcome measures specified in the key questions. The abstract screening criteria we applied are listed in Appendix B. We then applied a second, more stringent set of criteria for inclusion and exclusion at the full-text stage (Appendix B). In general, we included peer-reviewed, English-language reports of studies that had a sample population of individuals 18 years or younger with JIA according to the current ACR definition. For Key Questions 1, 2, and 4, the study duration had to be at least 3 months. To be included for Key Questions 1-4, studies had to include at least one of the DMARDs included in our list. In addition, Key Questions 1 and 2 each required comparators. For Key Question 1, the comparator was conventional treatment, and for Key Question 2, the comparator was another DMARD. Case reports could be included for Key Question 3. For Key Question 5, any treatment intervention or comparator (including none) and any study duration were acceptable. We restricted Key Question 5 to studies of specified clinical outcome measures for childhood JIA.

The remainder of this section describes in greater detail the criteria we used to screen the available literature.

Interventions and Comparators of Interest

For Key Questions 1, 2, and 4, we included DMARDs as listed in Table 1 as the interventions of interest. The comparator was conventional treatment (Table 2), defined as NSAIDs or intra-articular corticosteroids with or without methotrexate. Many studies evaluated DMARDs plus conventional treatment versus conventional treatment alone. We considered methotrexate to be a component of the test intervention if the comparator group did not receive methotrexate. We considered methotrexate to be a component of the comparator if individuals in both the treatment and comparison groups could receive methotrexate. Key Questions 3 and 5 did not require a comparator.

Table 1

DMARDs evaluated.

Table 2

Conventional treatments evaluated.

Analytic Framework

Figure 1 depicts the key questions within the context of the population, interventions, comparators of interest, outcomes, timing, and settings (PICOTS). In general, the figure illustrates how treatment of JIA in children with DMARDs versus conventional treatment (intra-articular corticosteroids and NSAIDs with or without methotrexate) may result in intermediate outcomes, such as changes in laboratory measures of inflammation, changes in the active joint count, or radiographic progression of disease, and/or long-term outcomes, such as clinical remission, changes in quality of life, changes in growth, and changes in development. Also, adverse events may occur at any point after the treatment is received.

Figure 1

Analytic framework.

Data Extraction

We developed separate data abstraction form/evidence table templates for abstracting data from included studies that addressed treatment effects (benefits and adverse effects) and the performance of clinical outcome instruments (Appendix C). Abstractors worked in pairs: the first abstracted the data, and the second over-read the article and the accompanying abstraction to check for accuracy and completeness. Completed evidence tables are provided in Appendix D.

For studies reporting efficacy outcomes, we extracted the following data from clinical trials and cohort studies: geographical location; study dates; funding source; interventions (including dose, duration, dose titration protocol [if any], and cointerventions [if any]); study design; population characteristics (including age, sex, race/ethnicity, type of JIA, baseline severity, and comorbidities); recruitment setting; inclusion and exclusion criteria; numbers screened, eligible, enrolled, and lost to followup; and results for each outcome.

For adverse events, we also abstracted data from case series and case reports. We developed an Excel spreadsheet to abstract the following data from both the peer-reviewed, published literature, as well as the gray literature, including published abstracts and letters to the editor: DMARD interventions, study design, total sample size, intervention sample size, gender, and the nature of the adverse event. There was wide variability across studies in how adverse events were defined, ascertained, and reported, and different terms were used to report similar events (e.g., “rash,” “skin changes,” “dermatitis,” or “dermatologic event”). To facilitate comparisons across studies and interventions for the purpose of this report, we developed a classification system that included 29 categories (including death), plus an “other” category. Patients who experienced multiple different adverse events thus contributed data points to the respective adverse event categories. We did not abstract multiple symptoms for a given patient when these symptoms were all attributed by the authors to a given diagnosis (e.g., a patient diagnosed with pneumonia and reporting symptoms of cough, fever, chest pain, and dyspnea contributed only to the “respiratory” adverse event category). We included a given diagnosis only once (e.g., we classified “pneumonia” as a respiratory adverse event rather than “infection”). A single investigator abstracted, categorized, and summarized the adverse events data for this report. Results are given in Appendix E.

Quality Assessment

For Key Questions 1, 2, and 4, we used the criteria to assess the quality of individual controlled trials and prospective cohort studies described in AHRQ's Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews.⁵

Individual studies were graded as “good,” “fair,” or “poor” in quality according to the following definitions:

A “good” study has the least bias and results are considered valid. A good study has a clear description of the population, setting, interventions, and comparison groups; uses a valid approach to allocate patients to alternative treatments; has a low dropout rate; and uses appropriate means to prevent bias, measure outcomes, and analyze and report results.

A “fair” study is susceptible to some bias, but probably not sufficient to invalidate the results. The study may be missing information, making it difficult to assess limitations and potential problems. As the fair-quality category is broad, studies with this rating vary in their strengths and weaknesses. The results of some fair-quality studies are possibly valid, while others are probably valid.

A “poor” rating indicates significant bias that may invalidate the results. These studies have serious errors in design, analysis, or reporting; have large amounts of missing information; or have discrepancies in reporting. The results of a poor-quality study are at least as likely to reflect flaws in the study design as to indicate true differences between the compared interventions.

If a study was rated as fair or poor, assessors were instructed to note important limitations on internal validity related to the following variables:

1. Initial assembly of comparable groups.
2. Maintenance of comparable groups (includes attrition, crossovers, adherence, and contamination).
3. Important differential loss to followup or overall high loss to followup.
4. Measurements: Equal, reliable, and valid (includes masking of outcome assessment).
5. Clear definition of interventions.
6. All important outcomes considered.
7. Analysis: Adjustment for potential confounders for cohort studies, or intention-to-treat analysis for RCTs.

Assessment of each study's quality was made by a single rater and then evaluated by a second rater. Disagreements were resolved by consensus. Final quality assessments for individual studies are included in the evidence tables (Appendix D).

Quality was not rated for the case reports and case series included for Key Question 3. No established quality measurement evaluation systems have been developed for studies evaluating the reliability and validity of clinical outcome measures (Key Question 5). We therefore adapted pertinent criteria from the QUADAS tool used to assess the quality of diagnostic tests studies.⁶ We considered the selection of study participants, independent and blind comparison of the study instrument to other outcome measures, and the appropriateness of the analytic approach.

Rating the Body of Evidence

We assessed the strength of the body of evidence for each key question using the a modified version of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework.⁷ Unlike GRADE, the EPC GRADE method does not make specific clinical recommendations, uses “low” to encompass the original GRADE categories of “low” and “very low,” and uses “insufficient” when an estimate of effect cannot be generated.⁵ In rating the strength of evidence, we considered the number of studies, the size of the studies, strength of study design, and the quality of individual studies. In addition, as part of the GRADE framework, we assessed the consistency across studies of the same design, consistency across different study designs, the magnitude of effect, and applicability. Finally, if applicable, we considered the likelihood of publication bias and (especially for observational studies) the potential influence of plausible confounders. We commented specifically when it was difficult or impossible to assess certain of these dimensions. The overall strength of a given body of evidence was rated qualitatively using the following four-level scale:

High—High confidence that the evidence reflects the true effect. Further research is very unlikely to change our confidence in the estimate of effect.

Moderate—Moderate confidence that the evidence reflects the true effect. Further research may change our confidence in the estimate of effect and may change the estimate.

Low—Low confidence that the evidence reflects the true effect. Further research is likely to change the confidence in the estimate of effect and is likely to change the estimate.

Insufficient—Insufficient evidence to make a decision or assign high, moderate, or low grade.

Assessing Applicability

We followed the recommendations in AHRQ's Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews⁵ by abstracting data on the population studied, the intervention and comparator, the outcomes measured and timing of assessments. We used these data to evaluate the applicability to clinical practice, paying special attention to study eligibility criteria, symptom severity and categories of JIA for the included sample, DMARD dose and comparators, and clinical relevance and timing of the outcome measures. Using notations on applicability from the evidence tables along with our summary tables, we summarized issues of applicability qualitatively.

Data Synthesis

We planned to perform meta-analysis if there were sufficient studies that were conceptually homogeneous and reported the needed data to compute a summary estimate. In deciding whether to conduct meta-analyses, we considered primarily the basic study design (e.g., RCT), the intervention, and the comparator. Because of the small number of included studies and heterogeneity in comparisons, meta-analysis was conducted for only one comparison; all other literature was synthesized qualitatively. Meta-analysis was performed using Review Manager, version 5.0.24.⁸ The pooled effects estimate for the binary outcome was expressed as a risk ratio (RR) with 95 percent confidence interval (CI). We tested the difference in estimates of treatment effect between the treatment and control groups using a 2-sided z test with statistical significance considered at a P value of less than 0.05. We examined heterogeneity by using the Cochran Q and the I² test.^9,10 We predefined heterogeneity as low, moderate, and high, with I² statistics greater than 25 percent, 50 percent, and 75 percent, respectively.⁹ Meta-analysis with a fixed-effect model was utilized because the observed heterogeneity was low.¹⁰ For Key Question 3, we used results from clinical trials and cohort studies to describe rates of adverse effects. We used case reports and case series to describe potential adverse events that have not been reported in clinical trials.

Peer Review Process

Peer review was conducted to provide independent evaluation of the systematic review methods and content. External stakeholders nominated to review this report included clinicians and representatives of professional societies, as well as members of the TEP. AHRQ concurred with these nominees to conduct peer review based on an assessment of their independence and expertise. The review was also available for public comment by other stakeholders and experts.