Research Questions and Scope of Work

A multidisciplinary research team was assembled, with participation of members of the nominating organizations (the subcommittee of the AAP on ADHD and the Deputy Medical Director of the APA), consumer groups, local experts, and research staff (Appendix B). The local experts and research staff constituted a "local research team." All others were regarded as partners. To identify the research questions for this Task Order, the group engaged in multiple consultations and considered the findings of a systematic review of published systematic reviews and meta-analyses (Appendix C). This process led to the identification of the following questions to be addressed by the evidence report:

• What is the evidence from comparative studies on the effectiveness and safety, both short and long term, of pharmacological and nonpharmacological interventions for ADHD in children and adults?
• Are combined interventions more effective than individual interventions?

An extensive period of time at the outset of the report process was spent consulting with members of the nominating partner groups, AHRQ, and the research team in order to refine the general questions. To answer these questions and avoid duplication of work, make efficient use of the resources available, and ensure maximum added clinical value, the scope of the evidence report focused on the following seven categories of research studies:

• Studies with drug-to-drug comparisons of pharmacological interventions.
• Placebo-controlled studies evaluating the effect of tricyclic antidepressants.
• Studies comparing pharmacological with nonpharmacological interventions (drug vs. nondrug studies).
• Studies evaluating the effect of long-term therapies (>12 weeks).
• Studies evaluating therapies for ADHD in adults (>18 years of age).
• Studies evaluating therapies given in combination.
• Studies evaluating adverse effects of pharmacological interventions.

The above categories will be described in detail in the Methodology chapter. A separate category of studies comparing the short-term effect (<3 months) of stimulants with placebos was not included in the scope of work for this Task Order. Individual studies comparing stimulants with placebo were included in this report only if they met the inclusion criteria for any of the other categories. The main reason behind this decision was that, as mentioned above, another group had been commissioned to cover this issue. Most of the members of the research team considered that even if an additional review was conducted, the conclusions would not be different from those obtained by the researchers at University of British Columbia, given that most studies to date have shown consistently that stimulant medication improves core symptoms, at least in the short term (Kavale, 1982; Ottenbacher and Cooper, 1983; Thurber and Walker, 1983). Therefore, it was concluded that a new systematic review on this topic would have added little to the current state of knowledge and would have consumed all the resources available to answer other questions. Instead, the team decided to focus attention on studies comparing the long-term effects (>12 weeks) of stimulants with placebo and complement the findings with those of previous systematic reviews and the concurrent work conducted at the University of British Columbia. In summary, focusing the scope of work on the seven categories of studies described above was the strategy used by the research team to ensure maximum added value, avoid duplication of work, and present the unique contributions of this Task Order to current knowledge on the treatment of ADHD in children and adults.

Inclusion and Exclusion Criteria

Citations of individual studies were regarded as potentially eligible and selected for further evaluation if they met the following generic criteria:

• They focused on the treatment of ADHD in humans.
• They were published, in any language, in peer-reviewed journals as a full report.

Hard copies were obtained for all potentially eligible studies. They were independently assessed by two members of the research team who then decided, by consensus, whether to include the study in one or more of the categories covered in the evidence report. If the studies included conditions other than ADHD, they were included if separate analyses for patients with ADHD were provided.

With the exception of the systematic review of adverse effects, the focus was placed on evidence provided by randomized controlled trials (RCTs), the simplest and more powerful research design to evaluate the efficacy and effectiveness of interventions. The specific inclusion criteria for each of the categories listed below were agreed to by all the members of the research team.

Literature Search

Citations of potentially eligible studies were identified through a systematic search of:

• MEDLINE (from 1966), CINAHL (from 1982) and HEALTHStar (from 1975), PsycINFO (from 1984), and EMBASE (from 1984) using the search strategy described in Appendix D. All databases were searched from the date of their release to November of 1997. The strategy was designed for search of MEDLINE and was modified to meet the specific features of CINAHL, EMBASE and PsycINFO.
• The Cochrane Library (issue 4, 1997).
• The reference lists any eligible article identified in any of the above sources.
• Web sites of organizations funding research on the treatment of ADHD.
• Files of members of the research team and partner organizations.

Data Extraction

Data extraction forms were especially developed and tested for this project (Appendix E). After consultation with all the members of the local research team, the partners, and the TOO, the forms were approved for content. Two reviewers extracted data independently from each of the full reports. Any differences were resolved by consensus and by referring to the information in the original report. Any differences that could not be resolved by the two reviewers who extracted the data were resolved by the Task Order Leader or by a member of the local research team designated by him.

The original reports were not masked because an empirical methodological study showed results indicating that masking was time consuming and did not have an important impact on the results of systematic reviews (Berlin, 1997).

The information extracted addressed 41 different aspects (Appendix E, pages 1 to 5) of the studies, which were selected by the team members a priori. Of these, 25 elements were regarded as essential to judge the validity of each of the studies. The selection of five of these elements (questions 37 to 41 in Appendix E) was supported by empirical methodological evidence from studies that showed a direct relationship between these elements and the likelihood of bias in RCTs. The remaining 20 elements were selected by consensus among all the research team members as clinically important but are not based on empirical methodological research.

Data Synthesis

Descriptive statistics were calculated for all the fields of the database. Evidence tables were constructed to describe the most salient characteristics of the eligible studies. These tables, which can be found after the References in this report, summarize, globally and category by category, all the information that was extracted.

Within each of the categories, four sets of tables were produced. The first three sets were regarded as the main evidence tables to support the information included in the Findings chapter, whereas the fourth was regarded as a separate source of information for readers interested in the data extracted in a very detailed fashion. The sets of tables have the following general structure:

• The first set contains a summary of the key characteristics of the studies reviewed, including the name of the first author, the year of publication of the study, the type of study design, the number of patients randomized (or the number of patients analyzed, if the number of patients randomized was not available), the diagnosis model, the interventions studied, the duration of exposure to each of the interventions, the quality scores, the number of elements extracted from the articles among those regarded as essential to judge the validity of each of the studies but that were not supported by empirical evidence (20 in total), the outcomes of interest measured (following the list provided above), and the key results for the outcomes of interest (expressed in terms of statistical significance set at p<0.05). Data on outcomes other than those identified a priori as being of interest were not reported in these tables. They were mentioned in the text of the report when they had clinical significance.
• The second set of tables includes a summary of the presence or absence in each of the studies of each of the 20 elements extracted from the articles that were regarded as essential to judge the validity of each of the studies but that were not supported by empirical evidence. The last column of these tables provides a count of these elements by article. This count should not be regarded as a quality score because no evidence exists on the relative weights of each of these elements. The counts were included to provide the reader with additional information to guide his or her decisions. The cells of all the other columns only include 1 and 0, representing whether the element was present or not in the articles, respectively.
• The third set of tables has the same structure as the second set, but instead of 1 and 0, the cells include the actual information provided in the study reports and extracted as part of the review.
• The fourth set of tables focuses on the actual results of the studies. It contains the name of the first author, the year of publication, the interventions, the tests used to measure the outcomes, data on each of the outcomes of interest measured with each of the tests, and the statistical significance (p<0.05) of the results on the outcomes of interest across the study groups. This table, which was requested by the AAP subcommittee, was subdivided by group of outcomes to help establish whether the effect of the interventions varies by outcome (e.g., intervention A may improve core symptoms but not academic achievement). One limitation for the production and interpretation of this set of tables derives from the fact that many tests were used to measure the outcomes and that even when the same test was used, the authors often made modifications that were not clearly described in the reports. Because of the magnitude and complementary nature of the tables, they are included in the report as a separate section. To make the data extracted easier to understand, the information on the studies was organized in the tables following an alphabetical order. The studies are also presented in alphabetical order within the text. In addition, every effort was made to describe the studies according to their methodological quality, taking into account both the evidence-based criteria and the count of the elements that were not supported by empirical evidence.

The local research team, in consultation with members of the partner organizations and the TOO, evaluated the overall quantity and quality of the data available. This evaluation led to the conclusion that meta-analysis would be inappropriate to summarize the evidence on each of the research questions or for each of the main categories of interest. The main reasons for this decision were substantial clinical heterogeneity across the studies (e.g., therapies evaluated, patient populations, duration), inconsistency in outcome measurements, low methodological quality, and incomplete data reporting (see detailed descriptions within each category). The use of meta-analysis to synthesize this type of data has been associated with a greater chance of obtaining imprecise and potentially misleading results (Ioannidis, Cappelleri, and Lau, 1998). Therefore, this report represents a systematic qualitative review of the existing evidence, emphasizing the implications for clinical practice and the directions that future researchers could take to fill existing knowledge gaps.

Every effort was made to present the information obtained in each of the categories following a uniform format. Overall, each section begins with a general description of the most salient characteristics of the studies and ends with a summary of the main findings of the studies. The way in which the results from the individual studies are presented varies substantially across sections. This reflects the different perspectives that the questions provide on the treatment of ADHD. For instance, two of the sections focus on pharmacological interventions (e.g., drug-to-drug comparisons, and antidepressants vs. placebo), whereas others focus on combined interventions, length of therapy, specific populations (e.g., adults), and outcomes (e.g., adverse effects). The tables complementing the text in each of the sections, as mentioned above, follow the same format.

Consultation With Partners

The production of the evidence report involved continuous communication among members of the local research team, the TOO, and representatives of the partner organizations.Three face-to-face meetings were held; and communication by telephone, fax, and electronic mail was frequent. The partner organizations provided input during each of the steps of the process, from the formulation of the questions through the literature search and data extraction, to data synthesis.

Peer-Review Process

Potential peer-reviewers were identified in April 1998. Thirty-eight individuals were approached by the San Francisco Cochrane Center and asked if they would be willing to review this evidence report. Twenty of these people responded positively to this request, and the report was sent out to them in early July. The potential reviewers included 14 content experts (practicing physicians, psychologists, researchers, representatives of organizations interested in the treatment of ADHD), 3 consumer representatives, and 3 methodologists. All reviewers were sent a "Structured Format for Referee's Comments" (Appendix G) and encouraged to provide comments on the text. The comments were returned to our Criticism Editor, Dr. Patricia Huston, who synthesized them for the McMaster EPC. All comments were shared with the Task Order Officer. All comments were reviewed and, where feasible, incorporated into the final report. Comments were received from 15 of the 20 reviewers who were invited to comment (Appendix G).