Overall Conclusions

The main overall conclusions from studies included in the Task Order are the following:

• Extensive research has been done on the treatment of ADHD for more than 25 years. However, few studies compare different pharmacological interventions head to head or with nonpharmacological treatments or evaluate the efficacy of antidepressants or combination therapies, the implications of long-term therapy, the treatment of ADHD in adults, or the adverse effects associated with the most frequently prescribed medications.
• The field would benefit from empirical methodological research evidence indicating the added value of nonrandomized within-subject and single-subject research designs for direct head-to-head comparisons between psychosocial interventions and other treatments.
• The reports of the studies available have numerous deficiencies that limit the assessment of their validity, relevance, precision, and, therefore, their clinical application. It was surprising, for instance, to find that most studies did not describe clearly important information such as the presence of comorbid disorders, the characteristics of the patients' families, the fidelity of treatments, compliance with treatment, or baseline measurement of outcomes of interest. Little information is available on the treatment of adult patients who belong to minority ethnic groups. In addition, the reporting of other important aspects of the studies was deficient.
• Most of the studies available had small sample sizes and did not provide information on the primary outcome of interest to the researchers. Therefore, it was unclear on most occasions whether statistically nonsignificant results reflected a true lack of difference among interventions or false-negative results from underpowered studies. Research during the past 20 years on this issue in other areas indicates that the latter is the most plausible situation in most cases (Freiman, Chalmers, Smith et al., 1978; Moher, Dulberg, and Wells, 1994).
• One of the most important challenges faced by researchers interested in ADHD and its treatment is to select a core set of uniform outcome measures to be used across studies in addition to those preferred by individual research groups. Standardized methods to obtain and report each of the outcomes, and the identification of clinically meaningful changes in the outcomes in response to treatment, would facilitate the interpretation of individual studies and the synthesis of data across studies.
• More research is needed to understand the relative value of the different aspects of a study. For instance, it is unclear whether having information on family characteristics is more or less important than having information on comorbid disorders. More efforts should be made to understand the implications that different diagnostic models for ADHD and comorbid disorders could have on the outcomes of treatment. This area provides a fertile ground of research on these clinically and methodologically important issues. Any future research efforts may also benefit from more communication among clinicians, providers, and consumers.
• Most reports lacked description of basic generic methodological aspects that increase their likelihood of bias. For instance, 97 percent of the articles did not describe the method of randomization, 95 percent did not provide information on concealment of allocation, and 87 percent did not describe the number of, or reasons for, withdrawals and dropouts in each of the groups. It is important to acknowledge that the studies may have included most of the elements in their protocols throughout the execution of the study but did not describe them in their report. Journal editors and peer reviewers should recognize the importance of these elements and encourage authors to include them in their reports. Most of the current problems that were encountered could be easily corrected if journal editors adopted evidence-based reporting recommendations such as the Consolidation of the Standards of Reporting Trials (CONSORT) statement (Begg, Cho, Eastwood et al., 1996) and kept track of new methodological developments that could increase the validity and applicability of research. The CONSORT statement was produced and published by an international group of clinical epidemiologists, biostatisticians, and journal editors in 1996. Its aim is to improve the standards of written reports of RCTs and to ensure that readers find all the information they require in the reports to interpret the trial results with confidence. This statement includes a checklist of 21 items and a flow diagram that authors can use to provide necessary information on the progress of patients through a study. The statement has already been adopted by over 70 major biomedical journals (Jadad and Rennie, 1998).
• The approach used in this review for the assessment of the quality of the studies by no means represents the only or most appropriate way to assess trial quality. However, the authors of this Task Order report included the only validated tool available that also appears to produce robust and valid results in an increasing number of empirical methodological studies. In addition, the scale was not used in isolation. Instead, it was complemented with separate assessments of other components for which empirical evidence of a direct relationship with bias exists. In addition, separate assessments of 20 other components were added to provide readers with a much wider and more clinically relevant picture. Any future decision on the assessment of trial validity should be made in the light of new empirical methodological evidence.
• The large number and heterogeneity of outcome measures and tests used in the studies limited efforts to compare and synthesize data across the studies included in this report.
• Researchers often use modified versions of the same tests (e.g., Conners) across studies but provide little information on their modified versions. The field would benefit by the selection of a core set of validated and clinically relevant outcomes to be measured in all the studies in addition to any other outcomes of interest to the specific groups of researchers. Few studies have been supported financially by sources other than governments or pharmaceutical companies. A great opportunity exists for consumer groups to support more research activities, given the number of important questions that remain unanswered and the implications of the results of research on the public.
• Future research efforts in the areas reviewed in this Task Order report will require collaboration and commitment among different groups of stakeholders. The MTA study funded by NIMH is an example of this type of collaboration.

The following is a description of the main implications for clinical practice and future research efforts from each of the seven categories of interest.

Limitations of This Evidence Report

The findings and conclusions of this evidence report are based on the information that was available in the published reports of the studies included. Additional information obtained directly from the authors could have overcome many of the reporting limitations described above. Contact with authors could have also led to reduction in the likelihood of publication bias through the identification of unpublished studies. The budget and time lines available, however, were insufficient to allow this.

The interpretability of these data included in most of the tables of this evidence report is limited. These data were obtained by the authors of the studies with many different instruments that were poorly described in most reports.

Another limitation of this report is that it does not include quantitative estimates of the relative effects of the interventions evaluated. However, meta-analysis was deemed inappropriate given the amount, heterogeneity, and quality of the data available. As mentioned above, the use of meta-analysis to synthesize this type of data has been associated with a greater chance of obtaining imprecise and potentially misleading results (Ioannidis, Cappelleri, and Lau, 1998). It is unclear, however, whether the problems found in this data set are greater or smaller than those in other areas. It is important, however, to recognize that qualitative systematic reviews can introduce other problems such as biased narrative description of the characteristics and findings of the studies included. By including detailed evidence tables in this report, it is hoped that readers will be able to replicate the findings, when appropriate.

It could also be argued that another limitation of this report is that it did not include a separate section for studies comparing the short-term effects of stimulants with placebo. This decision was motivated by the need to make efficient use of the resources available while ensuring the maximum added value from the Task Order report. Given the consistency of the findings of the individual studies included in this review, the three published systematic reviews, and the most recent unpublished review, as well as the limitations of the studies available, it is the belief of the authors that the decision to focus attention on the other seven categories resulted in the most efficient use of the resources available.

It could be argued that systematic reviews of behavioral interventions could benefit substantially by the inclusion of nonrandom within-subject and single-subject research designs. However, we regarded these study designs as too vulnerable to bias and of limited value for direct head-to-head comparisons between psychosocial therapies and interventions.

In many instances, it was concluded that there was a lack of evidence on the effectiveness of clinically important interventions. It is important to recognize that this is different from the lack of effectiveness of the same interventions.

Closing Remarks

This report represents a collective effort by representatives of different groups of stakeholders, including government agencies (AHRQ), professional organizations (AAP and APA), consumer groups (Children and Adults with Attention-Deficit Disorders [CHADD]), clinicians, and researchers with content and methodological expertise. Emphasis was placed on ensuring the relevance of the research questions, on building on existing knowledge, and on supporting future clinical practice guideline development or research efforts.

The report includes seven systematic reviews that incorporated state-of-the-art methodology and data that are ready for incorporation into evidence-based clinical practice guidelines or performance measures.

This report would best be used as a tool to guide future research priorities and as a platform from which to develop programs, guidelines, and policies rather than as a guideline for clinical practice. Our hope is that this report will be used to stimulate discussion, advance knowledge, and provide constructive challenge in an area of public concern: ADHD.