Data Abstraction

We designed an instrument to record data abstracted from each eligible article (Appendix 4). The instrument includes items for patient characteristics, interventions, cointerventions, outcomes, study methods, relevance to the specific research questions, and results of the study. The instrument has two components: the first three pages of the instrument apply to all articles specified for inclusion in the study. The remaining pages are individual instruments that apply to one of the five questions. To abstract an article, a reader used the initial abstraction instrument plus one or more of the five question-specific instruments.

The first few questions of the initial abstraction instrument allowed the reviewer to determine if the article actually met the eligibility criteria listed in Table 4. If an article was determined to be ineligible, it was passed to a second reader for confirmation. As shown in Figure 3, 143 articles were eliminated at this phase based on initial eligibility criteria. The remaining 220 articles, plus an additional 67 articles recommended by technical experts or found in the reference lists of reviewed articles (total 287), were subjected to the full abstraction protocol.

Managing the Database of Articles

Retrieved articles were kept in an electronic database (EndNotes Plus^TM). User-defined fields were used to record results of the eligibility screen, review for inclusion, and number of reviews performed. Paper copies of the articles were maintained in a master file. Throughout the literature review process, any articles identified from sources such as bibliographies or reference lists, manual searches of journals not contained in electronic databases, or references recommended by outside parties were added to the electronic databases.

Specification of Level of Evidence

We used a three-level system to rate individual studies:

• Class I
• Properly designed randomized controlled trials.
• Class II
• Class II(a): Randomized controlled trials that contain design flaws preventing a specification of Class I. An example of a design flaw is failure to blind raters or lack of followup data.Class II(a): Multicenter or population-based longitudinal (cohort) studies.Class II(b): Controlled trials that were not randomized.Class II(b): Case control studies.Class II(b): Case series with adequate description of the patient population, interventions, and outcomes measured.
• Class III
• Descriptive studies (uncontrolled case series).Expert opinion.Case reports.Clinical experience.

A well-done, prospective, multicenter or population-based cohort study can provide valuable information that, in some ways, is more representative of results in actual practice than are data from a randomized trial done in a highly selected sample. However, an uncontrolled case series is generally classified as Class III, indicating a low level of confidence for inferences about effectiveness.

Why is a control group needed to make inferences about effectiveness? Information about the long-term natural history of TBI provides too little certainty about the outcome to forgo the use of a control group. If the consequences of brain injury in the absence of rehabilitation could be predicted accurately, a control group would not be needed. Some evidence suggests a natural course of recovery from TBI in both basic and complex functions (Dikmen, Reitan, and Temkin, 1983), especially in the first 6 months after injury (Bond, 1979; Levin, Benton, and Grossman, 1982; Levin, Gary, Eisenberg, et al., 1990). After 6 months, survivors of TBI recognize their disabilities as they stabilize and often maximize their functional status, but major deficits in social and leisure activities tend to persist (Oddy and Humphrey, 1980).

A "gray zone" exists between Class II and definite Class III articles. Much of the research in rehabilitation uses quasiexperimental designs. In these observational study designs, control groups are sometimes identified from a separate population of people with TBI. One study compared patients undergoing inpatient rehabilitation with a sample of people with TBI who had been treated in a region of the country where formal inpatient TBI rehabilitation was not available (Aronow, 1987). This was an entirely separate patient group, and all the data except outcome measures came from an independent database.

The main difficulty with the quasi-experimental design is lack of control over the constitution of the compared groups. Since there is no randomization and generally no control over the details of the selection process through which the study participants received their separate therapies, the groups are likely to differ in the frequency of characteristics that are associated with the outcomes of interest. Even when significant efforts are made to match the experimental and the quasicontrol groups, groups still are likely to differ significantly.

Quasiexperimental designs rely heavily on multivariate statistical analysis and matching to counter this problem. While this is common practice, making inferences about effectiveness from statistically controlled data is controversial because these methods do not necessarily guarantee that the results are reliable or that serious bias and confounding have been eliminated. There are at least three major barriers to accuracy in the practice of statistical adjustment for risks. First, all of the confounding variables must be recognized and be associated with accurate and reliable measurement instruments. Second, data on all of these confounding variables must be available in comparable formats in the compared groups. For example, in the study mentioned earlier, baseline data, such as admission functional status scores, may be available for patients who entered inpatient rehabilitation but missing for patients who did not. Third, statistical methods must be selected and applied properly.

Since the first two barriers are virtually never overcome and the third remains somewhat controversial, the execution of a statistically valid quasiexperimental study is a daunting and generally unrealized task. Most studies using retrospective data suffer from the absence of well-measured data on a number of important confounding variables. The few prospective studies generally suffer from a lack of consensus on the proper measurement instruments for many of these variables. Finally, while the standard of practice for use of statistical methods to remove confounding is not completely defined, many studies do not use, or fail to report, available methods to improve the reliability and robustness of these methods.

Much of the literature relevant to the five questions addressed in this report falls into the "gray zone" between Class II and Class III. For this reason, critical appraisal of key studies plays a particularly important role in this review. A number of characteristics of these studies were considered relevant to all rehabilitation questions and were recorded in the data abstraction form. Evaluation of the following factors played a major role in critically appraising these articles:

• Prospective collection of data.
• Complete description of parent patient population.
• Large study population size (driven by hypothesis, power, type I error threshold).
• Study setting--a single center, many centers, or population-based.
• Descriptions of reasons for referral to service being studied.
• Methods described completely enough to allow study replication.
• Complete description of rehabilitation technique in question (independent variable).
• Complete and adequate description of differences between "control" and "experimental" groups.
• Conditions determining whether study subjects did or did not receive the rehabilitation technique in question.
• Information about potential confounders, including types and severity of injury; age; and others (including, in some cases, economic status, educational level, lack of family support).
• Measurement of confounding variables using instruments validated as accurate, sensitive, and reliable.
• Payer group.
• Choice of outcome variables that are meaningful to survivors as well as caregivers.
• Use of functional status and other health outcomes rather than surrogate intermediate outcomes.
• Measurement of outcome variables using instruments validated as accurate, sensitive, and reliable.
• Timing of outcome measurements.
• Assessment of survivor characteristics and outcomes by blinded observer.
• Use of multivariate statistical analysis: Were interactions sought and controlled for? Were risk estimates calibrated? Were all relevant confounders included as candidate variables?

The criteria used to classify articles and the features to be considered in critically appraising them were discussed at the subcommittee, committee, national expert panel, and Aspen Neurobehavioral Conference levels with the goal of maintaining consensus on at least the relative stratification of individual articles.