Appendix H. Original Calculations in This Report

Question 1

• Pneumonia rates standardized: This question concerned pneumonia rates (cumulative proportional incidence) in patients in dysphagia programs compared with historical controls. Pneumonia rates were usually reported as raw numbers of cases observed in a certain number of patients. For comparison purposes, these fractional rates (all having different denominators) were all converted by ECRI to percentages. One or more percentages were calculated and tabled for 10 studies (Evidence Tables 24-29).
• Mean pneumonia rates for multi-interval dysphagia programs: Two studies reported pneumonia rates for dysphagia programs after more than just one followup period. For descriptive purposes, rates for these years were averaged for each program.
• Pneumonia rates adjusted to standardized time interval: Each study collected data for a different period of followup [usually hospital length of stay (LOS)]; some researchers reported on different followup periods for different groups of patients within a single study. We adjusted all of these pneumonia rates to a common interval following stroke as described in Appendix D. Briefly, we found two studies that reported chest infection rates out to 30 days following stroke. We fitted a curve to these data using SPSS statistical software and used this curve to interpolate or extrapolate the proportion of chest infection cases for any chosen interval of followup after stroke. (We assumed the proportion of pneumonia cases that accumulated over a given interval would be similar for all the studies, although the proportion of patients acquiring pneumonia would be unique for each study.) We then used the pneumonia rate reported in each study, as the proportion given by our fitted curve, to calculate the expected rate (called the adjusted pneumonia incidence) over an arbitrary standardized interval of 2 weeks, chosen by us for its proximity to all of the reported intervals. This time adjustment was calculated for seven historical control pneumonia rates and three dysphagia program rates.
• 95 Percent Confidence intervals (CIs): None of the studies reported confidence intervals for their pneumonia rates. Therefore, we calculated and tabled the upper and lower 95 percent confidence limits for each rate, both time-adjusted and unadjusted. We also calculated the 95 percent CI for the within-study or between-study means calculated by ECRI. In all, we calculated upper and lower 95% confidence limits for 30 rates (60 calculations).
• Exploratory meta-analysis of historical control pneumonia rate for acute stroke: Some of the dysphagia program studies were case series. As such, they had no control group. Therefore, to estimate the pneumonia rate following stroke in the absence of a dysphagia management program, we carried out an exploratory analysis that involved pooling the results of the four historical control studies reported in the published literature. We calculated a range, an unweighted between-study mean, and a pooled N-weighted between-study mean; all three measures were presented as both time-adjusted and unadjusted.
• Effect sizes for individual studies (absolute difference between historical controls and dysphagia program pneumonia rates): Two of the four studies reporting pneumonia rates in dysphagia management programs (one for acute care of stroke patients and one for dysphagia patients in nursing homes) included within-study historical controls. However, neither of these studies calculated and reported an effect size. We calculated effect sizes using the difference between each dysphagia management program pneumonia rate and the within-study historical-control pneumonia rate. For the two studies with no within-study controls, we used the historical control pneumonia rate from our above-mentioned meta-analysis to calculate an effect size. In addition, for consistency, the one acute-care stroke study with a within-study control group was also contrasted with this out-of-study historical control from our meta-analysis, and an effect size was calculated for comparison to the effect sizes calculated for the studies without in-study controls. Altogether, five single-study effect sizes were calculated, both time-adjusted and unadjusted (10 calculations).
• Statistical significance tests: To test the statistical significance of the above effect sizes, we calculated the upper and lower 95 percent CIs around the above five effect sizes (the absolute differences), both time-adjusted and unadjusted. Any interval that did not include zero was reported by us as a statistically significant effect size (with alpha = 0.05).
• Meta-analysis of acute stroke dysphagia program studies' mean effect size: The three acute stroke dysphagia program studies were pooled and the range, unweighted between-study mean and N-weighted between-study mean pneumonia rate was calculated, both time-adjusted and unadjusted. We calculated the upper and lower 95 percent confidence limits for all of these means. These weighted and unweighted means were contrasted with the weighted and unweighted means from the above historical control meta-analysis, and the effect sizes were calculated by us as the absolute difference for each set of means (weighted and unweighted), both time-adjusted and unadjusted. The statistical significance for these four effect sizes was tested by calculating the 95 percent confidence limits for each difference and assessing whether each confidence interval included zero.
• Proportional reduction of risk calculated as a secondary effect size: For small proportions such as the pneumonia rates considered above (mean historical rates, 6.7 to 13 percent; dysphagia management program rates, 0 to 2.8 percent), substantial reductions of risk can appear misleadingly small when only the absolute difference is reported (6.4 to 9.2 percent in these studies). To balance this, we also calculated the proportional reduction in risk (74 to 100 percent in these studies) as a secondary effect size for each individual study and for all weighted or unweighted between-study means, both time-adjusted and unadjusted if appropriate (20 calculations).
  Calculations in Evidence Tables: To facilitate identification of the above-described original calculations in evidence tables, we provide the following list:
• Evidence Table 24: All fractions converted to percents for pneumonia rates.
• Evidence Table 25: All fractions converted to percents for pneumonia rates, mean calculated for 2 years, adjusted pneumonia rates to 1-week interval for pre-program year, years 1 and 2 and mean for 2 years, calculate 95 percent CI for pre-program year, years 1 and 2, and mean of 2 years, all with and without time adjustment (8 intervals X 2 limits each = 16 calculations), calculate differences between pre-program year and years 1 and 2 and mean, with and without time adjustment, calculate 95 percent CI around above differences (6 intervals X 2 limits = 12 calculations), assess statistical significance (6 intervals), calculate proportional reduction of risk for years 1 and 2 and mean, with and without time adjustment (6 calculations).
• Evidence Table 26: Calculate pneumonia rates, convert all fractions to percents, all rates adjusted to 2-week interval, 95 percent CI calculated for each rate, with and without time adjustment (6 X 2 X 2 = 24), calculated between-studies range, unweighted mean, and N-weighted mean, with and without time adjustment (6 calculations), 95 percent CI calculated for weighted mean, with and without time adjustment (4 calculations).
• Evidence Table 27: For pneumonia rates, convert all fractions to percents, time-adjusted all rates (three calculations), 95 percent CI calculated, with and without time adjustment (3 X 2 X 2 limits = 12 calculations), calculate difference from historical control pneumonia rate, with and without time adjustment (6 calculations), calculate 95 percent CI around above differences (3 X 2 X 2 limits = 12 calculations), statistical significance assessed (6 calculations), proportional reduction of risk calculated, with and without time adjustment (3 X 2 = 6 calculations).
• Evidence Table 28: Calculate pneumonia rate and 95 percent confidence intervals.
• Evidence Table 29: For pneumonia rates convert all fractions to percents, compute 95 percent CI calculated for pre-program year and program year, calculate difference between pre-program and program year, calculate 95 percent CI around above difference, assessment of statistical significance, compute proportional reduction in risk.

Question 2

• Sensitivity and specificity of clinical signs and symptoms: To examine the ability of the bedside examination (BSE) and its subtests to detect aspiration, we abstracted data from six different trials that described 24 subtests. For each of these data points, we calculated sensitivity and specificity (if it was not calculated already) or we verified the authors' calculations. The points were then plotted on a receiver operating characteristic (ROC) graph. For each of the nine different subtests, we examined the available data to determine whether it was appropriate to combine data from different trials in a meta-analysis. In most of these cases, there were too few data for a valid meta-analysis to be performed. In the others, design of the original studies varied too much to make them combinable.
• Meta-analysis of the sensitivity and specificity of the BSE: Six trials measured the ability of a complete BSE to predict aspiration (with barium swallow results used as the gold standard). After evaluating these trials, we determined that they could be combined in a meta-analysis. ECRI's standard method for meta-analysis of diagnostic tests is based on the logit regression method of Littenberg and Moses (1993) and is portrayed as a summary ROC curve. Following the method of Littenberg and Moses, we applied a correction term to one data point with reported sensitivity of zero; the zero value would cause the logit to be invalid. The details of this meta-analytic method are provided in Appendix E but, briefly, this meta-analysis was performed using SPSS-based meta-analysis routines developed at ECRI. Regression results were entered into a spreadsheet for calculation of the summary ROC curves. The spreadsheet calculated and plotted the summary ROC curve, the 95 percent confidence interval on the summary ROC curve, and the sensitivity and specificity values given by the curve at the mean, maximum, and minimum thresholds observed in the clinical trials. We also examined available data comparing the BSE to the barium swallow in terms of pneumonia prediction and found too few data to permit meta-analysis.
• Meta-analysis of the sensitivity and specificity of the 3-ounce water test. The summary ROC meta-analysis was repeated for the four clinical trials measuring the ability of the three-ounce water test to predict aspiration.
• Calculations in evidence tables All of the calculations of sensitivity, specificity, PPV, and NPV shown in Evidence Tables 33 through 38 are ECRI-performed.

Question 3

The original calculations in this section appear primarily in evidence tables. For this reason, these calculations are listed by table.

• Evidence Tables 41 and 42: Original calculations include the sensitivities, specificities, PPVs, and NPVs, as well as many of the percentages shown in these tables.
• Evidence Table 43: For pneumonia rates, converted all fractions to percents, calculated 95 percent CI, calculated difference, calculated 95 percent CI around difference, assessed statistical significance, and computed proportional reduction in risk.
• Evidence Table 44: Calculated test characteristics (sensitivity, specificity, PPV, NPV) for modified barium swallow (MBS), fiberoptic endoscopic examination of swallowing (FEES), and MBS with FEES.
• Evidence Table 45: Calculated positive agreement, negative agreement, MBS positive and FEES negative, FEES positive and MBS negative for four studies.

Question 4

• Performed statistical power calculations.

Synthesis of Results Common to the Four Questions

• Evidence Table 69: For the first four studies in this table, the percents, differences, confidence intervals, and reduction in relative risk were calculated as described under Question 1 above. For the remaining four studies these same calculations were carried out and statistical significance of the differences was assessed. In addition, Fisher's exact test was used to compute a p-value for each of the eight studies and statistical significance of these p-values was assessed.
• Illustrative meta-analysis: For eight studies, conducted meta-analysis by sum of Fisher p-values, conducted meta-analysis by z-scores, conducted statistical test of heterogeneity.

Future Research

• As part of proposed trial, provide statistical power analysis.
• Calculate number of subjects needed in proposed trial to perform statistical analysis.

Supplemental Analysis

Our supplemental analysis consists primarily of original calculations, and the reader is referred to it for further detail. Briefly, however, we performed a complete sensitivity analysis on the incremental cost-effectiveness calculation. This entailed selecting the variables and ranges to evaluate; a total of nine variables were analyzed. Sensitivity analyses were performed using the DATA decision tree software with which the trees were created (version 3.0.18, TreeÅge Software, Williamstown MA). Incremental cost and incremental pneumonias prevented were calculated with DATA, and the incremental cost-effectiveness was calculated with an Excel spreadsheet. The sensitivity analysis was repeated for barium swallow and for FEES.

Six different two-way sensitivity analyses were performed. Three pairs of variables were analyzed for both MBS and FEES. Two-way analysis of incremental cost and incremental pneumonias prevented was carried out with DATA, and the results of each analysis were read into an Excel spreadsheet developed by ECRI for this purpose. The spreadsheet validates the input data, calculates incremental cost-effectiveness for each of 121 points (11 points, 10 intervals for each variable), and provides output in the form of a table and a three-dimensional graph.