Limitations of the Review

This review is an update of an earlier comparative effectiveness review. Because of the vast size of the existing literature, for both the earlier review and this review, we have relied in part on previously published systematic reviews and have not conducted new meta-analyses pooling the findings of all existing trials. Therefore, the findings may be less comprehensive than they might be. Further, because we did not conduct new meta-analyses, we cannot account quantitatively for the heterogeneity of the literature.

Publication Bias

Our literature search procedures were extensive and included canvassing experts from academia and industry for studies. However, it is possible that other unpublished trial results exist for the treatments included in our report. Publication bias may occur, resulting in an overestimation of the effects of these treatments. Because we did not conduct new meta-analyses to calculate pooled effect sizes for efficacy, we cannot estimate the actual publication bias in this literature.

Study Quality

An important limitation common to systematic reviews is the quality of the original studies. Recent attempts to assess which elements of study design and execution are related to bias have shown that in many cases, such efforts are not reproducible. Therefore, the current approach is to avoid rejecting studies or using quality criteria to adjust the meta-analysis results. However, we did use as a measure of quality the Jadad scale, which is the only validated set of quality criteria for trials. As there is a lack of empirical evidence regarding other study characteristics and their relationship to bias, we did not attempt to use other criteria. The Jadad scores of the trials newly identified for this report ranged from 0 to 5 (mean, 2.9; median, 3). Thus the quality of included studies is a potential limitation for this report.

Other Potential Sources of Bias

In addition to the possible influence of study quality, we recognize several additional potential sources of bias: the applicability of the studies to the population that would be likely to benefit from the agents of interest and the potential bias inherent in interpreting adverse event data from studies.

We assessed the applicability of the trials included in the report using the method of Gartlehner et al.²⁵ In general, most trials were moderately to highly applicable to the population of persons at risk of osteoporosis (although the proportion of men enrolled in most of the trials is small). The exceptions tended to be smaller trials focused on groups of individuals with a particular disease or condition that increased their risk for osteoporosis; thus the results of these trials would certainly be applicable to those populations.

Any assessment of a broad range of potential adverse effects may be subject to findings due to chance alone. Interpretation of statistically significant differences needs to consider the size of the effect, the consistency of the finding, the possibility of other reasons for the effect, and biological plausibility, among other things.