As discussed in previous sections, empirical evidence clearly guides quality assessment of systematic reviews and RCTs. By contrast, little evidence helps guide the evaluation of observational studies beyond good epidemiologic practice and principles. Comparability of subjects was the only empirically derived element we designated for observational studies, relating to the use of concurrent versus historical controls groups. Chalmers et al. noted that the use of nonrandomized trials with historical controls exaggerated treatment effects in studies of anticoagulation for acute myocardial infarction. ¹⁶⁰ Concato, Shah, and Horowitz compared RCTs and observational studies using concurrent control groups for five clinical topic areas (BCG vaccine for tuberculosis, mammography to prevent breast cancer deaths, cholesterol lowering and the risk of trauma mortality, hypertension treatment, and the risks of both stroke and coronary heart disease). ¹⁶⁹ They found that estimates of effect were similar for RCTs and observational studies when the observational studies were rigorous i.e., using concurrent controls.

Two studies provide empirical evidence of bias in observational studies related to competing interests, which we have termed funding and sponsorship. The Cho and Bero study noted that both RCTs and observational studies reported in symposia proceedings tended to show favorable treatment effects. ¹⁶⁸ In a similar study comparing the publications found in symposia proceedings versus peer-reviewed journals, articles in symposia were more likely to have been supported by the tobacco industry and less likely to have government funding. ¹⁷⁰ Multivariate analysis indicated that peer-review was an important quality criterion rather than source of funding. This study lends support for a quality criterion of peer-review as an empirically based domain.