The statistical analysis plan and potential results scenarios

All clinical trials should have a statistical analysis plan. This gives an a priori opportunity to review potential scenarios for the results of the trial, what the results might mean and how this is translated into clinical interpretation. This process is especially important for placebo-controlled trials. The complexity of a multigroup design, perhaps with a no-treatment arm, brings about many possible interpretation permutations (see Chapter 3, The physiological aspects of placebo). The surgery can be shown to have more benefit than placebo or no treatment, or maybe no added benefit. A hierarchy of benefit may be proposed perhaps with surgery being more beneficial than both placebo and no treatment.

One of the advantages of outlining the various possible results scenarios is to prepare for any clinical conclusions and potential obstructions to change of practice (outlined below). Early consideration of difficult commissioning decisions can be shared upfront with the surgical community conducting the trial if the options are reported. Sudden realisation from surgical personnel that they have been involved in a study that questions the value of an established procedure can be alarming and have negative influences on any change of practice and acceptance of that change.

A further stage for the future might be to fully explore what the trial results might mean with all parties before the trial has even started. The discussion around whether or not a placebo treatment still has sufficient benefit or merit to be commissioned or provided as a treatment in itself could also be had at this time.

Translation into change of policy and practice

Many of the placebo-controlled trials of surgery reported to date have shown no benefit of the definitive procedure over the placebo-controlled intervention. The design is popular and used frequently to explore treatments with suspicious efficacy and effectiveness. Bearing in mind the ethics and academic justifications required (see Chapters 6 and 7) for a surgical placebo control, reasonable preliminary evidence is, therefore, required to show that part or all the treatment effect of the surgical procedure under investigation might be due to the placebo effect. As previously reported, investigation of a treatment that has no such placebo component, or even a general belief that surgery has no placebo effect at all (Dr Teemu V Karjalainen, Central Hospital, Finland, 2020, personal communication), would not require a placebo-controlled evaluation.

With this backdrop and focus on established, if questionable, procedures, investigators responsible for undertaking such trials must anticipate that any ‘no difference’ results of the trial will be disruptive to accepted clinical care pathways and guidelines. Investigators should also expect, and be prepared for, resistance from clinicians and patients whose beliefs and convictions are being challenged by the findings. Such trials will also generate interest from payers (state and insurance based), press and media.

Once change is indicated, there can be a long lag between research findings and change in practice, as exampled by trials of knee arthroscopy. In the case of knee arthroscopy for osteoarthritis, although the original publication was in 2002,⁷² it took 15 years for the findings to be partially adopted, despite several other high-quality studies replicating the findings. Similar resistance from the clinical community has been encountered with trials of vertebroplasty for osteoporosis⁵⁰ and subacromial decompression for shoulder pain.¹³ There are consistent features of the resistance and these include a belief by the surgical community that the patients in the trial do not represent the usual population undergoing the procedure. It is also suggested that the surgeons involved in the trial may not be sufficiently expert in the procedure. In other words, a feeling that the trial results ‘do not apply to me and my practice’ is commonplace. This sentiment is highlighted by the expressed views of 15 combined surgical associations of one European country. These associations have advised that, contrary to previous reports, the CSAW trial does not provide any new insights and there are no consequences from the CSAW study for this country’s health system. The response in the UK was starkly different with NHS England moving to de-implement subacromial decompression surgery by placing it on a list of ‘ineffective’ treatments.¹⁷³

The question then becomes ‘How can the challenge of effective and timely change of policy and practice be improved?’. There are ways to facilitate, and ideally consideration of any impact should be included in the design phase and conduct of the trial. Most importantly, this should include key leaders in the patient groups, professional associations and clinical communities involved in delivering the investigated treatment. If the results are likely to have global impact then consideration should be given to involving international investigators. There should be ‘buy-in’ from patients and professionals. As soon as results are known, further discussions and the production of joint statements are necessary. If the implications are that the procedure will likely be performed less frequently then advice for patients about alternative treatment is essential.

The policy-maker’s perspective

Policy-makers consist of two broad groups: (1) those who issue guidance about how health-care interventions should be used and (2) those who commission and pay for services. In most health systems, those who make decisions about service provision attempt to maximise the health returns obtained for the investment [e.g. by maximising the quality-adjusted life-year (QALY) output of any health services provided]. Evidence of value from studies employing a placebo control and the value of any placebo effect itself may be viewed differently by each side.

Payers (e.g. commissioners in the English NHS and insurance companies in many European countries) tend to value a QALY gain, regardless of its origin. If a policy provides a net health gain for a reasonable price and is acceptable to patients and society, then the mechanism by which that gain occurs may not be considered important. The health-care system is a mechanism for turning money into QALYs, and exactly how that occurs may not be important.

Guideline generators see things differently. Producers of guidelines tend to pursue the understanding of how a health gain is generated. There can be unease when a gain occurs through a non-specific placebo mechanism, rather than the anatomical, physiological and psychological processes that the intervention’s logic model presupposes. For interventions that may have a significant placebo effect, a guideline producer would like to see robust studies that explore that effect and enable them to separate out any placebo benefit. Therefore, the guideline producer tends to value more comparative studies with active control or placebos, whichever is clinically more appropriate.

There are potential impacts of establishing that an intervention has a significant placebo component contributing to the effect. First, there may be downgrading of any recommendation to use the procedure, as it has been shown not to ‘work’. Second, the payer who may have previously willingly paid for the procedure now follows the downgraded recommendation and declines to fund an intervention that may be effective, albeit with a large placebo component (an unanticipated mechanism).

The journal’s perspective

The view expressed by journals may be best conveyed by the editorial accompanying the placebo surgery methods paper in The Lancet:¹⁷⁴

Writing in today’s The Lancet, David Beard and colleagues review the role of placebo controls in surgical trials and present recommendations for their use. Over the past 10 years there has been increasing recognition of the importance of the placebo effect, particularly how strong this effect could be for a surgical procedure that involves high-intensity medical care, strong analgesia, and often physiotherapy. The growing use of placebo-controlled surgical trials to re-examine common surgical procedures that have a biologically convincing mechanism and a long history of use has led to a wave of unexpected results. Of surgical procedures examined with this rigorous method, half were proven to be no more effective than placebo.

The systematic debunking of many well established, definitive operations has become perhaps the biggest story in surgical research this decade. Common procedures such as vertebroplasty and subacromial decompression have been shown to be largely ineffective, but these procedures continue to be in common use. Challenging current practice is difficult in many areas of medicine, particularly where there are potential personal and commercial vested interests, including private practice. Stopping the use of a common but debunked surgical intervention will be especially tough because the alternative is not a newer or better intervention but often a continuation of the patient’s current treatment course. For some of these procedures, where insurers or care commissioners have prevented surgeons from doing them, the discourse has often centred erroneously on rationing. The argument is not really about cost-effectiveness but rather that when robustly assessed, these procedures have been found to not be effective at all and still risk adverse effects.

Beard and colleagues discuss the need to plan for a negative result at the outset of a trial, anticipating the disruption that such a result produces, and creating processes to allow a transition of practice. The responsibility for doing this lies not only with surgical researchers, but also with the wider medical community. It is only by having the tenacious drive to question and critically assess with the most robust studies that we can leave behind ineffective procedures and concentrate on the many areas where surgery cures and heals.

Reprinted from The Lancet, vol. 395, Editorial. Gaining control: placebos in surgery trials, p. 756, Copyright (2020), with permission from Elsevier¹⁷⁴