Stages 1 and 2: identifying relevant literature and eliciting expert opinion

Stage 3: Delphi study

A multiround Delphi study was conducted with stakeholders known to have an interest in the design of RCTs. Participants were asked about what guidance was needed on specifying the target difference in a RCT sample size calculation. A 2-day consensus meeting and a one-off stakeholder engagement session was embedded within the Delphi study (stage 4; see Stage 4: 2-day consensus meeting and one-off stakeholder engagement sessions for details). Findings from the first Delphi round were considered by the 2-day consensus meeting to aid construction of a draft DELTA² advice and recommendations document. A second-round questionnaire was sent with a hyperlink to the draft document. Views and comments on the draft document overall, the main body of the document, case studies, appendices and references were requested. Rounds 1 and 2 questionnaires are available on request from the corresponding author. A group of known methods experts, the inclusion of which was informed by the DELTA review and findings from stage 1, alongside representatives of key trial groups, were invited to participate in the Delphi study. Representatives for groups, including the UK Clinical Research Collaboration network of CTUs, the MRC Hubs for Trials Methodology Research (HTMR), NIHR/MRC/CRUK funding programme panels, the NIHR statistics group and the NIHR RDS were contacted, using publicly available contact information, and invited to participate.

Participants comprised one named individual per group (unit, board, MRC HTMR, RDS centre or programme, e.g. the director, chairperson or senior methodologist). These groups represent UK centres and networks of excellence that undertake high-quality trials research. As of 1 July 2016, there were 48 (fully or provisionally) registered CTUs, five MRC HTMR and the 10 regions in the NIHR RDS in England and the Research Design and Conduct Service in Wales. Based on the premise that a minimum of 30 participants would be required to participate in the Delphi process, and assuming one-third of invitees would agree to participate, it was felt that at least 90 invitations needed to be made. Owing to the arbitrary nature of this target, no strict maximum was applied and 162 invitations were made.

Stage 4: 2-day consensus meeting and one-off stakeholder engagement sessions

Stage 5: finalisation of advice and recommendations documentation

The provisional advice and recommendations were drafted on completion of stages 1–4 and circulated among the DELTA² members and Delphi participants for comments. UK funder representatives will be asked to assess the advice and recommendations to ensure that the document meets funding panel requirements and allow implementation of changes required for specific forms of publication.

Stage 1: systematic literature search

The search identified 1395 potentially relevant reports (Figure 2). Following the screening of titles and abstracts, 73 publications were full-text assessed. Of these, 28 were included in the review as representing a development of one of the previously identified seven broad method types (Table 4 and see Appendix 2). Minor developments were identified for the health economic (including cost–utility and value of information), opinion-seeking, pilot/preliminary study and SES approaches. No new methods were identified. Most developments (17 articles) related to the use of variants of the value of information approach. A number of helpful review articles that summarise different methods and variations in application were identified; these covered willingness to pay^66,67 and value of information^167,168 health economic-based approaches, and estimation of the smallest worthwhile difference formulation of a MCID, which covered anchor, distribution, opinion-seeking and SES methods.¹⁶⁹ Identified articles on relevant topics (e.g. that address statistical aspects of sample size calculations or an existing method, but contain no new development) were considered as potential references in this document, irrespective of whether or not they were included in this review.

FIGURE 2

Flow diagram.

TABLE 4

Included studies from literature review of methodological development in methods for specifying a target difference

Stage 2: search for guidance

A search for guidance documentation on the websites for the 15 trial-funding and advisory bodies listed within Search for guidance was performed (see also Appendix 2). On the majority of websites, trial design guidance emphasised the need for applicants to provide sufficient detail to justify the chosen sample size, often going on to discuss techniques employed to calculate sample size, but without providing any details or guidance on how this should be done. In particular, there was little specific guidance provided to assist researchers in specifying the target difference. The use of pilot/preliminary studies and ‘interim data’ was noted with limited further comment.

Stage 3: Delphi study

Invitations to participate in the Delphi study were sent (by e-mail on 29 July 2016) to 58 methods experts, along with 104 named representatives of key trial groups (including the UK Clinical Research Collaboration network of CTUs, the MRC HTMR, NIHR/MRC/CRUK funding programme panels, the NIHR statistics group and the NIHR RDS). Of the 162 individuals invited to participate, responses were received from 84 (52%), of whom 78 (48%) accepted the invitation and six formally declined to participate. Acceptance of the invitation was allowed up to 10 October 2016 (the last acceptance was received on 4 October 2016).

The round 1 questionnaire was open for completion between 11 August and 10 October 2016. Of the 78 experts and representatives who agreed to participate, 69 (88%) completed the round 1 questionnaire once invited by e-mail, whereas nine did not complete it. The demographics of those who ultimately participated in the Delphi study are given in Table 5. Participants represented a range of RCT roles, with design, analysis and evaluating funding proposals well represented. The majority of participants (57 of the 69 who completed round 1; 83%) were primarily affiliated with an academic institution and the majority of participants were from the UK (55 of the 69 who completed round 1; 80%). Views on whether or not specific topics and alternative designs (i.e. not a ‘standard’ two-arm, parallel-group design) should be covered within the output document are given in Figures 3 and 4. Delphi participants showed strongest support (≥ 25%) for extensive coverage on alternative research questions and handling multiple primary outcomes. Across most topics there was 50–70% support for proportionate coverage, except for mechanistic studies and public and patient perspectives on the choice of the target difference. Regarding alternative study designs, the strongest support for extensive coverage was for adaptive designs, cluster randomised trials and multiarm trials (all > 25%). Across all designs there was 50–60% support for proportionate coverage.

TABLE 5

Delphi participants’ demographics

FIGURE 3

Round 1 Delphi online questionnaire responses: specific topics to address within target difference recommendations.

FIGURE 4

Round 1 Delphi online questionnaire responses: alternative trial designs to address within target difference advice and recommendations.

A total of 56 free-text comments were made, covering personal views on specific topics, views on the framing of research questions, and the audience that should be targeted for the advice and recommendations. Comments also included suggestions for additional trial designs to cover references and case study topics. The round 2 questionnaire was open for completion between 1 September and 12 November 2017. Only participants who completed round 1 were invited to participate in round 2, in which assessment of draft guidelines was required. Only two rounds were performed to fit with the project timescale and progress. Of the 69 participants invited to participate in round 2, 38 (55%) completed round 2. Findings from the round 2 questionnaire are summarised in Figure 5. Over 80% either ‘somewhat’ or ‘strongly’ agreed that the document was useful overall for the recommendations, case studies and appendices; 21 suggestions for improving the main text were made (11 regarding the case studies and nine on the appendices). In round 2, 62 free-text comments were provided, which, again, covered a range of suggestions for improving the main text, adding an executive summary, improving the signposting of sections, incorporating views on the case studies and appendices, additional references, raising the issues of estimands and personal views on various topics. Comments made in round 1 and 2 questionnaires, along with feedback from stage 4, led to a substantial number of changes to the document prior to its finalisation. The most substantive being incorporating an executive summary and increasing the number of case studies.

FIGURE 5

Round 2 Delphi online questionnaire responses.

Stage 4: 2-day meeting and stakeholder engagement

An engagement session was held at the SCT in May 2016, when the project was introduced, and views on the scope and broad content of the output document were invited through audience participation. Following this, a 2-day workshop was held in Oxford on 27 and 28 September 2016, which involved 25 participants, including CTU directors, study investigators, project funder representatives, funding panel members, researchers with experts in sample size methods, senior trial statisticians and PPI representatives. The workshop included presentations of the findings from the initial two stages of the project, the SCT engagement session and round 1 of the Delphi study, and focused on decisions relating to the scope and content of the output document. An initial structure for the first draft of the document was developed in the light of the findings from the round 1 questionnaire available at the time of the meeting.

A revised structure was agreed by participants at the workshop. Drafting of individual sections was allocated to individuals. The recommendations on conducting a sample size calculation were initially drafted. The various sections were then developed into the first full draft of the document; this was circulated to all of the DELTA² project group for comment, with the draft revised in the light of these. An iterative process of comments and revisions was followed until the final version was agreed.

Subsequently, two further engagement sessions were held at PSI and JSM conferences. At the time of the session, the most current draft of the document was made available to participants. Both within- and post-meeting feedback highlighted the need to consider the role of estimands and the minimum (statistically) detectable difference in the sample size calculation, leading to revisions in the document. There was broad consensus, although not universal agreement, on the need for advice and recommendations and the main topics it needed to cover from stakeholders across the various meetings and from the Delphi study. Differences of opinion tended to be about which topics needed to be covered and how important it was that they were covered.

Stage 5: finalisation, adaptation and dissemination

The draft advice and recommendations were reviewed by the representatives of the project’s funders (MRC NIHR Methodology Advisory Group) on 2 October 2017. A number of revisions were made in the light of feedback received from the advisory group and further feedback from the authors. The revised text of the main documentation was finalised on 28 February 2018. It was endorsed by the MRC NIHR Methodology Advisory Group on 12 March 2018, with minor updating of references, and the final version was produced on 18 April 2018. Engagement with individual funders and funding programmes about the best way to utilise the document and adapt to their needs is ongoing. Journals could refer to this document and the reporting recommendations in the guidance for authors. Potentially, the CONSORT statement could be extended to incorporate more specific information on the sample size calculation, as detailed in the DELTA² recommendations for reporting.

Overview

The target difference is arguably the key value in a conventional sample size calculation, but also the most difficult to choose. The DELTA² project sought to produce more detailed advice and recommendations for researchers and funder representatives, to aid researchers in making this choice and funder representatives in assessing the choice made. Building on the DELTA guidance,^14,20 a number of aspects were explored through engagement with stakeholders and the findings are summarised in this paper.

Decisions on scope and content

As part of the process, we explored uncertainty about what methods for sample size determination should be covered. In particular, the views on two methods (value of information and SES-based approaches), which were included in the DELTA guidance,^14,20 were debated and the inclusion reconsidered. There was general agreement that they should be included again, but, in particular, the distinctive nature of the value of information approach required greater prominence. The need for some consideration of alternative statistical approaches (aside from the specification of the target difference per se) was also relatively strong. This resulted in specific appendices and boxes within the main text covering more common alternative statistical methods and trial designs, and dealing with related aspects such as compliance analyses and missing data.

The need for more practical advice and recommendations was raised multiple times in various responses and in the engagement sessions. This led to two main additions in the final document. First, 10 recommendations were made for specifying the target difference and a list of corresponding reporting items was included for instances when the conventional sample size approach is used. It is hoped that this will go some way to support researchers and funders undertaking and assessing sample size calculations. It is recognised that future adaptation to accommodate other study designs and statistical approaches will be needed. Second, a number of case studies were included, reflecting different trial designs and covering different conditions. Additional case studies could be added over time to provide a more complete coverage of the range of trial designs, statistical approaches and methods for specifying the target difference.

Overall, the DELTA² advice and recommendations are more comprehensive than the original DELTA guidance (and also more detailed, with over 25,000 words, compared with around 4000 words). It covers a much broader range of trials and approaches, with more practical advice and recommendations about how to undertake a sample size calculation for a RCT. A number of areas for further research were identified. Addressing these evidence gaps would help inform advice and recommendations for less common statistical approaches and trial designs.

Strength and limitations

The main strength of this advice and recommendation lies in the extensive preparatory work undertaken in both the DELTA² project and also the original DELTA work. The multiple avenues for engagement with stakeholders represent another strength, as this provided opportunities to solicit views on relevant topics and feedback on the draft document from various stakeholders. A variety of methods were used to inform the development of the document, including systematic reviews of the literature, a Delphi study using online questionnaires, engagement sessions with stakeholder groups and a 2-day workshop.

Participants in the various stages of the project were self-selected and may not be fully representative of all stakeholders. In particular, despite several attempts, there was limited involvement of industry statisticians, with the exception of the PSI stakeholder meeting, and participants were mostly academic statisticians. Overall, those involved were possibly more methodologically interested than those who did not engage. Timings of key meetings meant that flexibility was needed in the conduct of the stages and they were not carried out in a sequential manner, as originally envisaged.

The Delphi study had only 69 participants and had only two rounds, with a substantial drop off between rounds 1 and 2. Unlike other implementations of a Delphi study, a scoring system was not used to rank topics,¹⁷⁰ nor was a formalised definition of consensus¹⁷¹ used, as reflected in the more informal determination of consensus in this application.

The scope of some of the stages was purposely limited due to time and resource constraints. The journals searched for methodological developments were those thought to be most likely to publish new developments. It is possible that other developments have been published in other journals, which would have potentially been missed. Consulted stakeholders were predominantly based in the UK and the engagement sessions were limited in number and dependent on acceptance of the proposal at the respective stakeholder meetings.