Introduction

Several terms may usefully be defined. By commissioned research, we mean research where the topics to be researched are defined by the programme and not by the researchers who do the work. This implies that the programme is acting on behalf of the NHS and must have mechanisms for ‘knowing’ the needs of the NHS. This differs from both response-mode research (the traditional mode, with funders taking bids from expert teams of researchers) and researcher-led research (HTA’s term for the work stream introduced in 2006 where proposals are submitted by researchers but are rigorously assessed against NHS need).¹

To be relevant to decision-making in the NHS, any clinical trials would need to be pragmatic as opposed to explanatory. Pragmatic trials have been defined as those with broad inclusion criteria, carried out in many centres and with patient-relevant outcomes.³²

To employ a term given prominence in the Cooksey Report (2006),³³ NHS-funded research had to be restricted to public interest or market failure research, that is, work that the private sector would not be interested in carrying out. This is often due to the inability to patent that which is being tested (difficult outside new drugs or, in particular, interventions made up of services rather than tightly defined products).

As the HTA programme is a commissioned programme, one might expect it to prioritise research focused on the needs of the NHS. A substantial literature discusses methods for research prioritisation but there is much less on how potential topics should be identified, or on assessments of whether or not prioritised research is indeed ‘needs-led’.^9,34–37

Since its inception, the NHS research and development (R&D) has focused on identifying gaps in research relevant to the NHS and prioritising them. Setting priorities is difficult and complex, partly because there is ‘no agreed upon definition for successful priority setting, so . . . no way of knowing if an organisation achieves it’.³⁸

Different methods have been suggested, such as multidisciplinary involvement, public and patient involvement, the use of scoring systems, the Delphi process and information specialist involvement. Economic impact approaches include the payback approach or expected value of information models. Priority setting means an allocation of limited resources, which can be highly political and controversial. Developing a structured topic prioritisation process helps address this challenge.

Chase et al.⁹ described the different sources used by the HTA programme in 1998 to identify potential priorities. Overall, there were 1100 suggestions for the programme from four main sources: (1) a widespread consultation of health-care commissioners, providers and patients; (2) research recommendations from systematic reviews; (3) reconsideration of previous research priorities; and (4) horizon scanning. Nearly half (46%) of final programme priorities were from the widespread consultation, with 20% from systematic reviews and 10% from each of the other two areas. (The remainder came simultaneously from more than one source.) Chase et al.⁹ concluded that there was value in having a mix of sources. One of the aims of this chapter was to apply the approach of Chase et al.⁹ to all the RCTs published to mid-2011.

A small literature discussed the patient relevance of outcomes in publications, through surveys of trials published in a particular disease area. There are three notable examples:

• Gandhi et al.³⁹ looked at diabetes trials and found that primary outcomes were patient important in only 78 of 436 RCTs (18%).
• Montori et al.⁴⁰ also looked at diabetes trials and found that primary outcomes were patient important in only 42 of 199 RCTs (21%).
• Rahimi et al.⁴¹ looked at cardiovascular trials and found that primary outcomes were solely patient important (death, morbidity or patient-reported outcomes) in only 93 of 413 trials (23%).

Chalmers and Glasziou⁴² proposed a framework for considering avoidable waste in research, with four stages. The first concerned whether or not the questions addressed by research are relevant to clinicians and patients; if they are not, Chalmers and Glasziou⁴² argue that the research is wasted.

Although Chalmers and Glasziou⁴² give some examples of the ways in which research fails to address relevant questions, they provide no quantifiable measures of waste in this stage of the framework, unlike the other three stages (design, publication and useable report), for each of which empirical estimates of waste are provided.

The extent to which RCTs have been preceded by systematic reviews can indicate the source of the topic. A recent review of 48 trials funded by the HTA programme between 2006 and 2008 indicated that 80% had been preceded by a systematic review.⁴³

Questions addressed

The questions on which data were extracted are shown in Box 2.

BOX 2

The research questions answered under this theme T1.1. Type of commissioning work stream?

Methods

Nine questions were piloted in theme 1 (hereafter T1). One question (‘How was the relevance to the NHS assessed?’) was deemed not feasible owing to lack of data. However, data were available on the work stream (commissioned or researcher led) (T1.1), whether or not a prior systematic review existed (T1.2) and the source of the topic (T1.3). These are explored below.

Results

Question T1.3: the source for topic identification

Of the four main sources of identification, ‘widespread consultation’ contributed 64 (66.7%) topics followed by systematic reviews (25%, 24/100) and the Horizon Scanning Centre (3%, 3/100).

The balance of these sources shifted over time. When the number of trials increased in 2001–2, the proportion of topics from systematic reviews rose to 65% (Table 5).

TABLE 5

Source of commissioned topic by year

Question T1.4: type of Health Technology Assessment advisory panel

The source of topics varied by advisory panel (Table 6). Widespread consultation was the main commissioning source for two of the three panels (83.3%, 15/18 and 72.2%, 39/54, respectively). The exception was the pharmaceutical panel, where 50% (12/24) of the commissioned topics were from systematic reviews.

TABLE 6

Number of topic suggestions by source of information and advisory panel

Question T1.5: what was the priority given by the programme to the research?

The programme prioritised 70% of projects in the top band. Of the 71 projects prioritised up to and including 1999, 50 (70.4%) were classified as A-list topics (‘recommended for commissioning – must commission’) and 18 (25.4%) were B-list topics (‘recommended for commissioning’ only). The HTA MIS database did not provide sufficient information for 4.2% of trials (3/71) (Table 7).

TABLE 7

Summary data on the priority status of the research topic up to 1999

Question T1.8: adequate reporting of the proposed and published primary outcome

All projects were analysed to compare the proposed primary outcomes with those published (n = 109). We were able to classify the proposed primary outcome for 97 projects (88.9%) and the published primary outcome for 108 projects (99.1%) (Table 8); little changed between these two stages. Patient-important outcomes were reported in more than half of the HTA-commissioned projects, at both the proposed and published stages of the project (67%, 73/109 and 73.4%, 80/109, respectively). A number of outcomes could not be classified using the Gandhi et al.³⁹ three main headings. Fourteen proposed primary outcomes and 18 published primary outcomes were categorised as ‘other’.

TABLE 8

The commissioned, planned and actual primary outcomes

Thirteen projects (11.9%) had differences between the planned and actual type of primary outcome. These discrepancies were mainly due to a lack of information or having no information on the primary outcome in the planned documentation (proposal/protocol) (n = 12). The monograph was able to provide sufficient information for 10 of these projects to enable the primary outcome to be classified accordingly. Table 9 highlights where these discrepancies were between the planned and actual reporting of the primary outcome.

TABLE 9

Discrepancies between the planned and actual primary outcome measure

When diagnostic and screening projects (n = 20) were excluded, patient-important outcomes increased from 67% (n = 73) to 73% (65/89).

Over the period 1993–2002, 82.7% (67/81) of reported primary outcomes were patient important (Table 10). The years 1993–2002 provide a more accurate report of the type of primary outcome reported in the monograph, as a number of projects funded during the period 2003–10 have not yet published.

TABLE 10

Actual primary outcome as reported in the monograph by year of the topic advertisement (excluding diagnostic and screening projects)

Question T1.9: what was the time lag between prioritisation and publication of the monograph?

This question asked about the time taken between the programme prioritising a topic and the monograph publishing the results. The interval was 8–10 years (Table 11). The average was 8 years for trials prioritised in 1993 and 9 years for those prioritised in 1999.

TABLE 11

The year the topic received its project application reference by the year in which the monograph was published

Analysis

In Chase et al.’s⁹ review, 46% of programme priorities in 1998 came from the widespread consultation and 20% from systematic reviews. Our data show greater reliance on consultation but with variation from year to year. The key question concerns what can be inferred about the importance of the HTA projects to the NHS. It would be a mistake to equate widespread consultation with NHS relevance and systematic reviews with academic interest. There is no reason why this should be so. The processes the HTA programme had in place between the identification of topics and their advertisement as commissioning briefs means that the initial topic only served as a starter for the real work on NHS relevance.

Unsurprisingly, most projects that were funded had been prioritised; 70% had been given the top band (A) by the programme’s prioritisation processes. Band A, ‘recommended for commissioning – must commission’, meant that the programme would ‘go the extra mile’ to ensure that research was funded in that area. What to make of this 70% figure? The priority banding was the end of a process that started with the source of the topic, addressed in the previous question. This process involved detailed consideration of potential research priorities by panels of NHS experts (patients, clinicians, managers) as well as an overarching standing group on health technologies, meeting annually for 2 days. The priority band was a summary score produced by the whole process. The process was producing research proposals of which 70% were thought to be of high relevance to the NHS and so of a high priority.

By contrast with the finding by Jones et al.⁴³ that 80% of RCTs funded by the HTA programme and published between 2006 and 2008 were preceded by a systematic review, we found that 56% of all trial published to 2011 were preceded by a systematic review.

The finding that the statement of need did not change between the commissioning brief and that reported in the monograph in 101 out of 107 trials (94%) suggests no evidence of ‘drift’. Unfortunately, the data available in the database were not detailed enough to allow us to make further informative assessments in this area.

Primary outcomes tended to be patient relevant. Excluding those projects relating to ‘diagnostic technologies and screening’ increased these figures from 67% to 73%, much higher than previous studies (18% in Gandhi et al.,³⁹ 21% in Montori et al.⁴⁰ and 23% in Rahimi et al.⁴¹).

The lag between the programme prioritising a topic and publishing the results in the monograph series was 8–10 years. As this measures the time to publication in the HTA journal and not to publication in any journal, it overestimates to some extent. The key question is choice of benchmark: what is the right length of time against which 8–10 years should be compared?

Discussion

Question T1.9 on the 8- to 10-year time lag from topic identification to monograph publication was striking. However, we were unable to find any comparable estimate in the literature.

Although the answers to most questions were largely as expected, these questions only relate to meeting NHS need in an oblique and indirect way. Data availability limited the questions that could be asked regarding the core aim of the HTA programme, that is, how well the research it funds aims to meet the needs of the NHS. This is something that the programme should consider how best to address.