The protocol (of both the validation and the nested qualitative studies) has been published elsewhere (Kalpakidou et al.).29 Parts of this chapter have been adapted from Kalpakidou et al.29 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.
In this report, we first describe the involvement of patients and the public in our research. We then describe the methods of validation and the qualitative substudy separately.
Patient and public involvement
We regarded the involvement of patients and the public as an important factor in the design, delivery, interpretation and dissemination of our research.
Involvement in research design
As part of the PiPS development study,11,28 we asked for patients’ views about obtaining prognostic information. The majority (478/778; 61%) of patients with capacity to consent to participate in Prognosis in Palliative care Study I (PiPS1) indicated that, if the information were available, they would want to know their prognosis, 176 (23%) indicated that they would prefer not to know and 124 (16%) were ambivalent (they ‘did not know’ or ‘did not care’ about the issue). This reaffirmed to us the importance of ensuring that prognostic information is not ‘thrust upon’ patients who would rather not receive it and underlined the need to include further qualitative work in PiPS2 to understand the best way to present this information to patients and their families.
In preparation for PiPS2, we undertook a more in-depth consultation with seven individual cancer patients and one focus group of users comprising a further eight carers/patients. All users agreed that the subject was an important area for clinical research. We asked users to reflect on the involvement of patients without capacity in the study and in whom we should undertake blood tests. In the light of service user feedback, we decided to include patients without capacity in the research, but to limit their involvement to observational measurements only.
Two service users (PB and DAF) joined the study team early in the course of the research. They contributed to the design of the topic guide and the interview schedule used in the qualitative substudy and ensured that the content was appropriate and comprehensive. Our service user representatives also contributed to the creation of the patient and carer information sheets used in the quantitative and qualitative studies.
Involvement in research delivery
Three service user representatives, recruited via the Marie Curie (London, UK; www.mariecurie.org.uk) ‘Expert Voices’ users’ group, were actively involved throughout the conduct of the study. One service user representative acted as a member of the Study Steering Committee and had an important role in overseeing the study. The other two service user representatives (PB and DAF) were fully involved in the ongoing management and delivery of the study. They participated as research team members in minuted, monthly Study Management Group meetings (usually by teleconference) and contributed to discussions of issues arising during the trial, thereby ensuring the smooth running of the study. They contributed to the interpretation of the qualitative data arising from the interviews with patients and their carers and throughout data analysis. They were presented with emerging themes and anonymised quotes, and their interpretations were included in the ongoing and final narrative.
Dissemination of research findings
Most of the dissemination activities of the grant are yet to be undertaken. Nonetheless, our service user representatives have already coauthored poster presentations at conferences. PB and DAF, co-authors of this report, attended the Annual Supportive and Palliative care conference (21–22 March 2019) in Harrogate, where they were co-authors on a poster presentation relating to patients’ and carers’ perspectives on the use of the PiPS predictor models. The main results relating to validation of PiPS (both quantitative and qualitative) will be delivered as an oral presentation at the European Association of Palliative Care Congress in 2020 (online) and have been written up, but not yet accepted, for publication in a peer-reviewed journal. Preliminary results relating to recruitment processes were presented in poster format at the European Association of Palliative Care congresses in Madrid (2018) and Berlin (2019). Preliminary results relating to the qualitative subsidy were presented in poster format at the European Association of Palliative Care congress in Bern (2018).
Validation study
Design
This was a multicentre, prospective, observational, cohort study to validate various prognostic models in patients with advanced, incurable cancer. The study involved patients from 27 palliative care services across England and Wales and was sponsored by University College London (UCL), London, UK. A full list of participating units and the principal investigators (PIs) at each site is provided in Appendix 3.
Patients were recruited in three palliative care settings:
community palliative care teams (CPCTs) (including day hospice and palliative care outpatients)
hospital palliative care teams (HPCTs)
hospices [including inpatient palliative care units (IPCUs)].
The recruitment period was from August 2016 to the end of April 2018. Three months after recruitment ended, a list of study participants was sent to NHS Digital to determine dates of death. The accuracy of the studied prognostic models was then compared with actual survival.
Following Medical Research Council guidance,30 a subset of patients who agreed to participate in the main study, their next of kin (informal carers), patients who declined to participate in the main study and health-care professionals was recruited into a nested qualitative substudy to explore views about prognostication and the use of prognostic tools (see Methods of nested qualitative substudy, and Chapters 3 and 4).
Participants
The study involved patients with and without the capacity to consent to participate. Many patients at the end of their lives become confused, semiconscious or comatose, or may have pre-existing cognitive impairment; consequently, they frequently lack the capacity to consent to participate in research. The inclusion of patients who lacked capacity to consent was therefore of great importance because the study population should be representative of those patients commonly seen in palliative care services. Capacity to consent was assessed by the attending clinician using Department of Health and Social Care guidance.31 If capacity to consent was in doubt, the clinician carried out a capacity test as provided by the Royal College of General Practitioners’ Mental Capacity Act Toolkit for Adults in England and Wales.32
Inclusion criteria
Locally advanced or metastatic incurable cancer (estimated prognosis of survival < 1 year).
Age ≥ 18 years.
Recent referral to palliative care services. For community, day hospice or palliative care outpatients, ‘recent’ signifies fewer than three previous contacts with the palliative care service before recruitment to the study. For inpatient palliative care patients (including HPCTs), ‘recent’ signifies an appointment with member of the palliative care team no more than 7 days previously.
Sufficient English language skills to read and understand the patient information sheet and undertake study assessments.
Study assessments
Data were collected at a single time point and were usually obtained from a review of medical notes or a discussion with clinical staff. If patients were able to respond to questions, data could be obtained directly from them. The data required for the calculation of each prognostic score are shown in . The case report forms (CRFs) that were used are shown in Appendix 4.
Variables included in each prognostic score
Demographic-, disease- and treatment-related data and capacity recording
Demographic details of enrolled patients, such as age, gender and current location (e.g. home, hospital or hospice) were recorded. NHS number and date of birth were also recorded to be sent to NHS Digital. Information on the nature and site of primary tumour and sites of metastases (if any) were collected and patients’ capacity to consent to participate in the study was documented.
Key symptoms
The presence or absence of key symptoms was recorded: anorexia, dysphagia, dyspnoea, fatigue and weight loss. This was required for the scoring of the prognostic models.
Abbreviated Mental Test Score
The Abbreviated Mental Test Score (AMTS) assesses cognitive function.33 Patients who lacked capacity and so could not consent to participate in the study (including those who were unconscious) were attributed scores of zero.
Clinical assessments
The following clinical assessments were undertaken.
Clinical examination
Researchers recorded the presence or absence of ascites, peripheral oedema, delirium and decreased oral intake. Pulse rate was measured over 1 minute.
Measures of performance status
The Eastern Co-operative Oncology Group (ECOG) performance status34 was recorded. This is a measurement of a patient’s of everyday functioning and is required for the calculation of PiPS-A and PiPS-B.
The Karnofsky Performance Scale (KPS)35 was used to assess patients’ functional impairment. The KPS is required for the calculation of the PaP score.
The PPS16 describes a patient’s current ambulatory level, activity level, extent of disease, self-care abilities and intake and conscious level. It is one of the prognostic models being evaluated.
General health status
General health status was rated using a seven-point observer-rated scale with scores ranging from very poor (1) to excellent (7).
Time to terminal disease
The time to terminal disease (TTD) was estimated by clinicians. TTD is defined as the time that elapsed between the diagnosis and development of incurable disease and is required to calculate the FPN.
Blood tests
For patients with capacity to consent, a fresh blood specimen was taken. For patients without capacity to consent, there was no requirement to take a fresh blood specimen, but if a blood sample was being taken for another reason as a part of routine clinical care within 72 hours of study enrolment, then the relevant tests were requested. Even if no new specimen was being taken from a patient without capacity to consent, if relevant results were available within 72 hours of being enrolled in the study then these results were recorded.
The blood test results required were white blood cell count, lymphocyte count, neutrophil count, platelet count, urea concentration, albumin concentration, alkaline phosphatase concentration, alanine transaminase concentration, CRP concentration and lactate dehydrogenase (LDH) concentration. Blood specimens were processed locally in the routine clinical laboratory using usual local arrangements.
Clinicians’ estimates of survival
The attending clinician and nurse estimated survival of study participants independently. When the estimates agreed, this represented the combined multiprofessional prediction. When they were discordant, the clinician and nurse discussed the case and reached a consensus. To characterise the prognosticators in more detail, participants were asked to provide information about themselves (i.e. age, gender, professional training and years of specialist experience). Clinicians were also asked to provide their prognostic estimates using a number of different formats to facilitate comparison with the outputs of the prognostic scores. Clinicians were asked to (1) provide approximate estimates of length of survival: days (0–13 days), weeks (14–55 days) or months+ (≥ 56 days), (2) provide more specific estimates of survival to the nearest week (from < 1 week to > 12 weeks) and (3) estimate the probability of survival at specific time points (1, 3, 7, 15, 30 and 60 days).
Data management
A study database for the storage, management and analyses of identifiable data was developed via a secure web application named REDCap (Research Electronic Data Capture, UCL, London, UK) and using the UCL Data Safe Haven secure system. This system is certified to the ISO27001 information security standard.
The remainder of the (non-identifiable) study data [i.e. electronic case report forms (eCRFs)] were sent to a separate database that had been created and supported by a company named Sealed Envelope (London, UK). Data from the paper CRFs were manually transferred to the eCRFs by the researchers at each participating site.
Ethics approval
The study received ethics approval from the Yorkshire and the Humber – Leeds East Research Ethics Committee (REC) on 12 April 2016 (reference 16/YH/0132).
Ethics/protocol amendments
During the course of the study six amendments were submitted to and approved by the Health Research Authority (HRA): five non-substantial/minor amendments and one substantial amendment. In line with the relevant regulations, the substantial amendment was also submitted to and approved by the Yorkshire and the Humber – Leeds East REC. The REC was notified of the five non-substantial/minor amendments.
The amendments are listed below in chronological order:
The first non-substantial/minor amendment was submitted on 24 May 2016, involving the addition of new participating sites. This was approved on 21 July 2016.
A second non-substantial/minor amendment was submitted on 20 September 2016, involving minor changes to study documentation and the replacement of the PIs at four participating sites. These changes were approved on 26 September 2016.
A substantial amendment was submitted to the HRA and the REC on 30 January 2017, involving two main changes: a change/increase in the sample size and the inclusion of the nested qualitative substudy. These changes were approved by the HRA and REC on 7 February 2017.
A third non-substantial/minor amendment was submitted on 12 April 2017, involving the replacement of the PI at one participating site. This was approved on 18 April 2017.
A fourth non-substantial/minor amendment was submitted on 3 July 2017, involving the replacement of the PI at one participating site. This was approved on 12 July 2017.
A fifth non-substantial/minor amendment to the protocol of our qualitative substudy was submitted on 21 August 2017. The amendment clarified which carers could or could not be included in the qualitative substudy. This was approved on 10 October 2017.
Recruitment procedure
Patient identification
In each participating service, members of the clinical team maintained a screening log of all new referrals to the service. For the patients not eligible to participate, the screening log recorded age group, gender and reason for ineligibility.
Consent procedure
Study procedures differed between patients with and patients without the capacity to consent.
Patients with capacity to consent
Eligible patients were approached by a member of the clinical team about participation in the study. If eligible patients were not approached by a team member then the reason for failure to do so was recorded on the screening log.
Potential participants who had been approached by a member of the clinical team were asked if they were willing to speak to a member of the research team and were handed a patient information sheet (PIS). A member of the research team then discussed the study with the patient and sought their consent to participate. For community patients, this discussion sometimes occurred over the telephone. Written informed consent was usually obtained at least 24 hours after the PIS had been handed out but could occur on the same day if it was more convenient and acceptable for the patient. If the patient declined to participate in the study then the reason for this (if known) was documented on the screening log.
Patients without capacity to consent
For patients without capacity, a personal consultee was sought for advice. For patients with no personal consultee, the advice of a nominated consultee was sought. The nominated consultee was usually another doctor working in the hospital/hospice (who was not involved in the research), a social worker, a chaplain or the patient’s general practitioner (GP).
In a similar manner to the approach adopted for patients with capacity to consent, consultees of patients without capacity to consent were advised that it was usual practice to wait for 24 hours before giving assent. If the consultee gave telephone advice for the patient to be included in the study but they were unable to visit the unit to provide written evidence of assent, then verbal agreement was initially deemed sufficient to allow the research team to enrol the patient in the study and start data collection. However, in these circumstances an assent form was posted to the consultee to be signed and returned to the research team within 2 weeks of the patient being enrolled in the study. If no signed assent form was received then the patient was withdrawn from the study and all data were destroyed.
Patients with fluctuating capacity to consent
If a patient who temporarily lacked and then recovered capacity to consent was included in the study, then, when they recovered capacity, they would be informed about their involvement in the study and would have the opportunity to withdraw or confirm participation.
Quantitative statistical methods
Outcome measures
The primary outcome measures were survival of patients (measured from the date patients consented to participate), the CPS and the predictions of the PiPS-A and PiPS-B prognostic models. The secondary outcome measures were the predictions produced by the PPI, PPS, PaP and FPN.
Sample size
The sample size calculations were based on data collected during the original study.11
For the comparison between PiPS-B model predictions and clinical prediction of survival
The primary model of interest for this research was PiPS-B. To show at least a 5% improvement in correct predictions using PiPS-B compared with clinicians’ predictions, assuming 80% power and 5% significance level and using a McNemar’s test, 1267 patients with complete PiPS-B data would be required. The formula and the software used for these calculations were based on the work produced by Machin et al.36 It was estimated that to obtain 1267 complete PiPS-B data sets it would be necessary to recruit 1334 patients with capacity to consent (assuming 5% missing data). Furthermore, to recruit 1334 patients with capacity to consent, it was estimated that it would be necessary to recruit approximately 1778 patients in total (assuming 25% of patients would lack capacity to consent).
For the validation of the PiPS models
To validate predictions from a risk model it has been recommended that the validation data should have at least 100 events.37 The validation data for PiPS2 involved several centres. There is no guidance on sample size calculation for multicentre prognostic validation data. We expected clustering of patients within centres to be minimal, based on other studies in community care.38 However, to be conservative, we inflated the number of events required in the validation data to 150. Assuming an event rate of 17.8%, based on the original study, we estimated that we would require 843 patients to validate the PiPS-B model. In fact, to have sufficient participants to compare PiPS-B with CPS, we planned to recruit approximately 1778 patients, most of whom would be able to provide data for the validation of both PiPS-A and PiPS-B. Thus, the proposed sample size was more than adequate to validate both models. Using similar arguments, the proposed sample size was also more than sufficient to validate the other prognostic models (i.e. PPI, PPS, PaP and FPN).
Statistical analyses
Descriptive analysis
Predictors and outcomes were summarised using descriptive analysis. Categorical predictors have been reported as raw numbers and percentages. Continuous variables have been summarised using mean or median and standard deviation (SD) or interquartile range (IQR) as appropriate. The percentage of values missing for each predictor has also been presented. The survival times of patients have been summarised using median and IQRs, and Kaplan–Meier graphs.
Primary analyses
Validation of PiPS models (comparison between PiPS model predictions and patients’ actual survival)
For both PiPS-A and PiPS-B, in the original study,11 two separate models were developed to predict the 2-week (14-day) and 2-month (56-day) survival of patients. The models were then combined using the original PiPS decision rule, thereby generating three prognostic categories (i.e. survival of < 2 weeks, 2 weeks to 2 months and > 2 months).
The performance of prognostic models is typically assessed by calculating measures of model discrimination and calibration. The discrimination of a model refers to how well it differentiates between those at higher risk of having the event of interest and those at lower risk. Calibration refers to the level of agreement between the observed outcomes and the predictions.39 The discriminatory ability of the PiPS models has been assessed using the c-statistic. Separate c-statistics have been calculated for the 2-week and 2-month survival models. The c-statistic can vary between zero and 1.0, where a value of 0.5 means that the model shows no ability to discriminate. Model performance has also been assessed by plotting Kaplan–Meier survival curves for each of the three risk groups identified by the PiPS models (days, weeks and months+).
Model calibration has been assessed by comparing observed and predicted probabilities by calculation of the calibration in the large (CiL) and calibration slope (CS).40 For a prognostic model to be well calibrated, the CiL should be close to zero and the CS should be close to 1.0.41
Comparison between PiPS-B model predictions and clinical prediction of survival
To compare the accuracy of the model predictions and CPS, the primary analysis focused on the PiPS-B model. McNemar’s test was used to compare the proportion of overall patient deaths predicted correctly by PiPS-B (using the original decision rule11) with the corresponding proportion predicted correctly by clinicians.
Secondary analyses
As part of the secondary analyses, the PiPS models’ predictions for the 2-week and 2-month cut-off points were combined to produce a categorical prediction of survival (days, weeks or months+) and were compared with clinicians’ estimates and the corresponding observed values with respect to their accuracy. Linear-weighted k has also been used to compare the performance of the clinicians with that of the models.
The PPI, PPS, PaP and FPN prognostic models were also assessed as part of the secondary analyses. The calibration of these prognostic models were assessed using the CS,40 based on a logistic model for binary outcomes and the Cox model for survival outcomes.42 Model discrimination was assessed using the c-statistic for binary outcomes and Harrell’s concordance index for survival outcomes.41 The performance measures estimated for the various models were compared descriptively. The predictions made by these prognostic models were also compared with the corresponding observed outcomes and clinician predictions (where available or where possible).
As multiple comparisons are performed in this report, a Bonferroni correction43 has been used to adjust the significance level for the secondary analyses. The amended significance level of 0.05/7 (0.0071) has been used for the secondary analyses.
Bias owing to missing data was investigated and multiple imputation based on chained equations44 was used to impute missing predictor values.
Data checking
Before analysis, basic checks were performed to confirm the quality of the data. Incomplete or inconsistent data included missing data, data outside the expected range and other inconsistencies. If any inconsistencies were found, the corresponding values were double checked with the researchers and corrected if necessary. All changes were documented by the study statistician. Analyses were performed after dates of death had been obtained from NHS Digital, 3 months after completion of the study. Prior to the final analysis, all relevant data were entered, checked and locked, the analysis plan was finalised and approved and the analysis programs were prepared. The primary analysis was performed independently by two statisticians (VV and RO) to ensure accuracy.
Software
Data were downloaded from the study-specific online database provided by Sealed Envelope into a format suitable for Stata® (StataCorp LP, College Station, TX, USA). Statistical analyses were performed using Stata version 14.
Methods of nested qualitative substudy
A purposive sample of patients, carers and clinicians who had been asked to participate in the quantitative study in the Manchester and Derby areas (hospice, community and hospital sites) were also asked to consider participation in the qualitative substudy. Semistructured, face-to-face interviews were conducted with patients, carers and clinicians.
Sample size
The final sample size was determined by data saturation.
Study setting
For pragmatic reasons the setting for the qualitative substudy was mostly the Greater Manchester area. The demographic diversity of this area permitted us to recruit from a wide range of backgrounds. However, following advice from the Study Steering Committee, we extended the sampling frame to Derby, to recruit a small number of palliative home care patients.
Characteristics of patient and carer sample
The patient and carer sample comprised patients with capacity to consent and carers of patients with or without capacity to consent, who had agreed to participate in the quantitative study. We also approached patients and carers who had declined to participate in the quantitative study. We purposively sampled patients and carers according to prespecified characteristics so that our sample was as varied as possible and represented the views and experiences of a wide range of patients and carers. Gender and age are known examples of factors that may influence the decision to receive prognostic information45–49 and, therefore, were characteristics for which we purposively sampled.
Characteristics of health-care professional sample
The clinician sample was pragmatic and comprised health-care professionals who routinely cared for patients with advanced cancer and made prognostic predictions, such as palliative care specialists, oncologists, nurses and GPs.
Recruitment procedure
Patients and carers
Patients or carers in the qualitative substudy were initially approached by a member of the clinical team after they had been approached about the quantitative study. At this point they were handed the PIS for the qualitative substudy and asked if they would be happy to speak to a member of the research team about the study in more detail. Because interviewing patients/carers about prognosis is sensitive and could potentially cause distress, we employed a researcher experienced in interviewing palliative patients/discussing sensitive topics.
Health-care professionals
Health-care professionals were given at least 24 hours to decide whether or not they wanted to take part and were told that they were free to withdraw at any time. Written consent was taken prior to the interview.
Scope and nature of the interviews
Interviews were interactive and explored the acceptability of PiPS and other models. Interviews used topic guides (see Appendices 1 and 2) that were based on reviews of the literature, results of previous consultations with service users and the MORECare (Methods Of Researching End of life Care) recommendations for conducting research at the end of life.50 The topic guides were iterative to allow new themes that emerged during interviews to be explored with future participants. Interview duration was mindful of participants’ needs to ensure that they were not overburdened and were expected to last < 1 hour. Interviews took place at a venue of the participant’s choice.
Topics explored in patient/carer interviews
Patient/carer interview content included the following topic areas: experiences of being approached to take part in the PiPS2 prognostic study, perspectives on the development and use of prognostic indicators, views and opinions of how best, and the most sensitive way, to present prognostic information to patients and/or relatives/carers, and opinions about the usefulness of such an indicator/tool.
Topics explored in health-care professional interviews
Clinicians were shown the prognostic models, tried them out during the interview and commented on their perceived clinical usefulness (e.g. ease of completion and interpretability of outputs). Clinician interview content included the following topic areas: experiences of making predictions of survival length to patients/carers, the information currently shared with patients/carers about prognosis and how it is conveyed, opinions on the development and use in clinical practice of prognostic indicators/tools, and barriers to and facilitators of clinical use of prognostic indicators/tools.
Conduct of the interviews
Face-to-face interviews were conducted by the researcher at a location to suit the participant. Generally this was a private side room of the hospice/hospital ward. However, some patients preferred the interview to take place at their bedside, especially if they were feeling particularly unwell. Some participants wished to be interviewed in their own homes.
Qualitative data analysis
Interview data were entered into NVivo 10 (QSR International, Warrington, UK) and analysed using the five stages of framework analysis:51 familiarisation, developing a thematic framework, indexing, charting, and mapping and interpretation. During the first stage (familiarisation), the research team became immersed in the data52 by reading and rereading the transcripts and discussing emerging themes. A thematic framework was then developed based on the topic guide.53 After this, transcripts were indexed (coded) line by line using the thematic framework but we remained open to new themes that emerged.54 Next, the data were entered into a chart so that coded extracts could be attributed to individual participants. Finally, participants’ views were compared and contrasted and the data were presented schematically (mapping). Rival explanations were explored.
An iterative and inductive approach to analysis was followed so that data analysis started alongside data collection. Themes and issues identified from interviews informed further questions and probing in both ongoing and future interviews. Memoranda and notes were written about emerging themes to summarise or analyse a point and to relate emerging ideas and concepts to existing literature. The qualitative research team met on a regular basis to discuss the development of codes, themes, categories and theories about the phenomenon being studied.
Emotional/physical distress during qualitative interviews
Owing to the sensitive nature of the research relating to discussions of prognostic information of palliative care patients, we developed a distress policy for the researcher completing face-to-face interviews to consider. For the qualitative substudy we collected data from patients, their relatives/carers and clinical staff shortly (a few days) after patients had been approached to take part in PiPS2. If the patient/carer became upset during the interview, the researcher asked if they would like to stop the interview and, if the interview was stopped, whether or not they wished to resume. If further support was needed, the researcher linked participants to existing support mechanisms in the hospice/organisation. All patients had already been referred to palliative care services and the relevant clinical services were notified of any distress detected during the study for follow-up.
