Study quality assessment

A quality appraisal was conducted on cost-effectiveness studies or other economic evaluations, using the well-established Drummond checklist,⁴⁷ following advice from CH.

Data extraction strategy

For those studies which were of relevance to the current decision problem, data were extracted by two researchers (NH, TS) into three data extraction tables: one to describe the study design of each economic evaluation, one to describe the main results, and the third a checklist of review-specific criteria that the two reviewers had previously agreed upon. Examples of the study design and results tables are provided in Tables 26 and 27, with the review-specific criteria checklists presented in Tables 28–30, respectively.

TABLE 26

Example data extraction table for study characteristics

TABLE 27

Example data extraction table for study results

TABLE 28

Review-specific criteria to assess included cost-effectiveness studies that compared strategies to identify and manage LS

TABLE 30

Review-specific criteria to assess included cost-effectiveness studies that compared strategies to manage LS (no diagnosis)

TABLE 29

Review-specific criteria to assess included cost-effectiveness studies that compared strategies to identify LS (no management)

In the study design table, the sections included study, publication type, setting/perspective, industry role/conflicts of interest, population, study purpose, outcomes measured, diagnostic strategies, treatment strategies, study approach, health states, model duration/cycle length, the approach to uncertainty analysis, base-year prices and discount rate.

In the results table, the components were the analysis year, the base-case results and the main sensitivity analysis results.

Items on the review-specific checklist included the types of cancers included in the analysis (both in the input population and in the management section); whether or not the interactions of these cancers in the long term were considered appropriately; whether or not the diagnostic strategies used FH; which MMR gene mutations were tested for; whether or not differences in LS cancer incidence due to these MMR gene mutations, and the improved survival of LS CRCs relative to sporadic CRCs, were accounted for; whether or not the study considered the difference in CRC incidence between males and females; whether or not adherence to counselling, genetic testing and management strategies was included; whether or not the psychological impact of genetic testing was accounted for; and whether or not the study assessed the impact of diagnostic errors.

Synthesis of extracted evidence

The evidence base was assessed using narrative synthesis supported by abridged data extraction tables.

Summary of previous cost-effectiveness reviews

There were four papers identified whose cost-effectiveness section consisted purely of a review of previous cost-effectiveness studies,^77–80 as well as two cost-effectiveness model papers^65,69 that contained a review section. The purpose of reviewing these studies was to investigate whether or not a systematic review had been conducted previously, and also to check for any additional studies that our search may have missed.

Only two reviews^77,78 were studies from the last 10 years (2006 and 2012, respectively) and were clear on their search strategy and methods, but neither of them looked exclusively at LS. Furthermore, neither review identified more than four studies related to LS. Between them they identified only seven separate reviews,^{23,49,50,59,61,81,82} of which all but one had already been identified by our review. The one study we had not identified, by Hagen and colleagues,⁸² was written in German and therefore excluded from our review. The only results presented by Antonanzas and colleagues were from the Barrow and colleagues (2008)⁸¹ paper, which we excluded on the basis of population. The four studies^23,49,50,59 reported by Rogowski and colleagues⁷⁷ were all included in our review, which had a more comprehensive set of studies.

Of the other two papers with a review of previous cost-effectiveness studies, only one could be considered an actual review. The other,⁷⁹ from 2001, mentions the findings of the 1998 study by Vasen and colleagues,⁷³ but no further details are given. This study was excluded from our review on the basis of study design. The other review,⁸⁰ from 2008, identified six studies, two of which were not identified by Rogowski and colleagues.^53,73

Two of the included cost-effectiveness studies also conducted reviews of previous cost-effectiveness studies.^65,69 Again, the details of these reviews are not in depth and did not identify any additional papers that we had not already screened.

In the reports where authors made conclusions about the studies they had identified, the main conclusion was that strategies to identify and manage LS could be cost-effective, depending on the cost-effectiveness threshold of the relevant health-care provider, but that there was no conclusive evidence as to which strategies would be most cost-effective. Phillips and colleagues (2008)⁸⁰ suggested that further research was needed, particularly as the understanding of LS has changed in recent years. This assumption also explained the conclusion of Palomaki and colleagues (2009),⁶⁵ who criticised the studies they had included^51,53,73,83 for being inadequate as they did not include certain tests and did not address the differences in performance of these tests for different MMR gene mutations. These reviews identified 12 studies that were relevant to our review; this was significantly fewer than the number our search had identified, suggesting that there might be further work that these previous reviews had missed and justifying the need for our own systematic review. As our search had already identified the 12 studies that these reviews had also included, we were confident of the completeness of our review.

Summary of previous effectiveness models

As part of our review, we identified and assessed previous studies that included effectiveness models. This was to provide insight into what modelling had previously been done in this area and to help guide our own modelling efforts. We identified five previous studies that included some attempt at modelling without costs.^83–87 Four of these studies^83,85–87 looked exclusively at management strategies for people with LS; three related to the management (such as use of aggressive colorectal surgery to prevent further CRCs) and prevention of CRC, and one examined the prevention of EC. The study on EC⁸⁵ appeared to have been later updated to include costs, in a paper by Yang and colleagues (2011).⁷⁶ The general finding of these studies was that management strategies were effective for LS patients and that aggressive prophylactic surgery [total abdominal hysterectomy with bilateral salpingo-oophorectomy (TAHBSO) for EC or proctocolectomy/total colectomy for CRC] was generally the most effective strategy. All of the CRC studies employed some form of Markov modelling and the EC model was quoted to be a decision model built in TreeAge 2004 (TreeAge Software, Inc., Williamstown, MA, USA). The final paper⁸⁴ was a resource minimisation study employing a test accuracy study and a decision tree to compare strategies of testing for LS, and finding that a strategy that included BRAF testing was more effective than one without (reducing the number of genetic tests by 17%, for the same number of mutations identified).

These models were not of great use as they each focused on one aspect of LS and so did not give much insight into how to model both the diagnostic and management pathways for LS, including more than one cancer, as was required of our model. They did, however, indicate that a new model may be necessary.

Summary of cost-effectiveness studies

For the purposes of the review, we split the 32 cost-effectiveness studies into the three subgroups: those that looked at the short-term cost-effectiveness of identifying LS; those that looked only at the long-term cost-effectiveness of management of LS; and those that examined the long-term impact on cost-effectiveness of both the strategies to identify and manage LS.

Given the large number of studies which our review identified, we do not report each study in great detail, preferring to identify important similarities and differences and potential areas of improvement for which our model was intended to provide information. The summary tables of characteristics and results are shown below (see Tables 31–36). Our model was required to look at the long-term consequences of diagnosis and management of LS, so these studies were the main focus of our review. As such, the other two subgroups are reported in less depth, though their summary tables are reported in similar detail.

TABLE 31

Characteristics of cost-effectiveness studies that compared strategies to identify LS (no management)

TABLE 36

Results of studies that included strategies to both identify and manage LS

Following on from the summaries, we also present the results of our quality appraisal for these studies.

Lynch syndrome diagnosis-only studies

This was the largest subgroup of cost-effectiveness studies identified, including 15 studies looking exclusively at the short-term cost-effectiveness of strategies to diagnose LS.^34,59–72 All studies were published between 2000 and 2012 and summary details of each study can be found in Tables 31 and 32.

TABLE 32

Results of cost-effectiveness studies that compared strategies to identify LS (no management)

There was consistency in the perspective of these studies, as all but one were from a health-care perspective; the other⁶⁷ was from the perspective of a third-party payer. Most other aspects of the studies were quite varied.

There were six studies from the USA,^{59,65,67–69,71} six from European countries,^{34,60–63,66} one from China⁶⁴ and two where the setting was not clear.^70,72 Most studies were not specific about the ‘current practice’ in their respective countries, but their strategies were generally compared against a no testing strategy, implying that there was no standard ‘current practice’ in each of the countries or that no testing for LS was the ‘current practice’. Relevant to our assessment, one of the European studies was based in the UK,⁶⁰ but this was 9 years old at the time of writing and was not consistent with our input population; Pigatto and colleagues⁶⁰ investigated families referred for genetic testing, rather than newly diagnosed CRC patients aged < 50 years, who were the focus of our investigation.

With regards to the other studies’ input populations, one looked at EC patients⁶⁷ and three (including the UK study) examined families who were referred to genetic testing centres,^60,62,66 neither of which were the target population of our TA. The other 11 studies followed newly diagnosed CRC patients, and three of these^59,65,72 included the impact on relatives of the CRC patients.

As well as having different study populations, the different studies also assessed different tests, which included the following subsets: clinical criteria/FH,^{34,59–61,63,64,67,68,70} prediction models,⁶⁶ tumour testing^{65,34,59–64,66–72} and genetic testing.^{65,34,59–64,66–72} Genetic testing was split into gene sequencing for probands and targeted genetic testing for relatives. Not all genetic testing was conducted on all four known LS genes, primarily because MSH6 and PMS2 are more recent discoveries. All of the studies had strategies that included universal genetic testing and at least one that included a tumour-based test followed by genetic testing. Most also included some variant of clinical criteria, FH criteria or a prediction model based on these criteria.

In addition to the different individual tests, each of the studies evaluated a different combination and sequence of tests. In general, these combinations fell into the pattern of clinical criteria/FH/prediction model followed by tumour testing followed by genetic testing, though in some strategies certain steps in this sequence included multiple tests or were missed out entirely. Most studies included at least one strategy that did not include FH. Our assessment attempted to cover all plausible, well-recognised strategies and therefore included strategies similar to those presented in some of these studies.

As well as the testing strategies, we were interested in the acceptance of genetic testing and genetic counselling. Ten of the studies from this section^59,61–69 mentioned appropriate informed genetic counselling (two were unclear^71,72), but only four^61,65,68,69 explicitly modelled the acceptance associated with it or genetic testing. This indicated that this was an area where our model could provide added value.

With regards to the study design, nine papers^{34,60–64,66,68,70} included primary research (prospective or clinical) and 12^{34,59–63,65–67,69,71,72} included some form of decision modelling, usually using a decision tree approach. Six papers^{34,60–63,66} used a combination, using the study to inform the decision tree results. In every study, the costs and numbers of people with mutations detected were reported, and therefore the cost per mutation detected and the related incremental cost-effectiveness ratios (ICERs) could be calculated if they were not reported. In general, genetic testing alone had larger ICERs than strategies with other forms of testing included, and was stated to be more cost-effective, but as these ICERs are based on cost per case detected and not on life-years or QALYs of patients, this is not a measure of cost-effectiveness that has any meaning within our TA framework. For the other tests, there were mixed results for which were most effective, but, in general, when tumour-based tests were tested against FH or clinical criteria (CC) they were found to have lower ICERs compared with no testing. Given that the measure of cost-effectiveness has little meaning to our decision process – and that there are several different tests within the subgroups tumour-based tests, FH and CC – the results do not indicate which individual test is the most cost-effective.

Of the 12 studies^{34,59–63,65–67,69,71,72} that included models, seven^{59,62,65–67,69,72} conducted a sensitivity analysis (four univariate sensitivity analyses,^59,62,69,72 one probabilistic sensitivity analysis,⁶⁵ two threshold analyses^66,67). The only consistent results from these sensitivity analyses were that the most influential parameters were the prevalence of LS and cost of testing for LS.

Lynch syndrome management-only studies

This group of studies, which only assessed the long-term cost-effectiveness of management strategies for LS, was the smallest subgroup of studies identified by our systematic review. The earliest study was published in May 1998⁷³ and the most recent in May 2011.⁷⁶ Details of these studies can be found in Tables 33 and 34. There were four included studies,^73–76 three from a health-care provider perspective^73,74,76 (one of which claimed to be societal) and one from a societal perspective.⁷⁵ Again there was a range of settings: two US,^75,76 two European.^73,74 As with the diagnostic studies, there was one study based in the UK,⁷⁴ which assessed the cost-effectiveness of biennial colonoscopic surveillance in families meeting the AC or with confirmed MMR gene mutations. This was also the only study with a mixed-sex cohort and was therefore more relevant to our assessment. However, it only considered colonoscopic surveillance, and therefore CRC, as long-term concerns for LS patients. Of the other three studies, two investigated the cost-effectiveness of gynaecological surveillance and prophylactic gynaecologic surgery on an all-female cohort aged 30 years,^75,76 and one investigated the cost-effectiveness of colonoscopic surveillance (annually or 2.5-yearly) for an all-male cohort aged 25 years.⁷³ All populations were assumed to be asymptomatic at the start of each model.

TABLE 33

Characteristics of cost-effectiveness studies that compared strategies to manage LS (no diagnosis)

TABLE 34

Results of cost-effectiveness studies that compared strategies to manage LS (no diagnosis)

The modelling approach in these studies varied: both studies investigating colonoscopic surveillance used decision tree modelling;^73,74 Kwon and colleagues’ study⁷⁵ on gynaecological surveillance/prophylactic surgery used a Markov cohort model and Markov microsimulation approach; and Yang and colleagues’ similar study⁷⁶ used a decision model built in TreeAge.

All studies reported costs, but they reported different measures of effectiveness. Both studies investigating gynaecological management reported QALYs,^75,76 Vasen and colleagues⁷³ reported life expectancy and Dunlop and colleagues⁷⁴ reported lives saved. Generally in these studies, the authors concluded that any management strategies could be cost-effective. For EC, prophylactic surgery was seen as more cost-effective than gynaecological surveillance, even under sensitivity analysis. For preventing CRC, colonoscopic surveillance dominated no surveillance in the Vasen and colleagues study,⁷³ even under sensitivity analysis, and the cost per life saved in the Dunlop and colleagues study⁷⁴ was £14,925, which they concluded was a ‘favourable’ result.

Studies that incorporated both diagnosis and management of Lynch syndrome

This subgroup contained 13 studies^{22,23,48–58,88} focused on the cost-effectiveness of strategies to first identify and then manage LS, published over a period of 18 years (1995–2012). Six of these were published between 2010 and February 2013. As Ladabaum and colleagues (2011)⁴⁸ and Wang and colleagues (2012)⁵⁷ were based on the same model, they are discussed as the same study in this section of the review.

Details of all these studies can be found in Tables 35 and 36.

TABLE 35

Characteristics of studies that included strategies to identify and manage LS

Methods

As with the previous two groups of studies, the majority of these studies were from a health sector perspective. There was also one which was conducted from a government perspective⁵¹ and one from a third-party payer perspective.^48,57 The majority of studies were set in the USA, with one from Singapore,⁵⁸ one from Australia,⁵¹ two from Denmark^52,53 and one from the Netherlands.⁵⁰ No UK studies were reported. The most common input population in the studies, occurring in five separate studies, was newly diagnosed CRC patients and their relatives.^{23,48,49,50,54,57} These studies were particularly relevant to our assessment as our input population lay within this group of patients. Other input populations included healthy, unaffected members of the general population;^22,55,88 FDRs of known LS mutation carriers;^51,58 families referred to a genetics registry;^52,53 and, in one study, newly diagnosed EC patients.⁵⁶

Similar to the diagnostic-only studies, the majority of the diagnostic strategies in these studies included some form of CC/FH, tumour testing and genetic testing. Steps such as FH were sometimes missed out entirely, or strategies included several tests. CC/FH included the Bethesda and Revised Bethesda guidelines and AC I/II (and selected criteria from either guideline), as well as prediction models such as PREMM (prediction of mismatch repair gene mutations), MMRPro and MMRpredict. These prediction models were algorithms that used specified patient history, CC and FH to determine the likelihood of LS for a patient. Tumour testing in these studies included MSI, IHC, BRAF and methylation. The level of testing was not always the same; MSI was conducted on different numbers of markers in different studies, and sometimes IHC was conducted only for specific MMR proteins rather than all four. Genetic testing was divided into sequencing tests for probands and predictive tests for their relatives. Each of the studies used a different set and combination of these tests, making it difficult to decide which strategies were most commonly used.

Across the studies, the management strategies for CRC were fairly consistent. Colonoscopy every 1–2 years from age 25 years, with more aggressive colorectal surgery on diagnosis of CRC to prevent further CRCs, was the management strategy for patients diagnosed with LS in the majority of studies. This was consistent with the current published guidance identified by our systematic review. For the studies that considered EC, the management strategies included gynaecological surveillance/screening with prophylactic TAHBSO at an appropriate age and were once again fairly consistent across the studies.

The design of these studies was predominantly decision modelling with a mix of decision trees, Markov cohort modelling and a variety of ad hoc modelling.

As knowledge about LS has altered over recent years, we thought it prudent to assess how up to date the various studies were with regards to this knowledge. One parameter we thought particularly influential to the cost-effectiveness analysis was the risk of CRC in LS patients, as a lower estimate of CRC risk was likely to make strategies for diagnosing and managing LS less cost-effective. We summarise the values (and sources) for this parameter used in the studies in Table 37. There was a divide between those studies that were more than 5 years old, where lifetime CRC risk in LS patients was generally around 80%, and the more recent studies, in which lifetime CRC risk for LS patients was reported to be closer to 40–50%. This divide also provided us with information on which studies’ results were more likely to be comparable with ours; only 6 of the 13 studies^48,54–58 used a lower (40–50%) estimate of CRC risk. The risk of CRC is also known to be different in males and females, which was only modelled explicitly in two of the studies.^48,55 These were also the only two studies to consider more than CRC in the long term, suggesting that this was an area where further modelling could add value.

TABLE 37

Colorectal cancer risk parameter included in cost-effectiveness studies

Modelling of imperfect adherence to testing and management of LS appears to occur in over half of the studies; however, it is not always clear what patients are exactly complying with. In the genetic testing process, there is adherence to both genetic counselling and genetic testing. Some studies were explicit about the two separate sections; some appeared to treat this as one adherence issue; and some did not state it at all. Adherence to colonoscopic surveillance was included in a much clearer manner and was modelled in most studies, so that those who complied initially with surveillance continued to do so.

Results

The majority of studies reported life-years and costs (and the respective ICERs) as their main outcomes. Two studies, the update to Ladabaum and colleagues (2011)⁵⁷ and Dinh and colleagues (2011),⁵⁵ reported QALYs rather than life-years and one study⁵¹ reported CRC-free years as the main outcome. Mvundura and colleagues (2010)⁵⁴ reported QALYs as a scenario analysis, scaling their ICERs by 1.18 life-years per QALY, which is an approach that demands that a number of assumptions be made. One early study²² reported the short-term outcome of cost per mutation identified, as well as the long-term cost per life-year saved. A range of discount rates were used in the long-term calculations, generally ranging from 3–5% per year for both costs and benefits.

In general, the studies concluded that strategies that screened for LS were cost-effective compared with no screening, with all finding at least one strategy that fell below a pre-specified threshold. However, given the different strategies and costs for each country, there was little consistency over which strategies or individual tests were the most cost-effective. In two studies,^48,54 IHC with BRAF appeared to be the most cost-effective strategy. In all studies where strategies with additional tests were included, universal genetic testing was not cost-effective.^{48,49,54,56,57}

The minimum uncertainty analysis performed by these studies was a univariate sensitivity analysis. Most studies looked at a large number of parameters in their univariate analysis, but only reported those that were most influential on the cost-effectiveness results, which made comparison between them difficult. Influential parameters that were identified by more than one study included age of population, number of relatives, cost of testing, effectiveness of the diagnostic and management strategies, and prevalence of LS in the population.

Quality appraisal of cost-effectiveness studies

Though the studies were assessed using all of the criteria from the Drummond checklist,⁴⁷ specific criteria are reported in Table 38.

TABLE 38

Selected quality appraisal criteria from the Drummond checklist

One consistent problem in terms of the quality of the studies was that the reporting of perspective was poorly done, with 25 studies not stating and justifying their perspective.^{22,23,34,49,50,52,53,55–60,62–66,68–73,76} This included three studies^55,64,66 not stating a perspective, two^70,72 not stating a setting and two^23,76 incorrectly stating a health-care perspective as societal.

The depth of detail related to modelling (when it was reported) was mixed and ranged from brief descriptions to full details. Sensitivity analysis was conducted in seven of the diagnostic papers,^{59,62,65–67,69,72} three of the management papers^73,75,76 and all of the diagnostic-plus-management papers.^{22,23,48–58} Justification for ranges used in sensitivity analyses was poorly given, if at all. Reporting of sources was generally well done, as was the reporting of sources and methods of estimation for unit costs and quantities.

Of the three groups of studies (diagnostic, management and diagnostic plus management), diagnostic studies seem to be the least well reported; as well as only two^61,67 clearly stating and justifying their setting and viewpoint, only three^59,67,69 reported a currency and/or price date and only 5 of the 12 models^{59,62,67,69,72} gave details of the modelling.

As one of our included papers was only a summary,⁵² it did poorly in the quality assessment. The accompanying full report was in Danish and could not be quality assessed as it was not included in our report.

The small number of management-only papers made it difficult to draw overall conclusions about their quality, but again it appeared to be mixed, with similar problems to the other two groups of studies.