Meta-analysis of diagnostic tests for acute kidney injury

Peter S Hall; Elizabeth D Mitchell; Alison F Smith; David A Cairns; Michael Messenger; Michelle Hutchinson; Judy Wright; Karen Vinall-Collier; Claire Corps; Patrick Hamilton; David Meads; Andrew Lewington

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Hall PS, Mitchell ED, Smith AF, et al. The future for diagnostic tests of acute kidney injury in critical care: evidence synthesis, care pathway analysis and research prioritisation. Southampton (UK): NIHR Journals Library; 2018 May. (Health Technology Assessment, No. 22.32.)

Cover of The future for diagnostic tests of acute kidney injury in critical care: evidence synthesis, care pathway analysis and research prioritisation

The future for diagnostic tests of acute kidney injury in critical care: evidence synthesis, care pathway analysis and research prioritisation.

Show details

Contents

< Prev Next >

Chapter 3Meta-analysis of diagnostic tests for acute kidney injury

Introduction

In this chapter a meta-analysis of diagnostic accuracy studies is provided to evaluate the body of evidence available for three diagnostic tests of AKI and to provide input into the decision analysis in subsequent chapters. The primary health-care setting considered in the use of these tests was the critical care unit and the secondary health-care setting considered was cardiac surgery (pre and/or post intervention). The three diagnostic tests considered were the Nephrocheck test (which uses a combination of two proteins: TIMP-2 and IGFBP-7) and the biomarkers NGAL and cystatin C. The NGAL and cystatin C tests have been used in this setting for measurement of their concentration in samples of blood serum, blood plasma and urine and these media were considered separately. The pooled estimates of sensitivity and specificity and their variance from the meta-analyses directly informed the decision analysis in Chapter 5. In our searches we identified no previous reviews or meta-analyses considering Nephrocheck or cystatin C in this setting, but we did identify two relevant reviews of NGAL.⁴⁰^,²⁴⁵

Methods

Primary objective

The primary objective was to estimate pooled means and variances of sensitivity and specificity for each of the diagnostic tests considered. When appropriate data were available, these estimates were obtained separately for each of the health-care settings considered and for each of the sample media considered. When only one study was available, no meta-analysis was undertaken.

Identification of studies

Details on the search strategies used to identify studies and the process for study screening and evaluation, data extraction and quality assessment are presented in Chapter 2.

Full papers were retrieved for studies of all patients (< 18 years, ≥ 18 years) in which AKI diagnosis had been evaluated using any one or multiples of the three diagnostic tests considered (Nephrocheck, NGAL, cystatin C), in any of the sample media considered (blood serum, blood plasma or urine), in either of the health-care settings considered (critical care unit or cardiac surgery). Studies were excluded if they were not primarily located in either of these health-care settings.

Study methods

The gold standard for determining AKI diagnosis was defined as diagnosis according to the RIFLE,²⁴¹ AKIN²⁴² or KDIGO (Kidney Disease: Improving Global Outcomes)²³⁸ diagnostic and classification system, based on an assessment of serum creatinine levels and urine output (see Chapter 2).

Outcome measurements

The primary outcomes were sensitivity (the probability of the test being positive given that the true diagnosis is positive) and specificity (the probability of the test being negative given that the true diagnosis is negative), which are determined by comparison of the results of the experimental diagnostic test with the results of the gold standard method used in the study. Studies were excluded if the gold standard method used to determine the outcome was not described in sufficient detail. Studies were not excluded if the cut-off point used to assess the positive and negative status of the outcome in the experimental diagnostic test was not reported.

Diagnostic and staging systems for acute kidney injury

A number of diagnostic and staging systems for AKI have been used in diagnostic accuracy studies. The most commonly used make use of repeated serum creatinine measurements and measurement of urine output to diagnose and stage AKI. Three commonly used systems are the RIFLE, AKIN and KDIGO systems.

RIFLE classification of acute kidney injury

The Acute Dialysis Outcome Initiative group proposed the RIFLE classification, which defines five categories of AKI,²⁴¹ as shown in Table 8. AKI is staged for severity according to the criteria listed in Table 8, with any classification of risk or above being a diagnosis of AKI.

Acute Kidney Injury Network classification of acute kidney injury

The AKIN has defined diagnostic criteria for AKI and provided a staging system for the severity of AKI,²⁴² as shown in Table 8. AKI is staged for severity according to the criteria listed in Table 8, with any classification of stage 1 or above being a diagnosis of AKI. In particular, in contrast to the RIFLE criteria, the absolute change in serum creatinine defining AKI is defined as an abrupt (within 48 hours) reduction in kidney function as defined by stage 1 or above.

KDIGO classification of acute kidney injury

The 2011 KDIGO Clinical Practice Guideline for AKI (Summary of Recommendation Statements, 2012)²⁴⁶ defined diagnostic criteria for AKI and provided a staging system for the severity of AKI, as shown in Table 8. AKI is staged for severity according to the criteria listed in Table 8, with any classification of stage 1 or above being a diagnosis of AKI. This classification system uses the same time frame for absolute changes as the AKIN criteria and clarifies that for the relative changes the baseline values should be known or presumed to have occurred within the previous 7 days.

Summary of staging methods

There is a similarity between the staging and diagnostic criteria proposed for AKI, which is demonstrated in Table 10. It has been shown that the AKIN criteria can diagnose more patients correctly with AKI than the RIFLE criteria (not unexpected given the additional criterion – the absolute change in serum creatinine level), but it has not been shown to have a better predictive ability for in-hospital mortality.²⁴⁷ It has also been shown that the AKIN criteria do not improve the sensitivity of AKI diagnosis compared with the RIFLE criteria in the first 24 hours after admission to the critical care unit.²⁴⁸ Similarly, it has been shown than a higher incidence of AKI can be diagnosed using the KDIGO criteria than using the RIFLE criteria and that the KDIGO criteria are more predictive for in-hospital mortality, but there was no significant difference between the AKIN criteria and the KDIGO criteria.²⁴⁹ Other studies have suggested that the RIFLE, AKIN and KDIGO criteria are good tools for predicting mortality in critically ill patients and observe no evidence of a difference between them.²⁵⁰

TABLE 10

Comparison of common AKI diagnostic and staging/classification systems based on serum creatinine levels and urine output

Based on the definitions used in the different diagnostic and staging/classification systems and the evidence above we believe that there are broad similarities between the RIFLE, AKIN and KDIGO criteria and, for the purposes of this study, we defined a diagnosis of AKI, following the KDIGO criteria, as any of the following:

increase in serum creatinine of ≥ 0.3 mg/dl (≥ 26.5 µmol/l) within 48 hours
increase in serum creatinine to ≥ 1.5 × baseline, which is known or presumed to have occurred within the previous 7 days
urine volume < 0.5 ml/kg/hour for at least 6 hours.

Furthermore, in the studies identified for inclusion in the meta-analysis, studies that used either of the outcomes indicated by the shaded areas in Table 10 (RIFLE R, AKIN 1 or KDIGO 1 – a diagnostic-type outcome; RIFLE F, AKIN 3, KDIGO 3 or RRT – a failure-type outcome) were considered homogeneous for the purposes of the meta-analysis.

Key data extracted

The primary data extracted for inclusion in the meta-analysis are shown in Table 11. It is recommended in the STARD statement that a cross-tabulation of the index test results by the results of the reference standard is included in any study report,²⁵¹^,²⁵² but it was anticipated that this information would not be present in all study reports. In this situation the elements of the confusion matrix were calculated using information describing the diagnostic outcomes and estimates of sensitivity and specificity. For example, if the sensitivity (s) and number of true diagnoses [given by the sum of the number of true positives (TPs) and the number of false negatives (FNs), i.e. (TP + FN)] were reported in the study then the number of TPs could be calculated as s.(TP + FN). A similar calculation for specificity (p) allowed the estimation of the number of true negatives (TNs): p.(FP + TN), where FP represents the number of false positives. Finally, given these estimates for TP and TN and the numbers of true outcomes [(TP + FN) and (FP + TN)], simple subtraction provided estimates for FN and FP.

TABLE 11

Confusion matrix

Study exclusion

Studies were excluded from the meta-analysis if it was not possible to estimate values for the elements of the confusion matrix or if other key data could not be extracted. Further reasons for the exclusion of studies were if diagnosis was carried out in the emergency department rather than in the critical care unit and if the biomarker was measured on a relative scale rather than an absolute scale, for example unit of biomarker per unit of serum creatinine.

Data analysis

Simple diagnostic accuracy summaries [sensitivity, specificity and the diagnostic odds ratio (DOR) and its components – positive likelihood ratio (LR+) and negative likelihood ratio (LR–)] were produced for each study included in the meta-analysis. The sensitivity of a diagnostic test (T) is defined formally as the probability that the test will give a positive result if the patient has the disease (D+), in this case AKI. This is often referred to as the TP rate for a diagnostic test and can be expressed as a conditional probability:

s = Sensitivity = P (test = positive | status = diseased) = P (T + | D +) .

(1)

The specificity of a diagnostic test is the probability that the test will give a negative result if the patient does not have the disease (D–), which is equivalent to 1 minus the FP rate for the test and can be expressed as the conditional probability:

p = Specificity = P (test = negative | status = not diseased) = P (T - | D -) .

(2)

Confidence intervals (CIs) were estimated for sensitivity and specificity based on the Wilson score interval method.²⁵³

The LR+ of a diagnostic test is the probability of a patient with disease (D+) having a positive test result divided by the probability of a patient without disease (D–) having a positive test result:

LR + = P (T + | D +) / P (T + | D -) .

(3)

Similarly, the LR– of a diagnostic test is the probability of a patient with disease having a negative test result divided by the probability of a patient without disease having a negative test result:

LR - = P (T - | D +) / P (T - | D -) .

(4)

Confidence intervals for the LR+ and LR– were estimated using the method of Koopman.²⁵⁴

The DOR for a test is the ratio of the odds of a positive test result for a patient with disease relative to the odds of a positive test result for a patient without disease:

DOR = LR + / LR - .

(5)

Confidence intervals for log(DOR) were estimated based on the assumption that, as an odds ratio, the DOR is normally distributed. Estimates for DOR were then obtained by back-transformation.

The method of meta-analysis for diagnostic accuracy studies used here was the bivariate meta-analysis proposed by Reitsma et al.,²⁵⁵ based on the methodology of van Houwelingen et al.²⁵⁶ Briefly, if logit sensitivity (µ_si) and logit specificity (µ_pi) are

µ_{S i} = logit (s_{i}) = log (\frac{s}{1 - s})

(6)

and

µ_{P i} = logit (p_{i}) = log (\frac{p}{1 - p}), i = 1, . . ., k .

(7)

for each study i (with k studies included in the meta-analysis), the true logit sensitivity and logit specificity are then assumed to have a bivariate normal distribution across studies:

(\frac{µ_{S i}}{µ_{P i}}) ∽ N ((\frac{µ_{S}}{µ_{P}}), Σ_{S P}),

(8)

where

Σ_{S P} = (\begin{matrix} σ_{S}^{2} & σ_{S P}^{n} \\ σ_{S P}^{n} & σ_{P}^{2} \end{matrix}) .

where $σ_{S P}^{n}$ is the covariance between logit sensitivity and logit specificity. This model is extended by incorporating the variability due to sampling through the variance of sensitivity ( $s_{S, i}^{2}$ ) and specificity ( $s_{P, i}^{2}$ ), as measured in each study:

s_{S, i}^{2} = \frac{1}{n_{S, i} p_{S, i} {(1 - p)}_{S, i}}

(9)

and

s_{P, i}^{2} = \frac{1}{n_{P, i} p_{P, i} {(1 - p)}_{P, i}},

(10)

assuming that 0 < p and s < 1 and that the number of subjects used to estimate sensitivity and specificity is large.²⁵⁶ The final model is then a bivariate random-effects model of the form:

(\frac{µ_{S i}}{µ_{P i}}) \sim N ((\frac{µ_{S}}{µ_{P}}), Σ_{S P} + C_{i}),

(11)

where

C_{i} = (\begin{matrix} s_{S, i}^{2} & 0 \\ 0 & s_{P, i}^{2} \end{matrix}) .

This model was estimated using likelihood-based methods using the mada package²⁵⁷ in the R Environment for Statistical Computing (The R Foundation for Statistical Computing, Vienna, Austria). It has been shown that this method is equivalent to the hierarchical regression meta-analysis proposed by and further developed by Rutter and Gatsonis when there are no study-level covariates.²⁵⁸^–²⁶⁰

Separate meta-analyses were conducted for each diagnostic test, sample media and health service setting. Pooled estimates of sensitivity, specificity, LR+, LR– and DOR can be estimated from back-transformed parameter estimates. Estimates from each study and pooled estimates from the meta-analysis are presented in forest plots. A summary receiver operating characteristic (SROC) curve was estimated, with estimates of the confidence and prediction region.²⁵⁵^,²⁵⁹ Approximate estimates of the variance of sensitivity and specificity for use in the economic model were determined using the delta method.²⁶¹

Tests of heterogeneity were not used, as such statistical methods (Cochran’s Q, I²) do not account for heterogeneity explained by phenomena such as positivity threshold effects and are not recommended by the Cochrane Diagnostic Test Accuracy Group.²⁶² Estimating the prediction region in the SROC curve is one way of examining the extent of heterogeneity by depicting a region within which, assuming that the model is correct, we have 95% confidence that the true sensitivity and specificity of a future study would lie.²⁶⁰

Results

Papers selected for inclusion in the meta-analysis are described briefly in tabular summaries followed by a summary of diagnostic accuracy for each study. Pooled estimates of sensitivity and specificity and the SROC curve are also provided for each diagnostic test, health-care setting and sample type. A Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram that depicts the flow of information through the different phases of the systematic review to data extraction and final inclusion in the meta-analysis is shown in Chapter 2 (see Figure 2).

Nephrocheck

Critical care unit: plasma and serum

The searches identified no studies suitable for data extraction for this diagnostic test in either health-care setting for either plasma or serum.

Critical care unit: urine

Summaries of the baseline characteristics and test parameters for the included urinary Nephrocheck studies are shown in Table 12. Three studies were included,³³^,⁴⁴^,⁴⁵ with a total of 1289 patients [199 patients (15.4%) with a diagnosis of AKI and 1090 patients (84.6%) without a diagnosis of AKI]. The sample for use in the test was taken on enrolment in all of the included studies. The outcome used to define the presence of AKI was consistent across each of the three studies (KDIGO stage 2 or 3). Similarly, the threshold used to define a positive test was consistent in all included studies [(TIMP-2) × (IGFBP-7) = 0.3]. Diagnostic accuracy summaries for the included studies are shown in Table 13.

TABLE 12

Characteristics of studies included in meta-analysis for Nephrocheck in the critical care unit health-care setting using urine

TABLE 13

Diagnostic accuracy summaries for studies included in meta-analysis for Nephrocheck in the critical care unit health-care setting using urine

Figure 3 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for Nephrocheck in the critical care unit using patient urine samples. The pooled sensitivity estimate was 0.90 (95% CI 0.85 to 0.93) and the pooled specificity estimate was 0.49 (95% CI 0.46 to 0.53). Figure 4 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction and confidence regions are small, suggesting limited heterogeneity. This is to be expected given the highly controlled similarity in these studies.

FIGURE 3

Forest plot for studies included in the meta-analysis and pooled estimates for sensitivity and specificity Nephrocheck in the critical care unit health-care setting using urine.

FIGURE 4

Summary receiver operating characteristic curve for studies included in the meta-analysis for Nephrocheck in the critical care unit health-care setting using urine.

Cardiac surgery: urine

One study including 50 patients [26 patients (52.0%) with a diagnosis of AKI and 24 patients (48.0%) without a diagnosis AKI] was identified for the use of Nephrocheck in the cardiac surgery setting using urine samples.⁵⁴ A summary of the baseline characteristics and test parameters for the included study is shown in Table 14. The outcome used to define a diagnosis of AKI was a RIFLE classification of ≥ R within 72 hours of surgery. The threshold used to define a positive test was whether the maximum (TIMP-2) × (IGFBP-7) value in the 24 hours post cardiopulmonary bypass (CPB) was > 0.3. A diagnostic accuracy summary for the included study is shown in Table 15. No meta-analysis was performed for this single study.

TABLE 14

Characteristics of the study included for Nephrocheck in the cardiac surgery health-care setting using urine

TABLE 15

Diagnostic accuracy summary for the study included for Nephrocheck in the cardiac surgery health-care setting using urine

Neutrophil gelatinase-associated lipocalin

Critical care unit: plasma

Summaries of the baseline characteristics and test parameters for the included studies are shown in Table 16. Eight studies were included,³⁵^,⁴³^,⁴⁸^,⁴⁹^,⁵¹^,⁵⁹^,⁶⁶^,¹⁰⁰ with a total of 1670 patients [381 patients (22.8%) with a diagnosis of AKI and 1289 patients (77.2%) without a diagnosis of AKI]. The outcome used to define the presence of AKI was not consistent across the studies, with six studies³⁵^,⁴³^,⁴⁹^,⁵⁹^,⁶⁶^,¹⁰⁰ using a diagnostic outcome and two studies⁴⁸^,⁵¹ using a failure-type outcome. There was also heterogeneity in the time at which the outcome assessment occurred (unclear, 48 hours and 7 days). Similarly, the threshold used to define a positive test was not consistent across the studies, ranging from 242 pg/ml⁴⁹ to 558 ng/ml.⁴³ Diagnostic accuracy summaries for the included studies are provided in Table 17.

TABLE 16

Characteristics of studies included in meta-analysis for NGAL in the critical care unit health-care setting using plasma

TABLE 17

Diagnostic accuracy summaries for studies included in meta-analysis for NGAL in the critical care unit health-care setting using plasma

Figure 5 shows point estimates of the sensitivity and specificity from individual studies and the pooled estimates for NGAL in the critical care unit using patient plasma samples. The pooled sensitivity estimate was 0.72 (95% CI 0.65 to 0.79) and the pooled specificity estimate was 0.81 (95% CI 0.75 to 0.86). Figure 6 shows an estimate of the SROC curve with the 95% confidence region and 95% prediction region. The prediction interval shows that there is a degree of heterogeneity in sensitivity and specificity, probably reflecting the variability in outcome measures and cut-off points, as well as other unidentified sources of heterogeneity.

FIGURE 5

Forest plot for studies included in the meta-analysis and pooled estimates for NGAL in the critical care unit health-care setting using plasma.

FIGURE 6

Summary receiver operating characteristic curve for studies included in the meta-analysis for NGAL in the critical care unit health-care setting using plasma.

Critical care unit: serum

One study including 150 patients [43 patients (28.7%) with a diagnosis of AKI and 107 patients (71.3%) without a diagnosis of AKI] was identified for NGAL in the critical care unit setting using serum.³⁴ A summary of the baseline characteristics and test parameters for the included study is provided in Table 18. A diagnostic-type outcome was used to define the presence of AKI (AKIN stage 1 or above within 48 hours of admission). The threshold used to define a positive test was 110 ng/ml. A diagnostic accuracy summary for the included study is provided in Table 19. No meta-analysis was performed for this single study.

TABLE 18

Characteristics of the study included for NGAL in the critical care unit health-care setting using serum

TABLE 19

Diagnostic accuracy summary for the study included for NGAL in the critical care unit health-care setting using serum