Criteria for Inclusion/Exclusion of Studies in the Review

We included FeNO studies that enrolled patients with suspected asthma (KQ 1.a and KQ 1.e) or confirmed asthma (KQ 1.b–d) who were 5 years of age or older (except KQ 1.e; in which patients were 4 years or younger at the time of FeNO testing). Studies had to evaluate FeNO diagnostic accuracy or clinical utility according to PICOTS (Table 1) and Key Questions (KQs). Both randomized and nonrandomized studies were included for all KQs. We included longitudinal, cross sectional, and case control studies. Uncontrolled case series were included only if they reported adverse effects of FeNO testing.

We excluded studies that did not fit the PICOTS or those with mixed population (e.g. asthma and chronic obstructive lung disease) without reporting separate results for individuals with asthma. We also excluded surveys, narrative reviews, editorials, letters, or erratum, qualitative research, in vitro studies, and animal studies.

Literature Search Strategies

We conducted a comprehensive literature search of six databases. Specifically, they were Ovid MEDLINE® In-Process & Other Non-Indexed Citations, Ovid MEDLINE®, EMBASE, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, and SciVerse Scopus from the inception of the databases inception to April 4, 2017. A medical librarian developed and executed the search strategy (Appendix A). We used a web-based systematic review software, DistillerSR (Evidence Partners Incorporated, Ottawa, Canada), to facilitate study selection.

We searched relevant systematic reviews and conducted reference mining of relevant publications to identify additional literature. We searched gray literature through all of the following: U.S. Food and Drug Administration (FDA) device registration studies, ClinicalTrials.gov, Health Canada, Medicines and Healthcare Products Regulatory Agency (MHRA), AHRQ’s Horizon Scanning System, conference proceedings, patient advocate group websites, and medical society websites.

Independent reviewers, working in pairs, screened the titles and abstracts of all citations using pre-specified inclusion and exclusion criteria. Studies included by either reviewer were retrieved for full-text screening. Independent reviewers in pairs screened the full-text version of eligible references. Discrepancies between the reviewers were resolved through discussions and consensus. If they did not reach consensus, a third reviewer resolved the difference.

Data Abstraction and Data Management

We developed a standardized data extraction form to extract study characteristics: author, study design, inclusion and exclusion criteria, patient characteristics, interventions, comparisons, outcomes, and related items for assessing study quality and applicability. All study team members pilot-tested the standardized form using 10 randomly selected studies and iteratively modified it as needed. Single reviewers extracted data with a second reviewer verifying all entries. We noted whether FeNO measurement was done online (i.e., real-time gas analysis) or offline (exhaled gas is collected during tidal breathing into impermeable bag for subsequent analysis).

Assessment of Methodological Risk of Bias of Individual Studies

We evaluated the risk of bias of each included study using predefined criteria. For RCTs we used the Cochrane Risk of Bias tool to assess sequence generation; allocation concealment; participant, personnel, and outcome assessor blinding; attrition bias; incomplete outcome data; selective outcome reporting; and other sources of bias.²¹ For observational studies, we used items derived from the New Castle Ottawa scale.²² For diagnostic studies, we used the QUADAS-2 instrument.²³

Data Synthesis

We narratively summarized the key features and characteristics (e.g., study populations, design, intervention, outcomes, and conclusions) of the included studies and present in evidence tables for each KQs.

For diagnostic questions, we used the symmetric hierarchical summary receiver operating characteristic (HSROC) models to jointly estimate sensitivity and specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR−), and diagnostic odds ratio (DOR).²⁴ DOR is a single indicator of diagnostic performance that facilitates comparison across tests. It was defined as the ratio of the odds of positivity in subjects with disease relative to the odds in subjects without disease and is calculated as (true positives × true negatives)/(false positives × false negatives). ²⁵ We also drew the HSROC curves based on the estimates. For clinical utility and harm questions, we used the DerSimonian-Laird random effects model with the Knapp and Hartung adjustment of the variance.²⁶ We evaluated heterogeneity between studies using the I² indicator; we examined potential publication bias by evaluating funnel plots symmetry and Deeks’ funnel plot asymmetry tests if the number of studies was large (n>20).

To explore heterogeneity, we conducted subgroup analyses based on factors defined a priori:

• Robustness of “reference test” used in the literature
• Test cutoff values
• Risk of bias
• Control group description
• Tobacco use
• Asthma phenotype (eosinophilic, neutrophilic, paucicellular) or atopy status
• Use of inhaled/oral corticosteroids prior to FeNO testing
• Whether appropriate testing protocol was followed (alcohol consumption, fasting state or food intake, prior use of mouthwash)
• Body mass index (BMI) or weight
• Manufacturer and device model (chemiluminescence, electrochemical methods)
• Exhalation flow rate
• Age (ages 0–4, 5–11, 12 and above).

Grading the Strength of Evidence for Major Comparisons and Outcomes

We graded the body of evidence as per the EPC Methods Guide on Comparative Effectiveness Reviews on assessing the strength of evidence (SOE). We focused on the diagnostic accuracy measures, asthma control composite scores, exacerbations, and asthma- specific quality of life.²⁰ These outcomes are chosen because they are either clinically important from a patient or other stakeholder perspective or highly relevant for decision making (diagnostic accuracy measures).²⁷ Grading the SOE was done for each comparison and for each outcome.

For outcomes of efficacy and clinical utility, randomized trials start as high strength of evidence and observational studies start as low strength of evidence. The domains considered were: the methodological limitations of the studies (i.e., risk of bias); precision (based on the size of the body of evidence, number of events, and confidence intervals); directness of the evidence to the KQs (focusing on whether the outcomes were important to patients vs. surrogate outcomes); consistency of results (based on qualitative and statistical approaches to evaluate for heterogeneity); and the likelihood of publication bias. When imprecision was associated with a very small sample size (less than an arbitrarily chosen cutoff of 400) or with a wide confidence interval that includes no effect and a relative risk reduction that exceeds 25 percent, we rated down SOE two levels and labeled this as severe imprecision.

In diagnostic studies, observational studies can start as high SOE for diagnostic accuracy outcomes. SOE rating can be rated down primarily because of methodological limitations of the studies, lack of precision, and likelihood of publication bias. We did not rate down for statistical heterogeneity (which is always high in diagnostic meta-analyses) or consider diagnostic accuracy measures as surrogate outcomes.^{28, 29}

When studies were heterogeneous in population, intervention and methods; and not appropriate for meta-analysis, we have narratively provided a summary statement about the findings and conveyed our certainty in such findings as a SOE rating.^30–32 In this case and in the absecnce of a single pooled estimate of the effect size, we narratively rated the SOE considering the meaning and connotation of SOE domains (methodological limitations of the studies, precision, directness, consistency and the likelihood of publication bias).^{30, 32}

Based on this assessment and the initial study design, we assigned SOE rating as high, moderate, low, or ‘insufficient evidence to estimate an effect’.

Assessing Applicability

We followed the procedures outlined in the EPC Methods Guide for Comparative Effectiveness Reviews to assess the applicability of the findings within and across studies.²⁰ We determined the applicability for each outcome qualitatively using the PICOTS framework. We focused on whether the populations, interventions, and comparisons in existing studies are representative of current practice.

Peer Review and Public Commentary

A draft version of the draft report was posted for peer review and for public comments in April, 2017, and revised in response to comments. However, the findings and conclusions are those of the authors, who are responsible for the contents of the report.