Executive summary

Searching, screening and data extraction

Eligible studies were identified by searches of 11 electronic bibliographic databases, trial registers for ongoing and recently completed trials, inspection of the reference lists of all included studies and previously published reviews, and by contact with authors of included studies. There were no restrictions by language. Searches covered the period January 1990 to November 2009.

Two review authors independently examined the titles, abstracts and keywords of electronic records according to the eligibility criteria above. The results of this initial screening were cross-referenced between the two review authors and full-text records obtained for all potentially relevant reports of trials. These potentially eligible trials went through a secondary screening by each reviewer using a screening form based on the eligibility criteria for final inclusion in the systematic review, with disagreements resolved by discussion with a third author.

Two review authors extracted relevant data into a Microsoft Access 2007 database (Microsoft Corporation, Redmond, WA, USA) specifically designed for the review. Two measures of methodological quality were used in the review: the Cochrane Collaboration's risk of bias assessment and the Effective Public Health Practice Project (EPHPP) quality assessment. The dietary assessment tools and techniques used to estimate dietary behaviour were critically examined in terms of quality.

Analysis of effectiveness

For studies that reported the same outcome, we pooled the results using a random effects model, with weighted mean differences (WMDs), and calculated 95% confidence intervals (CIs) and two-sided p-values for each outcome. When outcomes were assessed more than once during follow-up, the final assessment was used in analysis. We assessed evidence for selection bias using Egger's test for small study effects. Heterogeneity among the trial results was assessed using both a chi-squared test and the I² statistic (we considered I² > 50% to reflect 'substantial heterogeneity'). We conducted sensitivity analyses to investigate possible sources of heterogeneity including study quality and sociodemographic factors that could act as effect modifiers. Causes of heterogeneity and subgroup effects were assessed using random effects meta-analysis. This was implemented in Stata (StataCorp LP, College Station, TX, USA) using the 'metareg' command and including trial characteristics as covariates. All statistical analysis was conducted using Stata statistical software version 11.

Economic evaluation

The intention was to estimate cost-effectiveness in two ways: through a systematic literature review and by building a de novo decision model to assess the cost-effectiveness of a 'generic' e-learning device compared with dietary advice delivered by a health-care professional. However, no perfectly fitting published economic evaluations were identified; thus, the results from the literature review were instead primarily used to inform the model design. The key assumption within the model was that the interventions were designed to promote weight loss in already obese people with a body mass index (BMI) of ≥ 30 kg/m². The evaluation was performed from a UK NHS cost perspective and outcomes were expressed as quality-adjusted life-years (QALYs). The estimate of relative treatment effect was derived from the systematic review of effectiveness, and the model was based on discrete event simulation techniques referred to as the 'E-Learning Economic Evaluation Model' (E-LEEM).

Description of included studies

Of the 43 included studies, one was a crossover trial and two were cluster-randomised trials, whereas the rest were parallel-group RCTs. The majority of interventions evaluated sought to reduce fat intake (28 interventions) and/or to increase fruit and vegetable intake (21 interventions). Other interventions sought to increase fibre intake, reduce overall energy intake, reduce or maintain weight, or reduce or maintain BMI.

All e-learning interventions were delivered in high-income countries: the majority were delivered in the USA (29 interventions), the Netherlands (five studies) and Belgium (three studies). Many interventions were offered over the internet or via a mobile device and without a specific setting. Of those that were delivered in a specific setting, nine were designed to be delivered in the home, eight in the workplace, three in community centres, four in schools/colleges and two in supermarkets. Many study reports did not provide information on the intended intensity or frequency of use; of those that did, intended intensity of interventions varied between a single session to weekly interaction with the intervention over 1 year.

A majority of interventions involved the presentation of nutrition, health and lifestyle information (31 studies), the entering of food consumption data (28 studies), and the presentation of personalised feedback on food and nutrient consumption (21 studies). Other interventions focused on the setting of goals and providing feedback on attainment of these goals (15 studies). The most commonly used behavioural change techniques reported were goal setting (behaviour) (14 studies); provide feedback on performance (14 studies); provide information on consequences of behaviour in general (14 studies) or to the individual (11 studies); barrier identification/problem solving (13 studies); prompt self-monitoring of behaviour (12 studies); provide instruction on how to perform the behaviour (12 studies); prompt review of behavioural goals (11 studies); and plan social support/social change (10 studies).

Of the 23 studies providing details of inclusion criteria, 11 offered the intervention only to those with BMI > 25 kg/m²; 10 required participants to be generally healthy and free of diagnosed disease; one trialled the intervention in patients with diabetes; and one required participants to have at least one diagnosed risk factor for cardiovascular disease. Three studies included adolescents aged < 18 years. Of studies targeting adults, the majority included participants with an average age of 40–49 years. One study included adults with an average age > 60 years.

Twenty-two studies measured total fat intake and eight studies measured saturated fat intake (either as grams consumed or as percentage of total energy consumed). Twenty-two studies assessed servings of fruit and/or vegetables consumed per day. Other food and nutrient outcomes measured included energy intake (nine studies) and fibre intake (five studies). In terms of clinical outcomes, 11 studies measured BMI, 14 studies measured weight or weight loss, four studies measured cholesterol, four studies assessed triglycerides and three studies assessed blood pressure. Four studies concentrated on the avoidance of eating disorders and used scales of eating disorder risk alongside measures of BMI and/or weight.

Methodological quality

The Cochrane assessment was at 'low risk of bias' in 27 studies for methods of sequence generation; 18 studies for methods of allocation concealment; 15 studies for methods of blinding; 22 studies for addressing incomplete outcome data; 11 studies for not demonstrating selective outcome reporting; and 26 studies for providing a conflict of interest statement. The EPHPP assessment was 'strong' in one study for selection bias criteria; 43 studies for study design; 36 studies for adjustment for confounders; five studies for criteria on blinding; 21 studies for data collection methods; and 24 studies for attrition criteria. Thirteen studies were rated overall as 'moderate' and 30 studies were rated overall as 'weak'; none was rated 'strong' overall.

Studies using 24-hour recall (two studies), or a combination of two or more dietary assessment methods (11 studies), especially two independent dietary assessment methods (two studies), had the strongest dietary assessment methods for measuring dietary change. Those studies that used a prospective dietary assessment method (e.g. multiple day diet record) alone are valid for measuring dietary change, as long as a compliance bias did not result in an overestimation of the intervention effect and dietary practices were not simplified to reduce respondent burden. The studies with the weakest dietary assessment methods were those using only shopping receipts (two studies), a screening Food Frequency Questionnaire (three studies) or a one- or two-item food group question (five studies). Most studies did not adequately describe their dietary assessment techniques and seven questionnaires were not validated for the target population or nutrient of interest.

Analysis of effectiveness

There was substantial heterogeneity in the estimates of effect of e-learning interventions on many reported outcomes; however, there was relatively little evidence for heterogeneity in the estimates of effect on total fat intake per day, total energy intake per day and BMI. When studies reporting the same outcomes were pooled in a random effects meta-analysis, e-learning interventions were associated with a WMD of 0.24 servings (95% CI 0.04 servings to 0.44 servings; p = 0.019) of fruit and vegetables per day (heterogeneity p < 0.001 and I² = 83%); WMD of −0.78 g (95% CI −2.5 g to 0.95 g) of total fat consumed (heterogeneity p = 0.18 and I² = 28%); WMD −0.24 g (95% CI −1.44 g to 0.96 g;p = 0.7) of saturated fat intake (heterogeneity p = 0.001 and I² = 78%); WMD of −1.4% (95% CI −2.5% to −0.3%; p = 0.012) of total energy consumed from fat (heterogeneity p < 0.001 and I² = 77%); WMD of 1.45g (95% CI −0.02g to 2.92 g; p = 0.053) of dietary fibre per day (heterogeneity p = 0.11 and I² = 60%); WMD of 4kcal (95% CI −85 kcal to 93 kcal; p = 0.93) of daily energy intake (heterogeneity p = 0.33 and I² = 13%); WMD of −0.1 kg/m² (95% CI −0.7 kg/m² to 0.4 kg/m²; p = 0.69) in BMI (heterogeneity p = 0.92 and I² = 0%); and WMD of 0.6 kg (95% CI −3.5 kg to 4.6 kg; p = 0.78) of weight (heterogeneity p < 0.001 and I² = 86%).

Economic evaluation

The literature review did not reveal any published economic evaluations of e-learning devices that were based purely on imparting nutritional advice. One published UK study that evaluated a device in conjunction with physical exercise suggested that it was not cost-effective compared with standard methods of providing advice. The base-case results from the E-LEEM model suggested that the incremental cost-effectiveness ratio was approximately £60,000 per QALY. The results were generally robust to most alternative assumptions, except the initial fixed cost of the device. In the base case this cost was assumed to be £854 per person. When the cost was assumed to be £0, the e-learning device was less costly and more effective than the alternative. Expected value of perfect information (EVPI) analysis showed that although the individual-level EVPI was arguably negligible, the population-level value was between £37M and £170M at a willingness to pay of £20,000–30,000 per additional QALY.

Explanations of effectiveness

We analysed the data with a view to determining reasons for the observed variability in effectiveness of e-learning interventions. We considered that potential reasons could include differences in:

• target populations
• target behaviours
• intervention content
• theoretical base
• mode of delivery
• 'dose' of intervention
• study quality.

Interventions trialled were so heterogeneous that no firm conclusions could be reached. More mechanistic research is required to address these questions, including whether or not adaptive e-learning tools are better suited for disease management than for health promotion; understanding the psychological mechanisms of action of interventions; and whether or not there is a relationship between 'dose' of the intervention and effectiveness. Our statistical assessment of the evidence suggests that trials with lower methodological quality may overestimate effects.

Can e-learning interventions change dietary behaviour?

There are many factors affecting what foods people eat and why, and an intervention targeted at individual behaviour change can address only a selection of these. E-learning has not yet proved itself to be more effective or cost-effective than other behaviour change approaches at an individual level for improving diet, or for preventing or reducing overweight or obesity. Nor is there any research comparing e-learning approaches with population-level approaches to tackling dietary change or reducing obesity.

Are e-learning interventions cost-effective to the NHS?

The results broadly suggest that the e-learning devices are not cost-effective at conventional levels of £20,000–30,000 per QALY gained and the probability that they are cost-effective was 25% at these threshold levels, a result that is perhaps not unsurprising given the relatively modest effects derived from the systematic review of effectiveness. However, the results were sensitive to the assumptions regarding the initial fixed cost of the devices; lowering this value dramatically increased the cost-effectiveness of the devices.

What is the potential population health impact of e-learning interventions?

Implications for health care

The current clinical and economic evidence base suggests that e-learning devices designed to promote dietary behaviour change will not produce clinically significant changes in dietary behaviour and are at least as expensive as other individual behaviour change interventions.

Implications for research

Although the return on investment (in terms of the expected value of the reduction in decision uncertainty) from further clinical trials of individual e-learning interventions is expected to be high, we believe that further such clinical trials should not be undertaken until theoretically informed work, which addresses the question of which characteristics of the target population, target behaviour, content and delivery of the intervention are likely to lead to positive results, is completed.