This study has allowed us to examine systematically the feasibility and acceptability of the INCLUSIVE intervention for initiating change locally in bullying and aggression through the school environment. It has also enabled us to assess the feasibility of conducting a Phase III RCT and to inform decisions about outcomes to be explored in such a trial.
Progression to a Phase III trial
Our results suggest that INCLUSIVE is a feasible and acceptable intervention. All progression criteria were met despite challenges arising from the later-than-planned start date and the particularly challenging schools included in our purposive sampling frame. Therefore, in accordance with the MRC framework for evaluating complex interventions, this suggests progression to a Phase III trial is appropriate.66 A cluster RCT would establish the effectiveness and cost-effectiveness of such a whole-school restorative approach in addressing aggressive behaviours. The evidence provided by such a trial will be important in helping schools, local authorities and the NHS improve health, improve behavioural and educational outcomes for young people, and reduce health inequalities. The RQs it should address include:
Is the INCLUSIVE intervention implemented over 3 school years more effective and cost-effective than standard practice in reducing bullying and aggression in English secondary schools?
Is the INCLUSIVE intervention more effective than standard practice in improving students’ QoL, well-being, and psychological function and attainments and in reducing school exclusion and truancy, substance use, sexual risk, NHS use and police contacts among students?
Is the INCLUSIVE intervention more effective than standard practice in improving staff QoL and attendance and reducing staff ‘burn-out’?
What factors moderate and mediate the effectiveness of the INCLUSIVE intervention?
However, we do not want to underplay some of the challenges identified in planning and delivering a whole-school restorative approach, especially in very large secondary schools. Although these challenges did not impede implementation during this study, to ensure the project is delivered over a full 3-year period of implementation, refinements and additional resources may be needed so that staff training reaches the whole school in an engaging format(s), needs assessment data are accessible and curriculum materials are responsive to schools’ needs. Our recommendations focus on how we would optimise the intervention and build on, and refine, our existing trial methods to undertake a Phase III trial in 2014.
Intervention design and delivery
Whole-school approach
Launching the intervention throughout the school was identified as a way of ensuring greater engagement with the intervention across all school groups. This would be strengthened via:
devoting more resources to timely launch events, targeting both students and staff
a greater web presence, including interactive online content to engage schools during a Phase III trial
the use of locally adaptable newsletters to inform staff and students about the activities and outputs arising from the intervention
arranging annual events to celebrate progress and achievements.
Needs assessment survey
Needs assessment data collection and tailored reports for each school were an acceptable and powerful external input that helped all the school action groups to identify priorities. This should remain integral to the intervention approach and logic model. It could be improved by:
ensuring that needs assessment data are delivered to intervention schools in a timely and accessible manner at the start of the intervention (i.e. September–October)
adopting a needs assessment approach that aims to identify the ‘positive’ features of the school environment as well as challenges and needs (i.e. an ‘assets-based’ approach)
continuing to compare each school against the average, but ensuring facilitators aid in the interpretation of these data, including through benchmarking against other schools with a similar socioeconomic intake as well as the average overall
ensuring that all reporting is accessible and student centred
using annual surveys in intervention schools to monitor progress and identify new/ongoing priorities.
Action groups
Action groups are an innovative and powerful mechanism for supporting student-led change to address key school-level risk and protective factors for aggressive behaviour. The action groups could be improved via:
recruiting students from a mixture of years (e.g. years 7–9) into the action group and, when necessary, inviting particular students to participate in order to ensure a diverse group
ensuring that the head or deputy head teacher is a member of the group to make sure it has sufficient power to change school policies
external facilitators working with the action group co-ordinator to identify the best time(s) for meetings locally and helping them consider any practical barriers and how these might be overcome
providing what students and staff identified as a ‘grown-up’ environment for group meetings.
External facilitators
External facilitators were consistently reported to have provided a highly valuable, ‘external push’ for schools. This support should be maintained in a Phase III trial. Ideally, the existing facilitators should be retained if possible, and additional educational consultants also recruited. External facilitators will continue to be educational consultants with former school leadership experience. An intervention manager will be required, to provide training and support to ensure programme fidelity by the external facilitators, including through use of a virtual learning environment to share resources and examples of best practice online. The intervention team should be managed separately from the research team, and should be housed within an educational institution, which we envisage would be the Institute of Education at the University of London. Key roles and responsibilities for external facilitators will include:
establishing an effective ongoing working relationship with the SMT in their schools
arranging a ‘catch-up’ call with school’s intervention lead in advance of each action group meeting to support planning, allocation of tasks and administration
advocating for both students and staff to promote ‘equality of voice’ and effective decision-making involving representatives of the whole school
supporting the co-ordination of training and curriculum implementation as required at their school(s).
Staff training in restorative practices
Staff training in restorative practices was consistently identified as being a critical component in implementing a whole-school restorative approach to behaviour change. However, a number of challenges emerged in terms of delivery and staff engagement. In a future trial, it would be essential to ensure trainers are aware of each school’s particular context, and improve the timeliness and reach of training. Therefore, the external facilitators would themselves be trained to provide training in the schools with which they work. Training would also be improved via:
ensuring that training is undertaken at the start of the school year to pump-prime other activities and increase awareness of the intervention across the school
a comprehensive pre-training audit to identify schools’ needs, what they hoped to achieve and the most appropriate method to ‘cascade’ learning through the whole school
more engaging, interactive training methods using ‘realistic’ examples from similar secondary schools
ensuring that students from the action team attend the training, and they are included and engaged in it.
Social and emotional skills curriculum
The student social and emotional skills curriculum was consistently identified as being a valuable and flexible component. In our pilot trial, as a result of the time needed for curriculum planning, this was delivered only in the third (summer) term. The curriculum could be refined further via:
New intervention partnerships
New intervention partnerships should also be developed with the Department of Education and Ofsted to maximise synergy with the broader policy environment and assessment frameworks within which English secondary schools operate. Similarly, consultations with further public health stakeholders in England should take place to explore how this intervention can be integrated and mainstreamed with ongoing policy programmes that aim to increase access to psychological therapies and support the social and emotional aspects of learning more strategically.
Trial design and methods
The methods were feasible and acceptable, with all schools remaining in the study and extremely high student response rates (> 93% at baseline and follow-up), but further refinements are, nonetheless, suggested based on the learning from the pilot.
Primary outcomes
The primary outcomes investigated within a Phase III trial should include one measure of bullying victimisation and one measure of the perpetration of aggressive behaviours. Our GBS and ESYTC measures performed satisfactorily and should therefore constitute these two outcomes, respectively. The trial should be powered on the basis of these two outcomes.
Secondary outcomes
The secondary outcomes investigated in a Phase III trial would include all those hypothesised for the pilot as well as validated measures of drug use and sexual risk behaviour (age at sexual debut; and contraception use at last sex). We will also seek to measure educational attainment, as the intervention is hypothesised to have demonstrable effects in this area, and this would probably be powerful evidence in enabling the scaling up of the intervention.
Sample of schools
The sample of schools in a Phase III trial should continue to be diverse but should reflect the overall population profile of schools in the study area (south-east England), rather than aiming to oversample particularly challenging schools (as the pilot did to ensure a diverse range of contexts were included). This pilot trial was initiated in July 2011, 3 months after originally planned, which seriously impeded our ability to recruit schools, although this was nonetheless completed.
Recruitment of schools for a Phase III trial will require a longer lead-in period. We recommend that:
the project be initiated in February to enable liaison with schools to proceed for 4–6 months before the summer holidays
the trial team partners with existing networks of schools such as the Institute of Education’s ‘Teaching Schools’ network and other school practice networks, such as UCL Partners schools network and ‘Challenge Partners’ (
www.challengepartners.org)
comparison schools continue to be offered £500 in total to cover the expenses for data collection as well as a report of information from baseline and follow-up surveys (once the trial has been completed).
Surveys
Surveys (of students and teachers) should be conducted in the summer term each year, with baseline surveys undertaken in the summer term prior to the school year in which the intervention occurs. Additionally we recommend that:
information is provided to students and staff in more accessible language, including about how anonymity is maintained within the trial
a Phase III trial should include baseline and follow-up surveys with teachers but also with teaching assistants and other school staff
staff surveys should also be conducted in the summer term prior to the school year in which the intervention occurs and prior to randomisation
baseline and follow-up surveys should be undertaken with the action team, including items on students’ social and educational characteristics.
Although this study found pen-and-paper questionnaires were feasible, were acceptable and produced a high response rate among students and staff, and some schools suggested that using school information technology facilities to undertake an online survey would not be feasible, we recommend that the cost of new technologies (e.g. small, touch-screen ‘tablet’ devices with 4G web capability) is reviewed prior to a Phase III trial. Although there is an up-front cost in purchasing such equipment, such online survey methods may improve data quality significantly and deliver savings overall, as a result of minimising any costs associated with printing questionnaires and inputting, checking and archiving data. In addition to reviewing the economic costs and benefits, we would also consult the NCB YRG and teachers to get their views on this method, and on the risks and benefits of online surveys in schools, prior to making a decision.
Quantitative data on intervention fidelity
Quantitative data on intervention fidelity should be collected in a Phase III trial. In addition to provider checklists and the documentary evidence that was collected in this pilot trial, the following data should also be collected:
structured independent assessments of intervention delivery drawing on audio recordings and observations of a sample of action team meetings, training sessions and curriculum sessions
information on the professional background and other characteristics of each action group facilitator in order to analyse how implementation and/or effects may vary
measures of coverage regarding all relevant intervention inputs and outputs and, when relevant, including baseline assessment.
Qualitative data
Qualitative data should continue to be collected in a main trial as part of its integral process evaluation in order to assess unexpected processes, explore causal pathways and assess variation in implementation by context.
Economic evaluation
Economic evaluation will be a core element of a Phase III cluster RCT. Our pilot economic analyses support the use of the CHU-9D scale with this population and the feasibility of CUA, although this should be supplemented with a CCA. However, at present, we do not believe that undertaking (complex) modelling to link observable trial outcomes to longer-term (health) events is warranted, given the inherent limitations with the existing evidence base. Such an exercise is likely to produce cost-effectiveness estimates that are so uncertain as to be of little practical use. Anonymised data linkage may also support longer-term cost-effectiveness analyses via routinely collected health-service, education and criminal justice system data. The potential for collecting unique pupil numbers from schools at baseline, which could be linked to national health, education and crime databases, should be explored prior to a Phase III trial in order to facilitate long-term data linkage if possible. These findings suggest that student participation, particularly ‘having a say’ in revising the school rules, is a core component of the intervention, and therefore our economic analyses within a Phase III trial would also estimate these ongoing costs in order to facilitate potential longer-term analysis via data linkage.
Scalability and generalisability
A key criterion for assessing the importance of public health interventions is their potential scaleability.101 This must be considered in all phases of evaluation66,102 and needs to be a key focus when planning a Phase III trial of INCLUSIVE. The INCLUSIVE intervention is potentially scalable because of:
its clear feasibility and acceptability in English schools
its focus on schools’ ‘core business’ of teaching, discipline and pastoral care, and its perceived potential by school managers to contribute towards schools’ government-required mission of increasing educational attainment and improving behaviour
its strong balance of fidelity to intervention processes and core components with adaption of non-core outputs to local needs and existing policies and practices.
71,72
Furthermore, the intervention has the potential to achieve a wider range of health benefits and reductions in health inequalities beyond reductions in student bullying and aggression. This is because the intervention aims to address a variety of school-level and individual-level risk factors for a range of intercorrelated health behaviours103–105 that affect health across the life-course.58 The intended secondary health outcomes at this pilot stage included improvements in students’ QoL and psychological well-being, as well as reductions in psychological distress and substance use. In addition to these, we intend a Phase III trial to examine intervention effects on drug use and sexual risk behaviour. This plurality of intended health benefits also contributes to the potential scalability of the intervention, in contrast to numerous single-focus curriculum interventions addressing each of these outcomes separately, which are unlikely to find a place in busy school timetables.
The potential for scale-up will be a key focus of the process evaluation within a Phase III trial. As well as working with existing school networks, we will explore with them issues to consider in scale up and the most appropriate ways to proceed to this should the intervention be determined to be effective. A key factor to consider is the appropriate provider. In the Phase III trial, the intervention facilitators will be based at the Institute of Education working in collaboration with the various networks of secondary schools discussed above, such as the Institute of Education’s ‘Teaching Schools’ network and ‘Challenge Partners’. This is appropriate in that it allows us both to recruit schools more efficiently and to manage the intervention carefully in order to maximise fidelity. It is also appropriate because the Institute of Education has a long-standing role in capacity building and professional development from working with a variety of networks of secondary schools. These arrangements are therefore promising ones for the delivery of a scaled-up intervention, should it be demonstrated as effective in a Phase III trial.
