Introduction
In this chapter, we propose a sustainability and implementation plan to deploy our user-friendly web-based decision-support model at the local level.
Models can have many purposes, including understanding or refining theory, system design or visualisation, forecasting and, crucially, the exploration and comparison of contrasting future scenarios.24 Furthermore, computational modelling for decision support is not just a data and mathematical problem; it requires a closed collaboration between the commissioners, modellers, reviewers and users.181–183
Trends in evidence used in local-level public health decision-making in England showed that key aspects are the importance of local evidence for local decisions, the critical role of local expertise in providing and interpreting the evidence and placing high value on local evaluation.184 This was an issue that also came across firmly in our co-production exercises.
The workHORSE model seems to be particularly well-suited to support these types of evidence usage at the local level. These trends also highlight that it is not merely a case of having a tool available and distributed. A successful implementation will require people trained to use the tool, the resources to run and maintain it, and the skill sets to make the most of the analyses produced. In addition, the way that the tool will be implemented depends on what will be the ‘entry point’ of the tool in the organisation’s decision processes.
The workHORSE model has been developed with these fundamental principles in mind. The necessary limitations of a complex research project such as workHORSE, unfortunately, precluded a broader, national consultation that could have better informed the development of the workHORSE model and provided insights on how it could be deployed. However, the current stage of evolution of the tool can provide a solid foundation for such more comprehensive consultation and further evolution of the tool and its use.
The workHORSE model has been primarily developed for the exploration of future NCD prevention scenarios using the NHS HCP. The development phase was more prolonged than initially expected to engage as many stakeholders as practicable. However, this enabled us to accommodate as many different types of scenarios as possible. The tool is also flexible enough to allow the exploration of many NCD prevention questions at the national and local level, as well as ensure that the model does not become obsolete if the NHS HCP changes its remit, design or processes.
Key strategic implementation factors
The workHORSE model represents a different paradigm when compared with the decision-support tools generally deployed for use in LAs:
The workHORSE model can support powerful basic and advanced interface capabilities.
The workHORSE model can support scenario design features, supporting analysis of the current and alternative implementations of the NHS HCP.
The workHORSE model is provided as an open-source computer code with a permissive and copyleft licence. This feature enables external audit and quality improvement, customisation, adaptation and evolution.
However, using the tool requires skills in ‘scenario development’. Scenario development is an activity that results in the development of ‘scenario narratives’ that reflect the intervention being considered, prepares the necessary quantitative inputs to represent those interventions through changing parameters in the model and, finally, decides the scope of the analysis (i.e. in terms of time horizon and outcomes to evaluate), as illustrated in Chapter 5.
The prototype user interface described in Chapter 4 was built as an example of how scenario development can be supported by the tool. Therefore, developing more sophisticated and user-friendly interfaces will be an essential component of any implementation, as the interface will need to be able to adapt to the needs of local teams so that it can be used to represent the programme and policy to be assessed as the programme evolves.
We propose that there are five major areas to consider when strategically planning to implement the tool in an organisation: (1) the technical aspects of the implementation, (2) keeping the model updated, (3) training users in scenario development, implementation and interpretation, (4) the resources required in terms of people and expertise and (5) exploiting the possibilities of an open-source approach to future-proof the model. A summary is available in .
Summary of implementation approach: areas, role, and resources
Technical aspects and implementation feasibility
From a technical point of view, we designed and developed the workHORSE app to be easily adaptable to the available hardware. With minimal adjustments to the source code, the workHORSE model can support workstations, local network clusters or cloud computing facilities (e.g. Microsoft® Azure, Microsoft Corporation, Redmond, WA, USA). We tested the feasibility of local implementations and ‘cloud’ implementations, and we managed to run the tool both locally and in the cloud. For example, in workshop 4, we used the model in our internal local network remotely and we have also tested the model in a high-performance computer hosted at the University of Manchester, Manchester, UK. That said, the computational requirements are relatively high compared with everyday apps. The workHORSE model requires about 12GB of random-access memory per core and, ideally, more than 20 available cores per user. Nowadays, workstations that could host the workHORSE model cost around £5000; however, there is always the option of renting computational resources from a cloud computing service that is scalable and pay as you go, costing approximately £5 per hour.
Although any interested party can download the source code and run the workHORSE app, this requires the user to install all of the dependencies and resolve any potential incompatibilities. We recommend this option for advanced users and developers only. To make the installation of the workHORSE model hassle-free and adaptable, we built a Docker container (URL: www.docker.com/; Docker, Inc., Palo Alto, CA, USA). Docker is a technology that allows the containerisation of apps with all of their dependencies and an operating system. Therefore, the user can type a command into their terminal and the full app, including all of its dependencies, can be downloaded and run in an isolated sandbox using the available hardware. All main cloud computing services support Docker, which means that users can initialise virtual machines on the cloud running the workHORSE model within minutes. We provide specific instruction on how to do this for computers running Windows 10 Pro (Microsoft Corporation, Redmond, WA, USA) or Ubuntu Linux (Canonical Group Limited, London, UK) in Appendix 8. We additionally provide up-to-date detailed deployment instructions with or without Docker [URL: https://github.com/ChristK/workHORSE/blob/master/README.md (accessed 1 March 2021)].
All technologies used are open source and widely used; therefore, we do not expect licencing issues that could generate further costs for users in addition to the investment in hardware and information technology support needed. We have released the Docker container for use by any interested party at Docker Hub [URL: https://hub.docker.com/r/chriskypri/workhorse-app (accessed 1 March 2021)].
Our research project was time-limited and explicitly did not include the additional funding necessary to support the use of the tool or provide production-ready user interfaces for end-users. In fact, developing user interfaces requires a substantial investment in software engineering and user-interaction expertise. The process usually involves a design phase (during which functionality is elicited), a development phase (which usually results in a prototype and a production-level interface) and, finally, extensive testing of the user experience and interface. We ask the interested reader to contact us if they want technical details of the technologies used.
Keeping the model updated
The evidence supporting the core epidemiology in the model will need regular updating, particularly in the light of the recent changes in mortality trends in the UK population, with a likely slowdown in CVD mortality.185
Evidence on the actual parameters representing the baseline scenarios (e.g. current implementation of the NHS HCP) will require a standardisation process to ensure its consistency over time, and therefore enable the most meaningful comparisons across place and time.
Information tools such as NHS England Fingertips can provide a user with data to implement a scenario for analyses in the workHORSE model, as definitions for key performance indicators are consistent with the model scenario design parameters. However, local cost and effectiveness data, and use of local services are not immediately available. Future efforts to provide dashboards or data repositories might wish to look at existing modelling tools and to our scenario design approach parameters to produce relevant data to be used directly or with minimal end-user processing.
The evidence informing the model described in Chapter 4 and Appendix 4 might benefit from regular updates. Rapid review methods can offer an efficient way to update key parameters.186 Interestingly, given the flexibility and broad remit of a model such as workHORSE, approaches to review the different evidence needs of the programme, scenarios, epidemiology and effectiveness will require rapid, pragmatic synthesis methods more aligned with the nature of evidence needed in public health real-world decision-making.
We do not anticipate any data governance issues for basic users of the model, as equations, rather than data sets, represent most of the data. Scenario specification can be mostly carried out ‘outside’ the model, resulting in values that can be used to change the parameter sliders available in the interface. For example, the cost per invite for multiple interventions invitation strategies can be summarised in a single value. This might require obtaining local cost data to make the outputs of the model relevant to the local policy context.
The preparation of scenarios was usually carried out by our team ‘off-model’. Different data requirements and governance arrangements might be relevant if primary data are analysed to generate new, locally relevant scenarios. These issues will also need to be considered when creating or updating synthetic populations, effectiveness measures, cost data or developing more realistic inputs not contemplated by the interface.
Training future model users, including scenario development
The main training goals will be to enable a basic user to operate the beta interface competently, assuming a basic working knowledge of programme evaluation and health economics concepts. We have developed a new user tutorial as an example of the type of training materials that can be developed [see NIHR Journals Library project web page URL: www.journalslibrary.nihr.ac.uk/programmes/hta/1616501/#/ (accessed 10 March 2020)]. Mainly, this describes the primary use of the interface and is particularly suitable for smaller organisations to explore simple scenarios. The key features are to provide both visual orientations on the interface and worked examples.
However, to create a more flexible and powerful interaction with the model, the interface should be developed further. The user will need to interact with the ‘scripted’ code that instructs the model on how to run scenarios.
Training users in scenario development
Training is usually a key factor to improve the user experience with software.
We think that a critical skill is training users to develop and interpret scenarios. This includes understanding some theoretical principles on how the model works and what it can do, plus the user’s knowledge and expertise on evaluation. Our stakeholder group identified this issue as a critical need if the tool is to be widely adopted, alongside appropriate investment to support its use.
What is a scenario?
A scenario is the representation of the intervention that a user wishes to analyse with the model. It is not surprising that the word ‘scenario’ might need a definition when used in a modelling context and there is a substantial variation in what the definition means. At least 77 different definitions of what a scenario is have been reported in a systematic review.187
The workHORSE model is designed to support the concept of ‘scenarios’ suggested by Spaniol and Rowland.187 In summary, a scenario is a narrative of possible futures of our current situation where alternatives to the current situation can be explored and are plausible; it can be used comparatively and it allows exploratory uses as well as normative uses (e.g. making a decision based on a cost-effectiveness threshold). The work to specify the scenario is an iterative process that usually goes back and forth to the simulation to refine and test the scenario.
Scenario narratives
Scenario narratives require preparation, including data, evidence and costings for the specific interventions to be tested. The narrative explains the problem and intervention to be evaluated and the evidence base supporting it, plus the rationale for any assumptions made. These details must be thoroughly documented. However, there are no accepted formats to standardise this practice. The approach championed in foresight studies might be a practical approach to produce scenario narratives that enable better conversations when thinking of using this type of decision tool for non-normative purposes, for example when thinking about options to redesign the processes of the programme.188
Simple scenarios
Simple scenarios [such as exercise 1 in the user tutorial, see NIHR Journals Library project web page URL: www.journalslibrary.nihr.ac.uk/programmes/hta/1616501/#/ (accessed 10 March 2020)] are straightforward to implement and are based on the ‘optimisation’ of existing parameters in the interface. Basic training, as described above, should be enough to exploit a basic interface for this purpose.
More complex scenarios
More complex scenarios would require interaction with more experienced users. We have co-produced an iterative scenario development exercise with Liverpool City Council colleagues, resulting in a peer-reviewed publication.11 Bespoke solutions or implementation of additional interventions that are not already included in the current model will require more expert modelling input, or the ability to interact with the code and develop scripts or further model functionality. We encourage organisations and users with technical capabilities to build on the current version and shape the tool for specific uses, as this will benefit the entire modelling community, while enhancing the tool itself.
An important concept of exploiting the modelling approach is to understand the rich range of outputs that enable different types of strategies to be explored. For example, targeting the intervention by age, sex, ethnic group, geography or socioeconomic level allows the exploration of different types of implementation of the NHS Health Check programme. For instance, during this project, we explored how ‘universal’ compared with ‘targeted’ approaches differ in their effectiveness and equity impact for CVD prevention, using an earlier version of the engine.164 In this paper, we found that, in Liverpool, the scenarios describing the implementation of the ‘universal plus targeted’ approach dominated the scenarios ‘increased’ and ‘current’ and reduced health inequalities. This paper also illustrates the possibility of conducting distributional cost-effectiveness analysis ‘off-model’. We refer the interested reader to this publication164 to gain additional insights on how the model can support more complex analysis strategies.
We consider that training users in scenario development and interpretation is likely to be more valuable if carried out at the local level, where the problems, policy and budgetary constraints are evident, and the need for decisions can benefit most from using the tool.
What resources are needed for local implementation?
We discussed above the technical resources required and here we will discuss the skill set that we consider necessary to use the model as intended. Although we have developed the tool with computing resources that can be available in most organisations, some aspects will require initial investments and funding to sustain its use over time. We will not discuss hardware or information technology resources, nor running costs for such a programme, as we think that will require a focus on a specific implementation project. Nor can we provide support for users willing to install and run the apps.
workHORSE is not a simple and intuitive tool. The complexity arises because of the need for flexibility to explore a broad set of scenarios that are difficult to anticipate. However, this, in turn, is making the tool more ‘future-proof’ in dealing with the inevitable future changes in the design of the programme and the evidence base informing the model. Developing more intuitive interfaces for such a model will require a substantial investment in GUI design and implementation: developments that were beyond the time and resources available in the project.
Interacting with the model, procuring and preparing inputs and interpreting outputs require users to have quantitative skills to operate the model. Furthermore, the user will need to know the specific characteristics of the local population and the issues relevant to the intervention programme being assessed. It is important to reflect all of these aspects in realistic scenarios so that the insights provided by the model are relevant.
Essential resources that a user would need to secure for effective use of the tool include the following.
Information technology and software engineering
Users would require information technology and software engineering to provide support in the local deployment of the software. Existing infrastructure might support local versions of the model, but it might require adaptation. In addition, maintenance and support might be required as needed, including third-party cloud deployment, which requires expertise in the technologies used (e.g. Docker). We have tested solutions using open-source software, reducing the need for a commercial software licence if the cloud provider protocols can interoperate with workHORSE code. Most cloud infrastructures are commercial and therefore operate on a pay-per-use basis.
The prototype user interface might need adaptation for specific user requirements, changes in the programme and to enable analyses that are not possible with the current functionality.
Modelling expertise
Additional technical expertise would be needed if users wished to update synthetic populations, incorporate new effectiveness and epidemiology data, or redesign the programme. A software engineering skill set is needed, including advanced knowledge of programming in R and C++ (Standard C++ Foundation; URL: https://isocpp.org) languages and advanced quantitative skills (i.e. to implement synthetic population approaches and other advanced statistical operations; see Chapter 4).
Data science expertise
Data science expertise is needed to conduct bespoke analysis and use or update additional data sets, mostly to keep the model epidemiology and parameters up to date. Models become obsolete very quickly (e.g. the initial cost-effectiveness analysis for the NHS HCP), usually because the intervention or the programme that the model addresses is changing.
We developed the workHORSE model to be flexible and accessible so that it can evolve alongside the issues addressed. The scenario parameters reflect the basic operations of a ‘detection and control’ individual-level intervention, primarily health care based, as it involves the prescription of drugs or intervention by a qualified professional at some point.
Specific programme expertise
Programme knowledge and expertise are essential for scenario design and interpretation. The tool provides a degree of flexibility in designing scenarios that is unique, requiring a thorough knowledge of the programme to fully exploit the tool’s capabilities. As shown in Chapter 2, working closely with stakeholders involved in the NHS HCP allowed us to develop specific interface features that are most relevant for local questions and decision-making.
Implementation approaches
Depending on resource availability and the mode of use of the tool by users and organisations, the implementation can be carried out in a ‘central to periphery’ strategy or a ‘local strategy’.
The ‘central to periphery’ strategy could be a centralised effort to develop an implementation programme that is adequately resourced centrally and in charge of the technical aspects. It could provide a bespoke standardised user interface, further maintenance and development of the tool, and training and support to users. The benefits of this strategy will include economies of scale, particularly around sourcing highly skilled staff.
The ‘local strategy’ (i.e. taking responsibilities for all the roles and areas at local or even regional level) will provide more flexibility. By implementing, developing and training local users to better satisfy local needs, it will respond best to the trends in evidence use at the local level in England. However, this will require the development of specific local or regional teams with modelling and analysis capabilities, or outsourcing of these activities with a dedicated budget. Furthermore, the increasing collaborative links between universities, local public health teams and public health training schemes can provide the necessary skills and research capabilities with a robust and local focus.
We estimate that a ‘central to periphery’ strategy can be a reasonable approach to resource the necessary skills, with a core team at a central level that is well resourced in modelling and software engineering capabilities and resources to support dedicated analysts at the local level. Alternatively, the ‘local’ implementation can serve local needs best with a small team composed of an analyst (with a data science background) and funded collaborations with local universities and software engineering providers.
The workHORSE model and code can be used for any of these levels of implementation.
Developing the model: exploiting the possibilities of an open-source approach
A Royal Society Open Science review on modelling has recommended open-source/open-access approaches to models used in policy decision-making.181 Furthermore, the original NIHR call for this project indicated the need for an open-source tool. This approach has also been increasingly suggested to future-proof further enhancing transparency and providing commissioners and users with a starting platform to develop models more relevant to their users’ needs.
We have therefore licensed the code under General Public License (GPL) v3. GPL v3 is a widely used licence that guarantees and enables end-users and developers to use the software, share it or further modify and develop it to suit their own needs. It is a form of ‘copyleft’ licence, which encourages the evolution of this work by ensuring that all derivative work should be open source and distributed under the same or equivalent licence terms. As an illustration, any user can adapt or extend the code for other purposes. For instance, a user can be interested in exploring alternative designs for NHS Health Checks or when new evidence on novel interventions need to be evaluated and considered for inclusion. Therefore, ‘tinkering’ with the code is entirely allowable with the open-source licence used and encourages the academic and non-academic communities to use and expand the model and improve its methods, as long as it remains open source and under the GPL v3.
The open-source approach allows better integration with data sources and evolving linked data sets. Efforts in PHE and the increasing availability and access to linked data sets can result in better and more efficient ways to update the data and evidence used in the model.
A key aspect is that enhanced transparency allows detailed inspections of the code and the equations in the model, although it requires advanced coding expertise to judge it adequately. Therefore, even an open-source approach does not ensure full transparency of the modelling approach, an issue that merits further thinking on how to increase the confidence of end-users to modelling activities in general. Our stakeholders signalled this (see Chapter 2) and therefore more research is needed. The research should focus on finding ways of communicating the workings of these complex mathematical models in simpler terms, while preserving enough detail to judge their internal and external validity.
Finally, the adoption of open-source code removes the cost of commercially licenced software from the implementation strategy, while enabling commercial providers to provide specific services interacting with the model engine (e.g. the development of more advanced user interfaces). Further work might be required to facilitate this approach, including the development of ‘application programing interfaces’ (APIs). As is usual with open-source code, this software is provided without warranty and the authors cannot be held liable for any consequences arising from the use of this software. Unfortunately, we cannot support users on installing or using the model, as we are not funded to provide this activity.
The code is available in the GitHub repository (San Francisco, CA, USA) [URL: https://github.com/ChristK/workHORSE (accessed 1 March 2021)].
