The impact of the National Institute for Health Research Health Technology Assessment programme, 2003 13: a multimethod evaluation

Aims

Recent NIHR briefing documents state the aim of the HTA programme as to ‘research information about the effectiveness, costs and broader impact of health-care treatments and tests for those who plan, provide or receive care in the NHS’ (p. 2),9,10 demonstrating that although the specific strategy of the programme has changed over time, the overarching aims of the programme have remained consistent.

Research funding

The HTA programme commissions and funds research via four routes:

• Commissioned workstream This stream funds research on specific topics identified by a range of stakeholders, from patients to professional bodies, and prioritised by the six advisory panels (described below). Typically, the HTA programme advertises three calls per year, which consist of a set of specified research questions that are described in commissioning briefs. Responses from applicants to each commissioning brief are reviewed based on their scientific merit, feasibility and value for money by the HTA Commissioning Board.

• Researcher-led workstream This stream funds research questions put forward by researchers. The Clinical Evaluation and Trials Board assesses proposals based on their relevance to clinical practice in the NHS and the importance of the outcomes to patients.

• Themed calls Themed calls aim to increase the evidence base for key health priorities through funding a number of projects across the NIHR in a particular area. They are evaluated separately by independent review boards set up for that purpose. Previous calls include obesity, medicines for children, diagnostic tests, dementia and primary care.

• Technology Assessment Reports (TARs) These provide evidence to support NICE’s technology appraisal and diagnostic assessment programmes (TARs are described in detail below).

Six advisory panels and the prioritisation group support the review boards across all of these programmes. The prioritisation group balances the relative priority of research topics for the commissioned research workstream and applications received from the researcher-led funding stream. The six advisory panels are as follows: Diagnostic Technologies & Screening Programmes; Elective and Emergency Specialist Care; Interventional Procedures; Maternal, Neonatal and Child Health; Mental, Psychological and Occupational Health; Primary Care, Community and Preventive Interventions. The role and remit of each of these advisory panels is described in more detail below (see Priority setting).

The programme supports both evidence syntheses and primary research across all six panels and across all the funding streams. Over the last 10 years, there has been a relative increase in both response mode-funded research and primary research, which is in line with the recommendations set out in Best Research for Best Health3 and the Cooksey review7 (see Policy developments affecting the Health Technology Assessment programme from 2003 to 2013, above). In 2001, the HTA programme had published only five clinical trials,11 whereas by April 2014 the programme had supported 530 primary research studies, 260 of them completed and published in the HTA journal. An overview of the HTA programme’s portfolio, by type of study and status of the project, is provided in Table 1.

Relationship with policy-making organisations

An important part of the HTA programme’s strategy is the way it engages and influences NICE, the National Screening Committee (NSC) and other policy-makers. The HTA programme has direct links with NICE through its commissioning of TARs to inform NICE guidance. However, HTA programme-funded research also feeds in to other aspects of NICE’s work. For example, HTA programme-funded research is often cited in NICE guidelines, but the links between the HTA programme and the guideline-producing parts of NICE are less strong than the links between the TAR programme and NICE. The HTA programme also has direct links with the NSC, which are less formalised than the programme’s links with NICE. In addition to its well-established links with NICE and the NSC, the HTA programme also has informal links with other policy-makers (e.g. NHS England). The remainder of this subsection discusses the links between the HTA programme and each of those organisations.

The HTA programme commissions independent TARs to meet the urgent needs of NICE and other users of TARs. 12 Nine TAR centres, based in universities and academic centres across the country and contracted by the Department of Health (DH), conduct all of the TARs for NICE. TARs provide evidence to support NICE’s technology appraisal and diagnostic assessment programmes. There are three types of TARs:

1. Single Technology Appraisal Reports (STAs) aim to ‘assess the strength and quality of the research evidence submitted by manufacturers to NICE as part of the evaluation of a single new drug or device close to when they are first licensed’ (p. 3)12 and are produced within 8 weeks.

2. Multiple Technology Appraisal Reports (MTAs) aim to ‘identify, assess and synthesise the research evidence (including data submissions) from across a number of interventions in a given healthcare area’ (p. 3). 12 MTAs typically provide estimates of the relative effectiveness and cost-effectiveness of different interventions. They are larger research reviews than the STAs and take 26–28 weeks to produce.

3. Diagnostic Assessment Reports aim to ‘identify, assess clinical outcomes and synthesise the research evidence for single or multiple diagnostic technologies in a given pathway’ (p. 3). 12 Diagnostic Assessment Reports typically provide estimates of the relative effectiveness and cost-effectiveness of different diagnostic technologies. Diagnostic Assessment Reports are also relatively large reviews and take 24 weeks to produce.

Recent users of TARs also include the Chief Medical Officer, the National Specialised Commissioning Team, and the Policy Research Programme. TARs produced for other policy-makers may take different forms, and their content is tailored to meet the needs of the particular policy-making organisation. However, they typically include a systematic review of evidence in a particular area and economic modelling.

The NICE Guidelines Programme also uses HTA programme-funded research, but the link between the policy impact and the HTA programme is less direct. The HTA programme also has direct links with the NSC, which relies heavily on HTA programme-funded research for formulating evidence-based advice for government. According to interviewees, the HTA programme sends research related to screening directly to the NSC. The HTA programme and the NSC also have regular meetings to discuss both ongoing and recently published HTA programme-funded research related to screening. The NSC also submits research topics directly to the HTA programme. However, the links between the HTA programme and the NSC seem to rely primarily on relationships between individuals rather than formal links between the two organisations.

Research budget

‘The research budget of the HTA programme has now grown considerably from the initial £1M in 1993/4 to £8M in 2003/4 and £60M in 2012/13’ [NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC), 3 April 2014, personal communication, reproduced with permission]. Adjusting for inflation, that growth is equivalent to an increase of more than 400%, in real terms, over the last 10 years. This funding increase reflects the growth in scope of the programme over that time period, particularly the expansion of the clinical trials element of the programme because clinical trials are typically much more costly than evidence syntheses.

Priority setting

The HTA programme identifies potential research topics from a number of sources. It engages key stakeholders in the NHS and the NIHR, in part through the NIHR horizon scanning centre. The programme also sources research topics from the James Lind Alliance (JLA) Priority Setting Partnerships and from research recommendations in published research, particularly systematic reviews, and guidelines. The programme also elicits research suggestions from policy-makers (notably NICE and the NSC) and other health-care-related organisations (e.g. the Royal Colleges and patient groups). Research topics can also be submitted by members of the public through the HTA website.

The initial prioritisation process consists of a number of steps. First, potential research topics are reviewed to assess whether or not they fall within the remit of the programme and to determine whether or not the research questions are sufficiently different from existing or ongoing research. Research topics that do not fall within the remit of the HTA programme are passed on to other NIHR research programmes.

The remaining research topics are then reviewed by one of the six Topic Identification, Development and Evaluation (TIDE) panels. These panels comprise a range of professional, public and patient members who act to advise the HTA programme on the importance of the research topic to patients and the NHS. There are six such panels in the following areas:

• Primary Care, Community and Preventive Interventions Covers interventions that are delivered in primary care or the community.

• Elective and Emergency Specialist Care Focuses on interventions delivered in hospitals or by specialists.

• Maternal, Neonatal and Child Health Examines interventions related to obstetrics, paediatrics and specific maternal health issues.

• Interventional Procedures Covers all surgical interventions, drugs used for interventional procedures and interventional radiology.

• Mental, Psychological and Occupational Health Covers rehabilitation, learning difficulties, mental health, cognitive deficits and occupational health.

• Diagnostic Technologies & Screening Programmes Covers all tests used to diagnose, monitor or select patients for treatment; or to monitor a disease or the effect of its treatment.

The members of each panel consist primarily of clinicians and health-care professionals working in the front-line NHS but also include public and patient representatives. At panel meetings, the members discuss and refine research suggestions. The members of the panel then vote on the relative importance of the topics to the NHS. The most important topics are then developed into vignettes that guide ongoing work on the selected topics. The vignettes are discussed at the panel’s Methods Group, which provides advice on the proposed research design and methodology. After the panel finalises the vignettes, they send them to the HTA Prioritisation Group.

For researcher-led work, the TIDE panel reviews anonymised extracts from proposals against the ‘criteria to guide the setting of HTA Priorities’. They evaluate the importance of the research question and produce a ranked list of proposals for the HTA Prioritisation Group.

After the six advisory panels have completed the vignettes and the TIDE panel has reviewed researcher-led proposals, the HTA Prioritisation Group then determines which research should be funded. The HTA Prioritisation Group consists of the Programme Director of the NIHR HTA programme, the chairs from each of the six HTA panels and the two funding boards, and senior representatives from NETSCC. The role of the Prioritisation Group is to develop a portfolio of research that reflects the needs of the NHS, fits within the available programme budget and provides good value for money. The group reviews and prioritises topics from the panels based on the vignettes prepared, and decides which should be developed into commissioning briefs. The HTA Prioritisation Group also reviews researcher-led proposals and prioritises them for consideration at the Clinical Evaluations and Trials Board. Finally, the Group reviews funding recommendations from the HTA commissioning boards and prepares a final list for approval by the DH.

The Health Technology Assessment journal

The HTA programme has its own journal, Health Technology Assessment, which is part of the wider NIHR journals library. The HTA programme endeavours to publish all HTA programme-funded research in the HTA journal as a monograph, and includes a full description of the methods, results and conclusions of the research. The monographs include an abstract, scientific summary and plain English summary. Although the programme aims to publish all HTA programme-funded research in the HTA journal, the research must be of sufficiently high scientific quality, as assessed by external peer reviewers and the journal’s editors, to be published in the journal. The purpose of the journal is to ensure that the full results of all studies are publicly available. However, the HTA programme also encourages the publication of results in other peer-reviewed journals. Prior publication of HTA programme-funded research results in other journals is not a limitation to the publication of a monograph in the HTA journal. Typically, the HTA journal article is one of the final outputs of any project.

However, not all HTA programme-funded research is published in the HTA journal. For example, STAs are typically not published in the HTA journal (with a limited number of exceptions) because they are usually based on commercial data that were provided in confidence, and they primarily review manufacturers’ evidence rather than present original research. Despite the growth in the size of the HTA programme, there is evidence that the programme has maintained its academic quality, with the HTA journal’s 5-year impact factor of 5.595 ranking it fourth among all journals in the Health Care Sciences and Services category. 13

Adding value in research

The HTA programme has also been influenced by the work of Chalmers and Glasziou (2009)14 on research waste and, more recently, by the same authors and others in a series of publications on research waste in The Lancet in January 2014. The five papers15–19 published in The Lancet each expand on one of the key themes identified in the earlier paper by Chalmers and Glasziou (2009):14

1. decisions about which research to fund based on issues relevant to users of research15

2. appropriate research design, methods and analysis16

3. efficient research regulation and management17

4. fully accessible research information18

5. unbiased and usable research reports. 19

The HTA programme and the NIHR have used these themes to develop the Adding Value in Research framework. 20 The aim of the framework is to ensure that NIHR-funded research ‘answers questions relevant to clinicians, patients and the public; uses appropriate design and methods; is delivered efficiently; results in accessible full publication; and produces unbiased and usable reports’ (p. 1). 21 In fact, the HTA programme has been considered an exemplar of good practice in many of these areas. For example, Chalmers and Glasziou (2009)14 note:

Some elements of these recommendations reflect policies already implemented by some research funders in some countries. For example, the NIHR’s Health Technology Assessment Programme routinely requires or commissions systematic reviews before funding primary studies, publishes all research as web-accessible monographs, and, since 2006, has made all new protocols freely available.

p. 8814

The HTA programme’s perceived success in adding value in research contrasts with the evidence that shows that researchers more widely are not making sufficient use of existing evidence in the design and execution of their research. For example, the study by Clark et al. (2013)22 showed that of 446 trials submitted to research ethics committees in the UK, only 4% used meta-analyses of data from relevant previous studies to plan target sample sizes. Similarly, an analysis of clinical trials by Robinson et al. (2011)23 found that less than one-quarter of previous trials were cited in reports. This evidence suggests that not all research funders adhere to the UK policy on research governance in the biomedical research sector. According to the DH’s advice on research governance ‘all existing sources of evidence, especially systematic review, must be considered carefully before undertaking research. Research which duplicates other work unnecessarily, or which is not of sufficient quality to contribute something useful to exiting knowledge, is unethical’. 24

The Adding Value in Research framework is used by the HTA programme and the NIHR more widely for self-assessment and ongoing improvement. Through the Research on Research programme that was established in 2007, the NIHR funds studies on how they manage their research programmes and the projects they fund. Adding Value in Research is one of three core strategic areas of the Research on Research programme. Recent work funded through the programme has shown that the HTA programme is performing well against several of the elements of the framework. Wright et al. (2014)25 provided evidence that the programme is supporting research of clinical relevance. Turner et al. (2013)26 found that 95.7% of all HTA studies either have published, or will publish, a monograph in the HTA journal, and that that percentage increases to 98% for studies commissioned after 2002. Chinnery et al. (2013)27 showed that these publications were available promptly. They also found that the median time to publication for a HTA monograph was 9 months shorter than an external journal article.

However, published evidence suggests that the HTA programme could improve the value that it adds. Work by Douet et al. (2014)28 showed that not all the reports published in the HTA journal provide sufficient information to replicate the work, with components missing in 69.4% of the 98 reports analysed. In particular, only 58.2% of reports had complete patient information. Jones et al. (2013)29 looked at the use of systematic reviews in planning a sample of randomised trials and found that although the majority of HTA programme-funded research referenced the systematic review in their application, only half used the systematic review to inform the design of the study. Turner et al. (2010)30 found that, in one particular research area, eight similar HTA studies had been funded by different international HTA organisations. Although the studies looked at the issue within different contexts, the similarity between studies suggests that potential duplication of effort and research waste may have taken place. The authors suggest that such resource wastage could be minimised by the use of a toolkit designed to help adapt HTA reports between different contexts.

Complementary to the Adding Value in Research framework is the ‘needs-led, science-added’ approach, which is a set of principles used across the NIHR, including the HTA programme, which aims to maximise the utility of research for decision-makers. The underpinning principle is that research should be ‘needs led’ in that it should reflect the key information needs of decision-makers, while also being of high quality. To this end, the NIHR undertakes a range of activities, such as involving stakeholders – including policy-makers, patients and the public – in topic identification and prioritisation (as described above); conducting peer review of proposals and making the funding decisions via expert panels; ongoing monitoring and contact with projects in process; and comprehensive publication of findings in the NIHR journals library.

Patient and public involvement in research

Both the Adding Value in Research framework and the ‘needs-led, science-added’ approach advocate the involvement of patients and the public in the design and conduct of research. The NIHR defines involvement of patients and the public as:

1. . . . an active partnership between the public and researchers,

1. research done with or by members of the public, not to or about them,

1. the public getting involved in the research process itself.

p. 231

Health Technology Assessment programme-funded studies are expected to demonstrate that they will involve patients and the public, and this forms part of the assessment of proposals submitted for funding. The NIHR provides guidance to researchers on what that involvement could consist of, suggesting that patients and the public could be involved in:

• designing questionnaires and patient information sheets

• helping to find participants and designing the best way to approach them

• participating in advisory or steering groups

• undertaking aspects of the research

• contributing to or commenting on the final report.

One of the main support mechanisms that the NIHR provides support for researchers is the Research Design Service, which can give researchers access to relevant patients or members of the public that they have recruited into their patient and public involvement (PPI) panels. In a study of a sample of HTA programme-funded projects, 85% of proposals described PPI representation in their application. However, only 41% of reviewer comments across the trials commented on the PPI plan, which suggests that more guidance could be provided for reviewers on this process. 32

Benefits, drawbacks and common pitfalls of bibliometrics

The advantages of bibliometric methods are straightforward: they provide a quasi-objective and quantitative method of evaluating the performance of research, researchers, institutions and research systems. However, caution is needed when interpreting bibliometric results, as a number of caveats and drawbacks exist.

First, and foremost, citations are not a complete representation of research quality and are only ever a proxy. Bibliometric analysis provides a quantitative reflection of research performance and does not take into account the nuance that can be achieved with a more in-depth, qualitative analysis. Although it is generally accepted that there is a strong correlation between citations and scientific quality at the article, individual researcher and national levels, it does not necessarily follow that an article with a low number of citations is of low scientific value. Furthermore, ‘impact’ as demonstrated by a high level of citation should be considered in a narrow sense as impact within academia, and does not necessarily imply that the research will go on to impact on policy, practice or society more widely. 39 In order to build a more complete picture of research quality and impact, bibliometric indicators should be considered alongside other research evaluation methods, as we have done in this study. It is also important to use a variety of bibliometric indicators, as each has its own particular strengths and weaknesses, for example in relation to sensitivity to skewed distributions or small sample sizes. The specific indicators used in this study are explained further below.

Another issue to be aware of is that different research fields exhibit different citation patterns, with some fields attracting a larger number of citations than others. This can be attributed to a number of factors, including the size of the field; the number of journals and the number of times per year these journals are published; the number of journals indexed in Web of Science (WoS); and the publication norms of the field (e.g. the publication of research as book chapters rather than journal articles is more common in the humanities than the fields of natural and social sciences). These factors contribute to research in some fields having a higher probability of being published and/or cited. Additionally, assessment can be distorted by ‘fashions’ in particular fields; areas seen as particularly topical may attract large amounts of funding, more researchers and more citations. It is possible to control for these differences between fields, to some extent, and further detail on how this was achieved in this study is provided below.

Citations can also sometimes be manipulated by researchers and research organisations to unfairly represent the value of their scientific output. Citation counts can be enhanced by researchers citing their own work (self-citations) as well as researchers publishing flawed or controversial work that is likely to gain a number of citations in other articles that criticise the original research. These types of ‘negative citations’ are not easily recognised using bibliometric analysis, as all citations are assumed to be equal. Identification of negative citations requires a more in-depth, qualitative analysis, which is resource intensive and outside the scope of the current study.

Finally, bibliometric analysis tends to under-represent the research output of non-English speaking countries. The majority of journals indexed in citation databases, such as WoS, are English-language journals and the small number of non-English-language journals indexed may unfairly represent the value of research published in other languages. This issue should not prove particularly problematic in evaluating the HTA programme’s research, as it can reasonably be assumed that all project outputs are published in English.

Bibliometrics in this study

Bibliometric analysis was used in this study to provide a quantitative analysis of the academic output and impact of HTA programme-funded projects. It was carried out at two levels: an overall assessment of the entire HTA portfolio and specific project-level profiles of the studies selected as case studies. Although many of the indicators were the same for these two purposes, there were differences in the methods used, particularly in compiling the relevant data. Each of these is described below, in turn (see Identifying the Outputs of Health Technology Assessment programme-funded research).

Bibliometric data source

The Centre for Science and Technology Studies (CWTS) was contracted by RAND Europe to provide the bibliometric analysis for this project. CWTS is an interdisciplinary research institute housed within the Faculty of Social Sciences at Leiden University in the Netherlands, which specialises in advanced quantitative analysis of scientific research and its connections to technology, innovation and society.

The CWTS maintains a bibliometric database of scientific publications for the period 1980 to the present, generated from the Thomson Reuters WoS database. WoS is a bibliographic database that covers the publications of about 12,000 journals in the natural sciences, the social sciences, and the arts and humanities. Each journal in WoS is assigned to one or more subject categories representing different fields of research. The CWTS in-house database makes a number of improvements to the original WoS data, most importantly by using a more advanced citation-matching algorithm and an extensive system for address unification. The database is based on the journals and serials of the Science Citation Index and associated citation indices: the Science Citation Index, the Social Science Citation Index, and the Arts & Humanities Citation Index, extended with six so-called specialty Citation Indices (Chemistry, Compumath, Materials Science, Biotechnology, Biochemistry & Biophysics, and Neuroscience).

Identifying the outputs of Health Technology Assessment programme-funded research

Indicators

As noted above, it is useful to use a variety of bibliometric indicators to reflect different aspects of the HTA programme’s research output. A summary of these is provided in Table 4. A more in-depth discussion of the meaning and use of these indicators follows below.

Case study selection

We conducted 12 case studies with individual HTA projects as the unit of analysis for the case studies. As described at the start of the chapter, the case studies were conducted using a payback case study approach. Following the discussion with the advisory board, we decided to select a purposive sample of case studies of projects that we expected to have had high impacts across a range of different areas. The intention was that this would allow us to explore the range of impacts emerging from the HTA programme and the routes by which that impact occurred. One disadvantage of this selection approach was that it tells us less about cases in which research did not have an impact – to partially address this we examined the barriers to impact that our case studies had overcome. The selection approach also means that we cannot generalise from the case studies to extrapolate the overall impact of the HTA programme. In addition, case studies were chosen to reflect the diversity of the research funded through the programme. Overall, case study selection was based on four main criteria:

• high impact

• mix of primary research, evidence synthesis and TARs

• mix of research areas

• timing of the publication of final results.

Each of these is described in more detail below.

Conducting the case studies

A case study lead was identified for each case study, and this researcher led all data collection for that case study. In addition, a second person was assigned to provide support on each case study, to act as a sounding board for discussion and also ensure consistency of approach across all of the case studies. A total of five researchers conducted all of the case studies, with overlaps between researchers to ensure that best practice and learning was shared among the team.

We gathered data from two main sources: document review and interviews. When possible, we conducted interviews with the CIs for each study in the early stages of the data collection process to establish the key impacts of the study, the publications resulting from the study and relevant stakeholders for further interviews. If the CI was not available at an early stage, another key member of the study team (e.g. the clinical lead) was interviewed initially, and the CI was interviewed at a later stage if possible. Interview data were then supplemented by analysis of relevant documents, including published papers and reports, guidelines, policy documents, systematic reviews, meta-analyses, project records (such as end of grant reports) and, where available, original proposal documentation and curricula vitae (CVs) of project team members. As well as verifying and expanding upon information from interviews, the document analysis was used to identify key informants for further interviews.

Interviews were then conducted with members of the project team, policy-makers, clinicians and patient representatives. Interviews were semistructured, based on a protocol tailored to the payback framework, but adapted for the types of research conducted by the HTA programme (provided in Appendix 3). The full protocol was used only with the CI (or in the initial interview with a key member of the project team when the CI was not able to be contacted), but relevant sections were selected and supplemented as required for use with other informants, based on their expertise and the gaps in our knowledge. Interviews were conducted by telephone and were recorded with permission from the key informants. Recordings were confidential and for the use of the research team only, and were destroyed upon completion of the study. We conducted between two and seven interviews per case study, each lasting approximately 1 hour. The interviews, in turn, supported the identification of further relevant documentation for analysis. All interview participants were given the opportunity to review and comment on the final case study, and request that any quotations attributed to them in the case study were anonymised or removed.

In each of the case studies, we analysed the initial and long-term outputs and outcomes of HTA research, including translation of the research findings into the knowledge base or clinical practice. Through key informant interviews, we explored possible attribution problems by asking the studies’ CIs, and other key stakeholders, what would have happened in the absence of the HTA research. In particular, the case studies focused on the outcomes of the research for the NHS and patients.

The final results of each case study are presented in a standard template using the payback framework to enable comparability across all of the case studies. The full text of all of the case studies is provided in Appendix 4. We conducted a cross-cutting qualitative analysis of all of the case studies, using the impact categories described at the start of this chapter. The aim of the qualitative analysis of the case studies was to identify the key impact mechanisms associated with HTA research, as well as success factors, i.e. things that support the successful translation and implementation of the findings of HTA research. To conduct this analysis, we coded all of the case studies in NVivo using the codebook utilised for the analysis of interviews (to allow comparison with those findings). Each case study was coded by the lead researcher involved in conducting each case study, plus one other researcher from the study team. One researcher coded all case studies to ensure comparability and consistency in the use of the codebook. Coding was conducted iteratively, with potential new codes identified, discussed as a team, added where appropriate and coded across all case studies to ensure consistency. Before the coding began, the team met to discuss all of the codes to ensure a shared understanding of the coding approach. Coding was then compared between both researchers who were coding each case study in order to analyse the level of consistency and resolve any disagreements in coding. The final coded case studies were then examined, code by code, with two key topics in mind. First, the nature, range and extent of impact observed across the case study set. Second, the barriers and facilitators of that impact, with the intention of making observations around actions that the HTA programme might be able to take to increase those impacts in the future. The codes were examined iteratively through a process of separate analysis and team workshops to identify emerging themes and observations. The analysis was conducted in the context of the findings of the earlier stages of the project. All of the case study authors were involved in the workshops and the analysis process to ensure that the tacit knowledge obtained through conducting the case studies, which may not have been fully captured in the case study text, was included in the wider analysis.

The primary route to impact of Health Technology Assessment programme-funded research is through guidelines, particularly National Institute for Health and Care Excellence and National Screening Committee guidelines

Interviewees reported that the primary route to impact of HTA programme-funded research on patients is through its impact on clinical guidelines, which, in turn, affects clinical care. If the HTA programme funds high-quality research that is then incorporated into clinical guidelines which clinicians follow; patients receiving that care can be said to have benefited from the initial research that led to the improvement in care. One interviewee thought that the impact of HTA programme-funded research on patient care was widespread (I19):

Somebody who is receiving treatment as a result of a NICE appraisal, based on an HTA assessment by the HTA programme, will be affected by the HTA programme but will have no idea of that.

I19

The fact that the HTA programme primarily has an impact on patients through clinical guidelines complicated the assessment of the impact of HTA programme-funded research because it is not possible to attribute that impact directly to the HTA programme, as both the underlying research and the resulting guidelines contributed to the observed impact. Two interviewees highlighted the challenges of attributing the impact of improved patient care as a result of better guidelines to the HTA programme. The interviewees reported that, when looking at the impact of HTA research on clinical practice, it is legitimate to look at the joint impact of NICE and the HTA programme (I4, I16). One interviewee described this joint impact in more detail:

They have helped each other considerably and probably have brought about a bigger impact than if they were working separately . . . They could exist without each other, but if NICE didn’t have the HTA then it would have to rely on what individual assessors have sent to drug companies, which is what happens in some other countries, and if NICE didn’t exist then the HTA programme would be producing reports that wouldn’t have the same traction.

I16

Another challenge in attributing the impact of HTA programme-funded research on patients to the HTA programme is that many users of HTA programme-funded research may not be aware that the HTA programme was the funder of the research. For example, users of scientific evidence do not always note the funder of research before reading and using the findings of research. Interviewees highlighted that many clinicians have heard of the HTA programme but that it is likely that their clinical practice has been influenced by the research funded by the HTA programme (I8, I19, I9). For example, one interviewee noted:

I think the impact is through guidelines. I don’t think the average clinician had the slightest idea what the HTA was, had never heard of it, would never look it up, directly. Physicians got HTA research results indirectly. They are following many of the things that HTA might have suggested.

I8

One interviewee felt that the attribution challenges resulting from the lack of awareness of the HTA programme among users of HTA programme-funded research may be more widespread, extending beyond clinicians to all users of HTA programme-funded research. The interviewee reported that the HTA programme directly or indirectly influences everyone who has an interest in the clinical effectiveness, cost-effectiveness and wider impacts of health technologies, whether or not those users of HTA research are aware of the HTA programme (I19).

Individual pieces of Health Technology Assessment programme-funded research can have an impact on the NHS and patients

More than half of the case studies (7 out of 12) provided some evidence that HTA programme-funded research has an impact on the NHS and patients, which is consistent with the findings reported by interviewees. In addition to the seven case studies41,42,44–46,49,52 that had an impact on the NHS and patients, four studies47,48,50,51 included limited evidence of changes in clinical practice.

The observed impacts included health benefits for patients; an increase in patient choice; and potential cost savings. Two case studies [ARTISTIC (A Randomised Trial of HPV testing in primary cervical screening)41 and Newborn CHD42 (Newborn Screening for congenital heart defects)] will probably have a significant impact on patients and the NHS in future, as they have both led to screening pilots that may eventually be implemented nationally. We also identified two cases where HTA programme-funded research confirmed the appropriateness of existing treatment [CoBalT (Cognitive Behavioural Therapy as an adjunct to pharmacotherapy for treatment-resistant depression in primary care trial)50 and SWET (Softened Water Eczema Trial)48]. However, identifying the impact of studies that provide the evidence for existing practice is challenging because changes in clinical practice are unlikely to occur. Finally, two studies [IVAN (A randomised controlled trial of alternative treatments to Inhibit VEGF in patients with Age-related choroidal Neovascularisation)43 and CUtLASS (Cost Utility of the Latest Antipsychotics in Severe Schizophrenia trial)51] demonstrated potential for impact on the NHS and patients but barriers to that impact exist within the health-care system (see External factors can influence the implementation of HTA-funded research, below). The impacts of the case studies on the NHS and patients are summarised in Table 8 and described in more detail in the following sections.

There is some evidence that Health Technology Assessment programme-funded research can have a direct impact on patient care

In some cases, clinicians may change clinical practice as a direct result of HTA programme-funded research if the effect demonstrated in the study is sufficiently large and they are aware of the study findings. One interviewee reported that widespread dissemination of the study results, or publicity of the results, may lead to adoption of the new technology before guidelines have been updated. Another potential direct impact that HTA programme-funded research may have on patient care is by improving the skills of clinicians in the NHS through their participation in research. One interviewee suggested that if clinical trials increase the skills of clinicians then that increase in skill is likely to have a positive impact on patient care.

The case studies found evidence that implementation of HTA programme-funded research sometimes occurs outside the conventional pathways (not via NICE or the NSC). For example, TXA in trauma is being used in the NHS as a result of the CRASH-2 study44 findings, even although TXA for trauma has not yet been included in NICE guidelines (as described in more detail below). The inclusion of TXA for trauma in the Joint Royal Colleges Ambulance Liaison Committee guidelines, and clinicians’ use of TXA off-label, has facilitated adoption in the NHS. However, the research team has also been actively involved in the widespread dissemination of the research findings, as described in the next section. A number of interviewees noted that the HTA programme’s framework for dissemination is not designed to support widespread dissemination (I14, I8, I9, I13) – one commented that it is ‘essentially an academic one’ (I9) – assuming that impacts on practice will occur through guideline bodies.

Data from Researchfish show that > 70% of dissemination activities recorded consisted of a talk or presentation, or attending a scientific meeting (Figure 3). However, although academics were a key audience for dissemination (n = 128), the largest group targeted by dissemination activities listed in Researchfish were health professionals and practitioners (n = 463), suggesting that HTA researchers are more active in direct dissemination than our interviewees might have been aware (Figure 4).

FIGURE 3.

Dissemination activities recorded in Researchfish.

FIGURE 4.

Audiences for dissemination activities recorded in Researchfish.

The HTA programme is not involved in the direct implementation of HTA research in clinical practice, but rather funds research that is intended to provide evidence for clinical practice in the NHS. Although the HTA programme is implementation focused in terms of the research topics it funds, the active implementation of research is currently outside its remit as a research programme. Some interviewees viewed the division of labour between the HTA programme as a funder of research and the NHS as an implementer as desirable, although one interviewee disagreed with the view that the HTA programme receives funding to do research and that the NHS receives money to implement new technologies:

Unless the HTA programme is thinking about implementation while it’s doing its work and developing implementation strategies, it’s less likely that things will get picked up.

I8

In order for HTA programme-funded research to be implemented in the NHS, the research must be implementation focused, such that the findings are useful to those who manage and provide care in the NHS. However, the challenge of translating research into clinical practice is not unique to the HTA programme. Interviewees stressed that implementation of research (from any source) in the NHS is challenging and takes time (I2, I12, I19).

However, it is possible for the HTA programme and researchers to influence the ease with which HTA research is adopted in the NHS, particularly through dissemination. One interviewee noted that the HTA programme relies too much on the NIHR’s existing communication channels and that it does not make adequate use of other communication channels, such as the clinical and professional networks of HTA programme-funded researchers (I12). Some interviewees reported that the challenges the HTA programme faces in terms of dissemination apply to the NIHR as a whole, rather than to just the HTA programme (I15, I13), and that the new NIHR dissemination centre may address some of the dissemination challenges that the programme faces (I6).

However, one interviewee noted that the primary objective of the HTA programme, as part of the NIHR, is to inform decisions that related to practice within the NHS, whereas the primary objective of the Policy Research Programme is to inform wider policy-related decisions (e.g. how best to commission services) (I19). The results of this divisions between the two research funders is that the main audience of HTA research is patients, clinicians, managers and organisations at the meta-level, such as NICE, which makes decisions on their behalf. That interviewee suggested that the HTA programme’s dissemination strategy should target clinicians and patients directly:

The HTA programme is designed to inform decision-making by clinicians, managers and patients in the NHS. So, the way that it does that through the policy-makers is by influencing NICE who influences decision-makers, but the whole idea is to go straight to the decision-makers so that clinicians are aware of the latest evidence in their field . . . It is the professionals that will know that and, increasingly, the patients that they treat, which is why [the HTA programme] is very keen on plain English summaries of research.

I19

Funding for the dissemination of research was identified as a challenge in many of the case studies. In some of the case studies, dissemination of the research findings seems to have been impeded by a lack of incentive to implement the research findings in the NHS or, in some cases, the opposition to their implementation by some stakeholders. We discuss this issue in detail in the next section.

There are particular issues associated with the dissemination of the research findings from TARs, which are unique to the TAR programme. For example, members of the RA45 team were not clear whether or not it was appropriate to actively disseminate their research findings or whether or not the research was intended solely to inform NICE decisions. In the TAR centre case study, the TAR team’s desire to disseminate their work was also constrained by the continual change in focus and research topics within the centre and a lack of time for dissemination of older studies while carrying out new ones.

It is plausible that HTA programme-funded research could lead to improved care in the NHS by improving the skills and knowledge of clinicians. Interviewees also suggested that HTA research could lead to direct improvements in health through participation in research, which one interviewee suggested could have an impact on the overall quality of care provided in the NHS because of the volume of research funded in the NHS. However, another interviewee noted that there is insufficient evidence on whether or not participation in research leads to improved skills among clinicians and whether or not it has a significant effect on patient care (I14).

External factors can influence the implementation of Health Technology Assessment programme-funded research

Wider system factors and the priorities of other stakeholders can have a substantial influence on the impact of HTA research. Looking across our case studies, we identified five ways in which this can happen: findings that challenge commercial interests; findings that are in stark contrast with what is expected; changes in the commissioning structure of the NHS; changes or uncertainties around the cost of implementation; and factors completely outside the health system.

Study design can influence whether or not research is implemented in the NHS

In several cases, the study design played an important role in facilitating the uptake of the findings subsequently. Many of the studies were clearly tailored to use in the NHS, reflecting the wider priorities of the HTA programme. It is important to note that there is typically a tension between this type of pragmatic design and the design of a robust clinical trial that is appropriate for inclusion in wider meta-analyses. Indeed, because of such design, several of the studies received methodological criticisms, particularly outside the UK. However, nonetheless, reflecting the priorities of the programme, many of the studies took this approach, which then allowed the findings to be used to assess the relative effectiveness and cost-effectiveness of the approaches considered in the specific context of the UK health system.

For example, the trial design for the CESAR study49 was considered controversial because only the patients randomised to the intervention were moved to the ECMO centre. In some previous trials, everyone who met the criteria for ECMO was transferred to an ECMO centre, and half received standard care whereas the other half received ECMO. The research team made the decision that transferring patients to an ECMO centre to receive standard care was not appropriate for a UK context because, in the UK, patients who were not going to receive ECMO would never be transferred but, rather, would be treated locally. This focus on designing the trial to reflect the UK context meant that results were less comparable to other studies in the field.

Other elements of the study design were also seen to contribute to the usefulness of the findings in several case studies. The EVAR study,46 for example, was the only one among four international trials that, in addition to comparing endovascular aneurysm repair to open repair, also looked at the use of endovascular aneurysm repair among patients who were unfit to undergo open repair to see if the less serious surgery could be an option for this group. Furthermore, it has the longest-term follow-up out of all of the studies internationally, offering additional information about the long-term safety and outcomes of endovascular aneurysm repair devices. In the CRASH-2 study,44 the choice of TXA as the drug to be investigated (among a choice of several potential options) was important for the subsequent uptake of findings, as it is inexpensive and already familiar to clinicians because it is used in other types of surgery. In SWET,48 the careful consideration of the outcome measure was very important to the objectivity of findings, as it was not possible to blind participants to their treatment allocation.

The Health Technology Assessment programme funds high-quality scientific evidence

The HTA programme has a number of measures in place to ensure that the research that it funds produces robust scientific evidence. For example, all research applications and HTA monographs are peer reviewed. Many interviewees praised the quality of HTA programme-funded research (I2, I3, I19) and viewed the evidence that HTA programme-funded research produces on the effectiveness and cost-effectiveness of health-care interventions as invaluable to policy-makers (I2, I3, I6–I19). For example, one interviewee reported that the most important achievement of the HTA programme has been its provision of evidence to the government about the effectiveness of health technologies:

It’s really important to clock that and the importance of the decisions that are informed by it.

I19

An important aspect of HTA programme-funded research is the health-economic evaluation that is a component of many HTA programme-funded research projects. Some interviewees report that the cost-effectiveness analysis is particularly important for NICE decision-making. One interviewee reports that it is ‘exactly what [NICE] would do’ and prevents NICE from having to conduct the analysis separately (I18).

As noted above (see The primary route to impact of Health Technology Assessment programme-funded research is through guidelines, particularly National Institute for Health and Care Excellence and National Screening Committee guidelines), the primary route to impact of HTA programme-funded research on patients is through clinical guidelines. Similarly, the primary impact of HTA programme-funded research on health policy is on clinical guidelines. Many interviewees cited the inclusion of HTA research in guidelines as evidence that the HTA programme informs decisions about the delivery of health-care services (I3, I6–I9, I14–I18). One interviewee noted that:

The HTA programme is the vodka in the cocktail. It’s the crucial ingredient without many things people know and love like clinical guidelines cannot work.

I3

However, the importance of HTA programme-funded research to clinical guidelines has changed over time because of the shift in attitudes towards evidence-based medicine. One interviewee noted that:

When HTA started, there wasn’t really guidance. So, the idea was that the programme was aimed at the practitioners. But, with the development of NICE particularly of course, but many other guidance producers, it has turned out, I don’t think intentionally, that guidance producers are the primary consumers of quality pragmatic primary research that the HTA produces.

I9

Understanding the remit of the HTA programme is essential to interpreting the impact of the HTA programme on health policy. For example, the HTA programme does not produce guidelines itself, but rather is a funder of research that results in scientific evidence, which may then be included in clinical guidelines. Two interviewees stressed that the HTA programme does not make policy decisions or produce policy advice, but, rather, stops very clearly at the point of funding research, analysing and presenting the findings and discussing their implications (I3, I19).

Health Technology Assessment research has had wide range of different types of impacts on policy

Data from the 68% (422 out of 619) of awards captured in Researchfish indicated that the HTA programme has had a range of impacts on policy. Of the 422 projects with data, 22% reported having an impact on policy (15% of the overall portfolio). More detail on the sample that responded and the specifics of Researchfish response rates is given in Chapter 2. The Researchfish data identified a range of different types of impacts across the portfolio of HTA funding, with some projects mentioning more than one impact within influence on policy. The types of impact include: influenced training of practitioners or researchers; membership of a guidance committee; participation in an advisory committee; citation in clinical reviews; citation in other policy documents; citation in systematic reviews; citation in clinical guidelines; implementation circular, rapid advice or letter; and gave evidence to a government review as shown in Figure 5.

FIGURE 5.

Impacts on policy recorded in Researchfish.

According to the Researchfish data, these impacts on policy occurred mainly in the UK, as shown in Figure 6 . Out of the projects that reported an impact on policy, nearly 60% were within the UK. However, 30% also reported an impact on policy and practice internationally. The kinds of international impacts reported include citation of the research in other countries’ guidelines, and participation and membership in international working groups and committees.

FIGURE 6.

Reach of impacts on policy recorded in Researchfish.

The case studies also illustrated the wide range of impacts on UK policy. Of the 12 case studies,41–52 10 studies41,42,44–49,51,52 indicated evidence of some impact on UK policy. The main routes to policy impact were through citation on guidance, typically although not exclusively NICE guidance, and through a NSC pilot. These are outlined in Table 9 and described in the following sections.

National Institute for Health and Care Excellence and the National Screening Committee are key users of Health Technology Assessment research

Both NICE and the NSC use HTA programme-funded research in their guidelines. For example, Raftery and Powell (2013)11 found that up to July 2013 the HTA programme has funded an academic assessment of all of the 293 topics appraised by the NICE Technology Appraisal Programme. The authors also found that HTA programme-funded research underpins many of the NICE and NSC guidelines. For example, they found that HTA programme-funded research was the main supporting evidence referenced for 9 out of the 20 disease-specific screening programmes in England. They also found that 55 out of the 106 screening topics considered by the NSC since 1986 were informed by HTA programme-funded research.

Most interviewees identified NICE and the NSC as important users of HTA research (I2, I3, I7–I9, I15, I16, I19). However, interviewees also noted that the way policy-makers use HTA research will vary depending on the type of HTA programme-funded research (e.g. systematic review or primary research). For example, the HTA programme produces bespoke evidence syntheses for both NICE and the NSC, which have a direct impact on policy (I2–I8, I15, I16, I19). Similarly, all of the HTA research that is funded to support NICE technology appraisals has a direct, nearly one-to-one, impact on NICE technology appraisals (I7, I15, I16, I18, I19). One interviewee noted that in both these instances, it would be exceptional if the HTA research did not have an impact on policy (I15). HTA research also feeds into NICE clinical and public health guidelines (I15, I16, I18), which, as one interviewee (I15) explained, happens in three ways: the guidelines may reference a HTA programme-funded trial, take advantage of some early finding or subsume previous NICE guidelines that included HTA research. Lastly, the HTA also funds non-bespoke evidence syntheses for the NSC when they have identified a gap in the existing evidence. However, for those non-bespoke evidence syntheses, the research is more likely to have a longer-term impact.

The importance of NICE and NSC as routes for policy impact are also illustrated by the case studies. Seven45–49,51,52 of the case studies had an impact on policy by influencing NICE guidance and a further two41,42 had an impact on NSC policy by leading to a pilot – we elaborate on the details of those impacts below.

Other policy-makers also use Health Technology Assessment programme-funded research

A number of other policy-makers also use HTA programme-funded research. A number of interviewees recognised other guideline producers, such as SIGN, and individuals involved in the production of systematic reviews or experts in a particular field in which the HTA programme funds research as other important users of HTA research (I5, I14, I15, I17).

One interviewee noted that the wider policy teams within the DH would not usually have direct engagement with the HTA programme, but rather that the DH R&D directorate brokers that on their behalf. The interviewee noted that a particular policy team’s awareness of the programme would depend on how interested that team is in the source of the research:

Whether they would just note that this evidence was produced by a programme and that it was robust evidence and therefore they use it or whether they would know the HTA programme by name . . . would depend on their personal level of interest. But that’s the same as whether they would know that the British Heart Foundation funds heart research or the MRC funds medical research.

I19

Although there are other users of HTA research, the primary users of HTA programme-funded research seem to be NICE and the NSC. One interviewee noted that a possible criticism of the HTA programme is that it is too NICE centric, commenting that:

It is extremely important that [the HTA programme] continues to serve a wide range of masters, not just NICE. NICE should be the primary customer. But [the programme] should continue to consider needs of other decision makers.

I3

The potential limitations of an entirely NICE-focused policy dissemination approach are illustrated by two examples from the case studies,43,44 for which despite strong evidence that a change in policy may be warranted, the NICE guidance has not been updated for regulatory reasons. In the case of both the CRASH-244 and the IVAN study,43 impact on NICE guidance has been hampered by the lack of any incentive for industry to seek a marketing authorisation for the drug in question.

In the case of the CRASH-2 study,44 NICE have not reviewed TXA as part of their standard guideline review process. This is partly because there was no direct push to do so, as the drug manufacturers have not applied for a UK marketing authorisation for use in the trauma setting, and this is unlikely to change, as it is now a generic drug and there is therefore little incentive for pharmaceutical companies to pursue this. Following some lobbying by members of the study44 team, NICE developed an evidence review based on the results of the CRASH-2 trials,44 describing its use in this context. According to the NICE website, TXA may be included within the scope of the NICE clinical guideline on the assessment and management of major trauma, which is estimated to be published in February 2016 (www.nice.org.uk/guidance/indevelopment/gid-cgwave0642). The CI is continuing to press for the drug to be authorised in the UK, as that will allow the drug to be marketed properly. The findings have also been included in the 2012 Joint Royal Colleges Ambulance Liaison Committee guidelines, which recommend pre-hospital TXA for all patients triaged to a trauma centre.

In the case of the IVAN study,43 the situation is more extreme. The drug Avastin, which was found to be as effective but much more cost-effective than Lucentis for neovascular AMD, has not been assessed, and the existing NICE guidance still recommends the use of Lucentis. To assess Avastin would require the DH to commission NICE to carry it out, but this is made more difficult by the fact that the drug’s manufacturer does not intend to apply for a licence to market Avastin for a new indication of neovascular AMD. There is no commercial incentive for the pharmaceutical industry to carry out the relevant safety trials and apply for a licence, as both patents are held by the same company. However, in December 2014 NICE was asked by the DH to begin developing a guideline for the diagnosis and management of macular degeneration. The scoping workshop for this guideline was to be held in April 2015 and it was expected that the guideline would be published in August 2017. It was unclear whether or not this guidance would cover Avastin.

Although in both of these cases industry interests have hampered the inclusion of the relevant drugs in NICE guidance, the impact on practice differed significantly as described above. TXA is widely used off-label for the treatment of trauma, but Avastin is not commonly used in the NHS, as industry has threatened such use with legal action. This may reflect a difference in circumstances: TXA is a generic drug and no company has any interest in pursuing market authorisation; however, in the case of Avastin the pharmaceutical companies involved have an interest in preserving the use of Avastin solely in the cancer indications for which a licence was granted, while marketing Lucentis for its licensed indication of AMD.

Impacts on policy may be delayed by the time taken to publish and the increased focus on primary research in the Health Technology Assessment programme

Although the HTA programme has an important impact on health policy, there are some factors that may delay the programme’s impact. Interviewees highlighted two factors that can delay the impact of HTA research: the increased focus on primary research and the time to publication for research.

The Health Technology Assessment programme produces high-quality, rigorous academic research

There was a clear consensus from the interviews conducted that the research conducted through the HTA programme is of excellent academic quality (I2–I5, I8, I12, I15, I18, I19). This is reflected in the bibliometric data shown in Figure 8. This sets out MNCS per year, both for all publications, as well as separately for papers in the HTA journal. In both cases, the average value that would be expected (normalised for field and year of publication) is one, so on average work funded by the HTA programme is cited more than twice as frequently as would be expected on average, with little difference in the citation rates between the HTA journal and external publications.

FIGURE 8.

Mean normalised citation score per year, for all publications from HTA programme-funded research and for papers in the HTA journal only. Note: this is based on a sample of publications identified through three sources: all articles and reviews published in the HTA journal during the period 2004–12; papers listed on all HTA project pages of the NETSCC website; and papers reported by researchers in Researchfish. As such, this may not be a complete record of all publications related to HTA programme-funded research.

Although MNCS can be skewed by very highly cited articles, the proportion of publications in the top 10% of papers in their field shows that this does not seem to be the case for HTA-supported research (Figure 9). Across the time period of the study, 31% of papers in the HTA journal, and 29% of papers published in other journals, featured in the top 10%.

FIGURE 9.

Proportion of publications in the top 10% of papers in their field, for all publications from HTA programme-funded research and for papers in the HTA journal only. Note: this is based on a sample of publications identified through three sources: all articles and reviews published in the HTA journal during the period 2004–12; papers listed on all HTA project pages of the NETSCC website; and papers reported by researchers in Researchfish. As such, this may not be a complete record of all publications related to HTA programme-funded research.

The fact that HTA journal articles and articles published in other journals performed similarly in terms of MNCS, proportion of papers in the top 10% and proportion of papers uncited (7% of publications in the HTA journal, 6% of wider publications) suggests that the two are fairly similar in terms of citation distribution, and that publication in addition the HTA journal is the norm.

It is worth noting that interviewees also commented on the ‘unique’ combination of this high academic quality with application focused research (I2, I3, I18). For example, one interviewee commented that ‘. . . [HTA research] is academically rigorous and high quality, but nonetheless clearly and deliberately linked to policy and practical decisions’ (I3). Some comments were also made around the way that the HTA’s internal processes supported this. One interviewee commented that the HTA is ‘. . . rigorous about the methods in uses. For example, the HTA will insist that there is a systematic review before there is a randomised trial. They set high standards’. (I5) Another commented on the prioritisation process, stating that ‘prioritisation in HTA ensures really important questions that need really good quality research are addressed rigorously and be made publicly available’ (I18).

Research quality is also illustrated by the case studies, most of which produced further highly cited articles in addition to the Health Technology Assessment journal publication

Of the 12 case studies,41–52 1041–44,46–51 refer to additional publications outside the HTA journal article. Based on a bibliometric analysis, the quality of these publications as indicated by the number of citations received is high (Table 11). With one exception,48 all of the studies had more than one-quarter of their publications in the top 10% of the field, and this was very high, ≥ 50%, for seven42–47,49 of the case studies. Several of the studies also produced a large number of publications. Although normalised citations take account of the fact that more recent papers have had less time to accumulate citations, it is worth noting that some of the case study projects have not yet published all of their outputs, and we cannot anticipate the impact of future publications in the analysis. For example, there are at least five further publications planned for the IVAN43 project.

Although we did not explicitly explore HTA publication policy in the case studies, one researcher (ARTISTIC41) explicitly praised the policy of allowing publication in other journals before the HTA monograph was produced, allowing study results to be released sooner and made available in highly visible journals.

The exception is the Technology Assessment Report programme, particularly the Single Technology Appraisals, for which academic publication can be challenging

For the RA45 study, the only publication was the HTA journal article. The case study suggests that the results did have the potential for wider publication but that this did not take place because of the time constraints involved in the conducting TARs (see Appendix 4). Researchers immediately move to a new topic and have to produce work on a tight timescale. This means that they typically do not follow up on a particular study and do not continue to work in a particular field, limiting opportunities for publication. It was also noted that it can be difficult to publish TARs because a lot of the data were provided by pharmaceutical companies, as commercial-in-confidence reports, for which the primary sources could not be cited.

Many of the same concerns also apply to STAs. 52 However, in this case, the main output of the work is a report that is used by the NICE Appraisal Committee in their deliberations but is not typically published in the HTA journal (there are a limited number of exceptions among the earlier STAs, but publication as a HTA monograph is no longer carried out). The case study suggests that in addition to the concerns described above for MTAs, the STAs are not perceived to be of the same scientific rigour as MTAs, which further limits publication opportunities (see Appendix 4). However, at ScHARR-TAG (School of Health and Related Research Technology Assessment Groups) they have found a route of publications for many of their TARs in the Pharmacoeconomics journal series.

It should also be noted that, although the HTA journal article is the only output from the RA45 study, it is extremely highly cited, receiving 20 times the expected level of citation for the field, and being among the top 15 most highly cited publications across the entire HTA portfolio analysed at the programme level. However, it is noted in the case study that, despite the high level of citation in this case, solely publishing in the HTA journal could be considered to be limiting in that there is a perception that it is not often read by clinicians (see Appendix 4).

The Health Technology Assessment programme is viewed as an important funder of clinical research, which has had a positive impact on the careers of Health Technology Assessment researchers and research capacity in the UK

The HTA programme has led to the investment of a large amount of money in clinical research, and in HTA research in particular, and the many interviewees felt that this had played an important role in building capacity in the field in the UK (I4, I6, I9, I14, I15, I19). One interviewee felt that this sizeable investment in the field was the key achievement of the programme, stating that:

The most important achievement of the HTA programme is itself. That is to say, it is the commitment of a very large sum of research money on improving practice. If you wanted to celebrate what it does, that’s what you should celebrate, rather than any individual achievement or accomplishment.

I9

This is reflected, according to one interviewee, in the ability of the UK to retain capacity in this field, commenting that:

There are few people working in HTA that go somewhere else. You tend to find in other fields people get sucked to other countries. So, I think it shows that people feel that there is quite a critical mass of researchers in this field in the UK. There hasn’t been large-scale flight of people to other countries. You can’t say that about other fields of research.

I4

This is supported by evidence in the Researchfish database, which provide data from some researchers on the ‘next destination’ of staff who had moved on subsequent to undertaking the research. Positions are generally within the UK, although instances of relocation to China, Qatar and Sweden are mentioned.

There is limited evidence from Researchfish that HTA funding can advance researchers’ careers in other ways. For example, there are 85 instances of individuals gaining skills through HTA programme funding. However, only a very small number of qualifications were reported [three Master of Science/Master of Art; one Doctor of Medicine; eight Doctor of Philosophy (PhDs); five undergraduates].

There is also evidence that further funding was secured based on existing HTA awards recorded in Researchfish from 40 awards. Some of these awards reported more than one instance of follow-on funding, and, across the 89 examples described and set out in Table 12, the median value was £189,029. Over one-quarter of this funding is follow-on funding from NIHR, DH or NHS.

However, one person made a specific comment around the career challenges resulting from working on TARs, particularly STAs that have been introduced relatively recently (I4). These offer fewer opportunities for publications, as they are typically not published in the HTA journal because they are commentary on work that others have done as part of the manufacturer’s submission rather than a lot of original work (in most cases). This has potential career implications in terms of their publication record for the researchers involved.

Overall, the impression from interviewees is that ‘the HTA is prominent in clinical research in the UK’ (I15) and that it ‘support the academic community in this country’ (I19). There is also some suggestion by one of the interviewees that clinical research had previously been in decline in the UK, and that the HTA programme, and NIHR more generally, had played an important role in building back up that research base, although it had taken some time (I6).

Half of the studies were extended or had additional elements added to the initially planned work

In several cases, when the study was considered to be successful because it demonstrated significant impact on the primary outcome, additional follow-up time points were added to the study to allow the effects of interventions to be understood over a longer time period. This took place for the EVAR,46 CoBalT,50 IVAN43 and ARTISTIC41 studies. In some cases, this was coordinated with other international studies on the topic in question. For example, the IVAN43 study has been extended to allow the team to bring back participants for 4- and 5-year follow-ups to look at the longer-term effects of the treatments, matching the 5-year outcomes data captured by the similar CATT (Comparison of AMD Treatments Trials) study in the USA (see Appendix 4.3).

In addition, five case studies (CRASH-2,44 IVAN,43 SWET,48 ARTISTIC41 and EVAR46) described additional ‘nested’ studies, which were made possible by the initial HTA study, using the data produced through the study in a different way, or extending a particular aspect of the study. For example, two ‘nested’ studies, also funded by the HTA programme, stemmed from the CRASH-244 trial. One focused on developing and validating a model to predict death in patients with traumatic bleeding, as well as using that model to then evaluate the effect of TXA on mortality levels. The other looked at the data available in the CRASH-244 data set on intracranial bleeding and the effect of TXA in traumatic brain injury. The IVAN study43 also provides interesting examples of additional ‘bolt-on’ funding, such as creating a serum and deoxyribonucleic acid (DNA) biobank to analyse genetic interactions with the drugs, and exploring the mechanistic aspects of vascular endothelial growth factor (VEGF) metabolism. Although these were of little direct relevance to the aims of the HTA programme-funded project, the samples collected were then able to be used in subsequent analyses funded from other sources.

Most of the studies shaped future research in the field, either in terms of topic or methods

In several cases, members of the study team went on to conduct further trials building on the expertise developed in the HTA study. Some of these were also HTA funded. In other cases, members of the study team advised others on how to set up their trials. For example, members of the SWET48 study team shared their learning about the development of objective outcome measures, how to deal with open trails and patient recruitment. Sometimes studies shaped the development of other trials through publications rather than direct communication, such as the case of the Newborn CHD42 study, which was cited in the methods for two other trials internationally.

In two cases (RA45 and Newborn CHD42), the models developed in the studies42,45 were used by others in further work. For example, the Birmingham Rheumatoid Arthritis Model version 2 (BRAM2) model, which was refined in the RA45 study, has been used in other cost-effectiveness assessments of treatments for rheumatoid arthritis, both by members of the study45 team and others internationally.

Two of the studies41,46 pooled results with other similar studies internationally. In the case of the ARTISTIC study,41 members of the research team were involved in conducting the pooled analysis to the pooled analysis, which included the data from the ARTISTIC study,41 but, in the case of the EVAR study,46 the team led the synthesis, which was funded by the HTA programme and is scheduled for publication in the next year. This raises an interesting point, as with these studies there is sometimes a trade-off between the extent to which the study is tailored for applications in the context of the NHS, and comparability of the study with international results, allowing for this type of pooled analysis, as described previously (see Study design can influence whether or not research is implemented in the NHS, above).

Three studies contributed to stimulating their research field more widely

The publication of the CRASH-244 study findings in 2010 seems to have contributed to wider interest in TXA in the research system, as reflected by an increased level of publication around the topic in PubMed since that point. The IVAN study43 was one of the first of its kind in the UK, so also contributed to shaping the work that has followed in the field. Similarly, at the time the EVAR46 study was conducted, trials were not commonplace in the vascular surgery community at the time, and the EVAR46 study team was one of the early groups to start to conduct trials in this field. As such, the trial46 and the team became a source of knowledge and advice for others in vascular surgery who were looking to conduct trials.

Although no further studies have been conducted on the use of water softeners for the treatment of eczema, SWET contributed to increased interest in the relationship between water hardness and eczema. For example, work looking at the association between hard water and the development of eczema is ongoing, and preparing a national follow-up trial looking at whether or not water softeners might help prevent eczema from birth, rather than treat established eczema. SWET48 also influenced the field more widely through the Harmonising Outcome Measures for Eczema initiative, which was established after the trial (www.nottingham.ac.uk/homeforeczema/index.aspx). Members of the team contributed their expertise in developing objective outcome measures for eczema research and this should contribute to the improvement of eczema research internationally.

All of the studies had some capacity for building impacts for the individuals directly involved in the study, although scope for this was often limited because the researchers were already well established

There was evidence that the studies benefited the careers of those involved, from overall reputational benefit of being involved (ARTISTIC,41 RA,45 STAs,52 CoBalT50) to the award of PhD or other qualifications to members of the team (ARTISTIC,41 Carotid Stenosis,47 EVAR,46 CoBalT50); the winning of awards, honours or prizes (ARTISTIC,41 EVAR,46 SWET48); and through team members securing advisory roles, for example on guidelines committees (STAs,52 Newborn CHD,42 ARTISTIC41).

In 841,42,44,46,48–51 of the 12 case studies, however, it is indicated that the researchers involved in conducting the study already had established careers, so the scope for this particular study to contribute to their career progression was limited, or at least the study was only one of many important studies that contributed to their reputation and any other indicators of esteem. So although you can often see an upwards career trajectory for researchers involved, it is hard to attribute it to this study in particular. This also, perhaps, suggests that the HTA typically funds more established researchers, although this may be an artefact of the case studies selected.

Many studies made important contributions to building capacity more widely, outside the immediate study team

In several cases, the studies contributed to the building of capacity outside the immediate study team. In some cases, this refers to building of wider research capacity within the institution. For example, the IVAN43 study allowed the Clinical Trials Unit (CTU) at Bristol to apply for UK Clinical Research Collaboration (UKCRC) registration and expand rapidly in both number of staff and expertise, whereas the EVAR46 study provided wider capacity-building opportunities for others working at Imperial College London, in terms of learning how to set up and run trials in the area through mentoring and observation with members of the study46 team, who, having previously run the UK Small Aneurysm Trial, were quite experienced. Three studies also mentioned the development of enhanced networks (CoBalT,50 IVAN,43 SWET48).

Five studies43–46,49 mentioned wider research capacity-building activities outside the institution. Work conducted through the RA45 study contributed to the development of HTA internationally, as members of the team briefed the work in other countries as an example illustrating how the HTA process works in the UK. Members of the CRASH-244 study proactively undertook a range of capacity-building activities as part of this work, including travelling to a number of the key collaborating sites overseas (e.g. in Nigeria and Pakistan) to support them in terms of developing relevant capacity in the management of research, around issues such as monitoring of trials and ethics processes. Three cases43,46,49 also suggested that the studies were important in building up research capacity in a new or emerging area, for example by bringing more researchers into the field (CESAR,49 EVAR,46 IVAN43). For example, the IVAN43 case study suggests that the work, as the first large-scale randomised controlled trial (RCT) in ophthalmology in the country, served as a flagship trial, and established models and structures that have benefited subsequent studies, advancing ophthalmology research in the UK overall.

Finally, a number of case studies mentioned that the work had an impact on the teaching and training of clinicians (Carotid Stenosis,47 CESAR,49 EVAR,46 Newborn CHD,42 CUtLASS,51 CRASH-244). In addition, for the IVAN43 study it was suggested that the work led to an increased awareness of research among clinicians, and increased enthusiasm for, and interest in, initiating and participating in studies subsequently. This reflects, perhaps, the fact that the study43 was both large and a flagship trial in the field, as described above.

Interviewees suggest that there is little overlap between the majority of the research funded by the Health Technology Assessment programme and industry, as the programme intentionally does not fund research that industry would support

Typically, HTA programme-funded research is in areas for which there is little or no commercial incentive to carry out research, such as when a drug is off patent. As such, that means that typically industry involvement in research is limited in most cases (I6, I14). However, there are notable exceptions to this, and, in some cases, the research priorities of the HTA and the pharmaceutical industry conflict directly (I7, I14, I19). One example noted by an interviewee is the IVAN43 trial, in which a HTA study compared two drugs owned by the same company, one of which was not licensed for that indication. The IVAN43 trial is described in more detail as one of the case studies in Appendix 4. As another interviewee noted, the research that the programme funds will sometimes come into conflict with industry because it ‘challenges some of the assertions that it makes about the effectiveness and cost-effectiveness of its products’ (I7). STAs are an example of this, when TAR teams critique manufacturers’ submissions to NICE. This inevitably will lead to conflict with industry in some cases.

Given the focus of HTA programme-funded research, it is perhaps not surprising that the production of new intellectual property or the discovery of new molecules is relatively unusual, as suggested by one interviewee (I7). This is supported by the data available in Researchfish, which indicates that there are five instances of Interventional Procedures reported (in five separate awards), which focus around the production of resource materials and manuals that have been copyright protected.

More than half of case studies demonstrated impacts on industry and product development

Of the 12 case studies,41–52 seven41,42,45,46,49,51,52 indicated clear impacts on industry or product development, as set out in Table 13. Most of these impacts on product development were through the refinement of products or markets, rather than the development of new products. Several studies also suggest that there may have been an impact on industry through increased sales, which, although plausible, is hard to evidence. In addition, the TARs had an impact on industry via NICE, which is discussed in the next section.

The Health Technology Assessment programme also has an indirect impact on industry through the National Institute for Health and Care Excellence

Although the research funded through the HTA programme does not typically overlap with industry interests (with some exceptions), the programme does have an impact on industry through NICE (I3, I4, I8, I14). As one interviewee explains, it is difficult to disentangle the impact of the HTA programme from the impact of NICE here:

One of major outputs of HTA programme is the work for NICE, which impacts the pharmaceutical industry. Imagine the HTA programme without NICE . . . the pharmaceutical industry would not be as impacted upon. But people have been doing HTA for ages. In countries that don’t have an equivalent of NICE, people can take HTA reports or leave them . . . they just get diffused into the general literature.

I4

One of the ways that the impact of the HTA programme specifically can be observed is through its methodological impact on the pharmaceutical industry (I4, I8, I9, I14). Because of the work that NICE commissions through the HTA programme, the methods that NICE uses, particularly for the economic analysis, are those developed and used through the HTA programme. One interviewee explains this as follows:

Pharmaceutical companies are trying to get products licensed and the regulatory framework for research to achieve product licensing is out by different regulatory bodies. That is independent of the HTA, but the important connection with the HTA is through the work that the HTA programme commissions for NICE, which includes a lot of cost-effectiveness analyses in primary and secondary research, because the pharmaceutical industry is particularly concerned about NICE decisions, which means that they have to use the methods that are used for NICE to make its decisions. Those methods are the methods used by the HTA programme.

I9

The RA45 case study provides a useful illustration of these two ways in which HTA research, particularly TAR studies, can impact on industry, largely because of the influence they have on NICE. First, the study45 led to the inclusion of new drugs into NICE guidance, so the relevant companies will now be able to receive reimbursement on the UK market for those drugs. In addition, there is anecdotal evidence that the cost-effectiveness model developed and refined through this study45 has been used by some pharmaceutical companies in their applications for approval of similar therapies by NICE. This illustrates how the TAR studies can have methodological impacts on industry.

Similarly, the STAs52 are likely to have an impact on industry insofar as they influence NICE decisions and hence the drugs purchased by the NHS. There may also be individual examples of methodological impacts, although none is noted in the case study,52 and perhaps the scope for methodological advancement in the short turnaround of a STA is limited.

The cost-effectiveness analysis conducted in Health Technology Assessment studies can affect industry but also the economy more widely through more efficient allocation of NHS resources

The HTA programme may sometimes have a positive impact on industry when a particular drug, device or other treatment is shown to be cost-effective, but it may also have the opposite effect when it shows that a particular drug is not cost-effective (I7, I19).

However, this cost-effectiveness analysis has wider implications. The HTA programme has an indirect impact on the economy by providing evidence on the cost-effectiveness of different treatments, which policy-makers can use to allocate resources more efficiently (I3, I7, I19). This ultimately may have even further impacts on the health and hence productivity of the population as set out by one interviewee:

[The HTA programme] results in more effective use of healthcare resources, which both frees up some of the NHS’ money and has led to a healthier, potentially more productive, population.

I7

As described previously, two of the case studies43,48 provide some limited evidence of potential cost savings for the NHS or patients. The IVAN43 study is likely to have played a role in saving the NHS money by contributing to a reduction in the cost of Lucentis to the NHS through renegotiations, and SWET48 showed that water softeners are not effective for the treatment of atopic eczema in children, potentially saving patients money, as they can avoid spending money on treatments that do not work, although the evidence around this is unclear.

In addition, three of the case studies44,47,49 suggest use in the NHS of new effective (and cost-effective) treatments has led to health benefits for patients (CRASH-2,44 CESAR49 and Carotid Stenosis47). It may be possible to extrapolate this to economic benefits, through the improved health and productivity of the population, but to do so in any concrete way is extremely challenging. The CRASH-244 case study does allude to this. The study44 authors, in the HTA journal article, reflect on the groups most affected by trauma. It is noted that the poor are disproportionately affected by trauma, with the risk higher for more disadvantaged groups in the UK or internationally. It is also noted that the group that is predominantly affected by trauma is young men – the average age of participants in the CRASH-244 trial was 35 years old, and 85% of participants were male. They suggest that this means that the social and economic benefits of reducing death by trauma could be very significant. However, providing clear evidence of these type of benefits, or quantifying them, has not been possible within the scope of any of the case studies.

At a higher level, there is evidence from a previous study looking at a sample of 10 HTA programme-funded studies, that if 12% of the potential net benefit of implementing the findings of that sample of 10 studies for 1 year was realised, it would cover the cost of the HTA programme from 1993 to 2012. 33

Health Technology Assessment research is used outside the UK, particularly by other Health Technology Assessment organisations and those conducting systematic reviews

Research funded by the NIHR HTA programme is viewed as a reference resource by people internationally (I3, I16), and a lot of HTA organisations around the world rely heavily on NIHR HTA research (I3). This is because, in contrast with HTA programmes in most other countries, the programme does original research rather than relying on existing studies and analysing them (I8). One interviewee also described how ‘people in other countries that are in the HTA or systematic review business are often aware that there is an important HTA trial under way and are keen to see its results’ (I14). Several interviewees mentioned in particular a high number of web hits and downloads from the HTA website, including internationally (I7, I14, I16). Data provided by NETSCC show that the HTA website received 228,777 (unique) hits between March 2014 and February 2015, and that over the same period there were 111,102 documents downloaded, of which 93,879 were of full HTA monographs. It is not clear, however, how many of these were within or external to the UK.

It was generally felt that the programme is well known internationally, particularly among other HTA organisations (I3, I4, I7, I8, I10, I14, I16), with one interviewee describing the programme as having ‘a very strong brand’ (I19). However, by contrast, some interviewees (I3, I4, I9, I12, I18) expressed concern that there was confusion regarding the identity of the programme, both internationally and in the UK. In particular, there is confusion between the HTA programme and NICE, perhaps reflecting the fact that these roles are often combined in one organisation in other countries. There is also some confusion between the HTA programme and NIHR more widely, part of which may come from the fact that funding from the different NIHR streams is not always clearly delineated (I5).

One interviewee also noted that HTA research is often cited in guidelines outside the UK as ‘international HTA programmes turn to the HTA programme’s work to inform their own guidelines’ (I8).

Most case studies had some international impact, across academia, policy and practice

Of the 12 case studies,41–52 nine studies41–47,49,52 found evidence of international impact. As illustrated in Table 14, these spanned academia, policy and practice in particular. Each of these areas is explored in more detail in the following sections.

The Health Technology Assessment programme has an impact internationally through the international reputation of the National Institute for Health and Care Excellence

In addition to the influence of the HTA programme independently, it also has an impact internationally because of the reputation and influence of NICE internationally, influence which is amplified by NICE’s policy of providing free access to its publications. As described above (see The Health Technology Assessment programme also has an indirect impact on industry through NICE), part of the reason that NICE is so important to industry is because of its international influence (I3, I4, I8). One interviewee described how NICE had an impact on policy and practice internationally:

NICE’s influence is both national and international: other countries are setting up their own version of NICE or taking NICE’s views into consideration when designing their own policies. NICE made HTA programme influential.

I8

This is seen in the two case studies focusing on TARs,45,52 as described above, where any international impact was mediated by NICE rather being a direct impact of the HTA research itself. However, as described above, many of the studies directly impacted policy internationally. This is partly because of the sample of case studies selected. Many of them are important internationally known studies in their fields, which show significant results. As such, they are likely to impact on international policy directly. Impact via NICE as an important decision-making body with international recognition may be more significant for other less well-known studies.

The Health Technology Assessment programme has contributed to the cultural change in attitudes towards medical research, which has involved a paradigmatic shift towards evidence-based and, more recently, economic evidence-based medicine

Many interviewees commented on the role that the HTA programme has played in the shift towards evidence-based medicine (I6, I8, I9, I14, I16, I19). This includes not just its role in providing an important part of the underpinning evidence required to support an evidence-based NHS, but also in changing attitudes within the research community towards conducting this type of research, and its importance and validity. For example, one interviewee stated that:

The research establishment has completely changed its view about the validity and important of the HTA’s kind of work. The HTA isn’t the only driver of this change but it exemplifies the change and was at the centre of that change. The HTA did not cause the changed but it played a role in the general paradigmatic change in what is meant by good health research in the last 20 years.

I8

Another commented that:

[the HTA programme] has had a fundamental role to play in increasing the focus on clinical and applied research . . . it has also been important in changing perceptions about the importance of applied and practice based research, clinical and applied research. It has changed the views of the funding bodies and the universities about that.

I19

In addition, one interviewee commented:

In the 1990s and early 2000s, the HTA programme was one of the drivers behind the change in the attitude towards research in that research should be applicable, needs-led and impactful work.

I8

Interviewees also commented on the importance of the HTA programme in changing attitudes towards research in the NHS (I14, I19). For example, one interviewee stated that:

Initially, for the first 50 years of the NHS’s existence, it did not fund research. Now, it puts 1% of its budget into research. The HTA programme is only a small part of that but it’s the part that provides the justification for spending money on research.

I14

Another commented that:

for the people in the NHS, people feel that some research is very abstract and obtuse and a long way from practice, and the HTA programme answers questions that are deeply practical questions to people in the NHS and the patients that they serve.

I19

One interviewee noted that the HTA programme has also had an important impact on policy-makers, beyond its impact on policy, through helping to change the perception of policy-makers from thinking of research as very academic and esoteric to something that is practice and answers relevant questions (I19). This is, however, difficult to observe and evidence.

Comments were also made about the role the programme has played, alongside NICE, in the increasing focus not just on effectiveness, but also cost-effectiveness in medicine (I3, I8, I9, I16, I19). For example, one interviewee commented that:

Over the last 30 years or so, the whole way of thinking of medicine has changed to an evidence-based mode and, in the last ten years or so, to an economics of medicine evidence-based mode and the HTA programme is the principal vehicle.

I16

This is summarised in what one interviewee felt was the most important achievement of the HTA programme:

Changing perceptions of HTA research, resulting in a shift in esteem with which health service research was held, and making sure people take account of population effectiveness and cost-effectiveness.

I8

The Health Technology Assessment programme has had an impact on the wider Health Technology Assessment movement internationally

Interviewees suggested that the HTA programme is viewed favourably compared with other HTA programmes internationally (I8, I16), although many respondents note that the HTA programme is very different from other countries’ programmes because most other HTA programmes focus on only secondary research/systematic reviews (I3, I14). The HTA programme’s connection to policy-makers, particularly NICE, was seen as a particular strength of the programme compared with other countries (I16). Another important difference is the separation of functions between the HTA programme in NICE. In many countries, the appraisal and assessment functions are combined in one body (I6).

Broadly, the programme is considered to be a thought leader in the HTA space internationally (I10, I15, I19). For example, one interviewee described the programme as a ‘pioneer in this area’ (I10), and another set out that ‘a lot of its international standing is based on its academic strength’ (I19). One interviewee described how the UK HTA programme had played an important role in a wider international movement, increasingly recognising the importance of this type of research:

There’s a big international movement in health technology assessment and the HTA programme has played a big and leading role in that internationally . . . in helping governments to see the benefits of assessing health technologies and appraising them.

I15

Oversight from the Health Technology Assessment programme is generally considered proportional and supportive, and in several cases can be seen to directly contribute to the success of the work

The general sense across the case studies was that the level of oversight provided on the work was largely appropriate. For example, one interviewee from the ARTISTIC41 case study suggests that ‘they maintained some oversight, but it was very light touch’. Similarly, according to the CRASH-244 case study, the CI did not consider the monitoring to be disproportionate to the scale of the funding. However, one study did comment on increasing bureaucracy (EVAR46), referring in particular to delays in the contracting process for the follow-on work delaying progress.

Although the overall picture was similar, interim reporting requirements seemed to differ between studies. For example, for the ARTISTIC study,41 this consisted of submission of interim reports, meeting minutes from the Trial Steering Committee (TSC) and the independent Data Monitoring Committee. The CESAR49 and SWET48 studies suggested that the project had to submit annual reports, whereas for CoBalT50 these were every 6 months, and SWET48 was subject to a monitoring visit towards the end of the recruitment period. This may reflect changes in the process over time, or differences depending on the scale and nature of the research.

Many case studies provide evidence that the HTA was supportive. This could be through their intermediary role in the case of the TARs, which was highlighted as important to the TAR team in the case of the RA45 study by one of the researchers, as they were involved in negotiating with NICE when timelines were perceived to be too short. They also provided support for follow-on work for several studies. As an example, the IVAN case study notes that regular updates were provided, and the HTA programme remained aware of progress throughout, which meant that there were no concerns about granting a 3-month extension to the study43 when requested by the team (see Appendix 4). More generally, several studies include comments on the positive attitude of the HTA programme overall and commented favourably on the interaction with the programme. For example, one interviewee from the IVAN43 case study commented that ‘of all the funders I’ve ever dealt with, I’ve always found the HTA to be the most reasonable and interested in making the research happen. It was and continues to be an extraordinarily refreshing process compared to previous funders’.

Several studies also provide evidence that interaction with HTA programme management supported the successful completion of work or its impact. In the CoBalT50 study, this took the form of advice on how to conduct the study. The HTA group suggested ways to improve recruitment rates, which had been problematic, and this advice proved to be very helpful. In the case of the ARTISTIC41 study, this took the form of flexibility around the reporting of the study. One member of the team noted that being able to publish results in other journals ahead of the monograph being released allowed work to have a quicker impact. In the case of the IVAN43 study, there was some urgency in getting the project started because of the changing policy environment, and the case study43 notes that the HTA programme did expedite the process as much as possible to ensure that the study43 was able to take place.

The Health Technology Assessment journal is generally considered to make an important contribution to the academic impact of Health Technology Assessment programme-funded research

As described above, the HTA journal provides a way to ensure the research funded through the programme is published in full as standard. However, several interviewees (I3, I4, I6, I7, I16) commented on the quality of the journal in addition, noting its high levels of citation and the fact that it may be more widely picked up (and potentially by different audiences) compared with publication in a standard academic journal. One interviewee commented that ‘something that is published through the HTA programme is probably more read and more cited than a more standard paper in an academic journal’ (I4).

This is supported by the bibliometric evidence in Figure 10, which shows the MNJS for the publications analysed. This measure indicates the level of citation of journals in which HTA research is published normalised for research field. The average value is one, so values of ‘< 1’ indicate that research is cited more frequently than would be expected for that field. The MNJS indicates that the HTA journal is cited 2.31 times more frequently than other journals in the same field. The other journals that HTA-supported research is published in tend to be slightly less visible, with an average MNJS of 2.14 over time, but this is still more than twice the average for their fields of publication. The MNJS has remained fairly stable over time, with no obvious trend.

FIGURE 10.

Mean normalised journal score per year, for all publications from HTA programme-funded research and for the HTA journal only. Note: this is based on a sample of publications identified through three sources: all articles and reviews published in the HTA journal during the period 2004–12; papers listed on all HTA project pages of the NETSCC website; and papers reported by researchers in Researchfish. As such, this may not be a complete record of all publications related to HTA programme-funded research.

Also noted in interviews was the high number of downloads of the HTA journal (I3, I6, I16), including from outside the UK. Data provided by NETSCC indicate that over the year 1 March 2014 to 28 February 2015 there was a total of 111,102 PDF downloads, of which 93,879 were for the full HTA monograph.

However, some respondents questioned the added value of the HTA monograph to the independent publications resulting from the same research (I7, I14). For example, one interviewee suggested that ‘the monographs simply reproduce the academic publication or don’t go sufficiently beyond it to justify the production of the monograph’ (I14), and another commented that it is ‘hard to know impact of monographs over and above the journal articles’ (I7). One interviewee suggested that this should be monitored to ensure that the monograph does provide a lot more detail than other published outputs to avoid research waste (I14). However, one interviewee from the ARTISTIC41 case study stressed the importance of the HTA monographs. The interviewee noted that the monographs are invaluable to policy-makers because they provide much more detail than the academic publications. This should be contrasted, however, with the role the HTA journal plays in ensuring that virtually all HTA research is published as outlined above.

The Health Technology Assessment programme tries to ensure that its research delivers added value, although some respondents provided specific examples of duplication of research effort

The HTA programme, and the NIHR more widely, have adopted the ‘Adding Value in Research’ framework20 based on the work of Chalmers and Glasziou (2009). 14 As described above, one element of this in which the HTA programme is considered ‘exemplary’ (I1) is in the publication of the vast majority of its research. This is an area in which there is also suggestion from the interviewees that other research funders are trying to emulate the achievements of the HTA programme (I1, I10). Another area receiving praise was the use of systematic reviews in the determination of which primary research to fund (I1). One interviewee stated that they ‘don’t know of any research funding agency in the world that has done those two things systematically as the HTA programme’ (I1). Another interviewee pointed to flexibility with the HTA in their funding processes allowing them to save money, pointing to a specific example where they were open to alternative approaches which might offer the same information more cheaply than a clinical trial (I5).

Avoiding duplications of research is also important and an area where the picture is more mixed. Some interviewees pointed to the importance of some individuals in the HTA programme in ensuring that this does not take place (I1, I15), but others could identify specific examples where such duplication between funders had occurred (I1, I14).

One researcher also commented on the process of applying for HTA funding as being bureaucratic and not very user friendly, suggesting that they could benefit from ‘a user group to advise them on the experience’ (I14). This suggests that there may also be potential to prevent wasted time by streamlining this process, balancing, of course, the need for appropriate data collection and oversight.

The Health Technology Assessment programme is recognised as being one of the first public funders to require patient and public involvement and has continued to be viewed as a leader in this area; however, a number of respondents noted that the impact of that involvement is unclear and not well monitored

Several interviewees felt that the HTA programme had played a role in the development of the use of PPI in the UK and that it was an early actor in this area (I3, I5, I7, I8, I13, I19). One interviewee commented that ‘the HTA programme was the first national, publicly funded programme to insist on PPI’ (I8), another that ‘it’s probably one of the programmes with the longest standing relationship with involvement of patients and the public’ (I13).

In particular, the programme was praised for involving PPI throughout the research process, and several commented that it is a leader in this field in demonstrating that this is feasible. Two interviewees singled this out as the most important achievement of the programme: ‘giving PPI credibility’ (I8) and ‘demonstrating that a research funder can involve PPI throughout the research process’ (I13). One of the interviewees described how this process works:

The HTA programme has a lot of engagement: within its own processes has public and patients involved in all stages of the research, prioritising and funding process. Public/patients are on all of the boards and committees and conduct peer reviews. There is a patient and public steering group.

I7

However, despite this positive picture overall, there was some questions among some respondents around the impact of that involvement. In particular, at present it is hard to see how effective that PPI is, and what impact it has on the way research is funded and conducted. A lack of monitoring was mentioned by two interviewees, one commenting that:

What I can’t put my hand on my heart and say is: the impact of the PPI involvement is this . . . I can’t say that they always have an enormous impact. My gut-feeling is that, they do, but I can’t prove it.

I7

This lack of evidence around the effectiveness of PPI is not unique to the HTA programme. 59 However, the observation remains that although the HTA programme has been a leader in terms of its processes for PPI, the impact of its investment in PPI is less clear and does not appear to be well monitored.

Evidence from the case studies supports the notion that the HTA programme actively encourages the use of PPI in its studies, but it is again difficult to conclude anything about the impact of that PPI. There was much less formal PPI at the time many of these studies started, and the change in attitudes to PPI over the time period is noted in five45–47,50,51 of the case studies. Where it is mentioned, there is generally a positive attitude towards PPI, but it is hard to conclude anything quantitative from our sample about the value of PPI.

What we can say, however, is that nearly all of the studies involved some PPI. This tended to be focused on patients and their families, rather than the public. Six case studies showed significant involvement of patients for example through focus groups, protocol development, determination of outcome measures, dissemination strategies and steering committees (Newborn CHD,42 SWET,48 CESAR,49 CoBalT,50 IVAN,43 CRASH-244). For example, SWET48 involved the National Eczema Society and a local group of carers in the work, with a patient representative involved during all stages of project from design to write up and dissemination, and the CRASH-244 study involved patients in defining outcome measures and the development of protocol at start of study. Four of the studies stated that inspiration from patients fed into the studies through the clinical work of members of the team (CUtLASS,51 EVAR,46 CoBalT,50 Carotid Stenosis47). For example, in the CUtLASS51 case study it is noted that there was significant insight into patient need as result of the lead researchers being clinicians directly treating patients and in contact with families. In the case of the two TARs,45,52 PPI input was via NICE, with patient representatives contributing to TAR selection and sitting on the NICE Appraisal Committee. In particular, one study (Newborn CHD42) was remarked on as being at the forefront of PPI for the time.

Most PPI was through patient groups or appointed representatives. Of the six case studies42–44,48–50 with significant independent PPI, four studies42,43,48,49 engaged with patient and family groups and three studies48–50 used appointed patient representatives. Other forms of input, each used in one study, were practitioner input, the use of learning from patients in a previous project (around appropriate outcome measures), and use of learning from patients in pilot work.

The structures and relationships that the Health Technology Assessment programme has developed, linking its research to the National Institute for Health and Care Excellence and the National Screening Committee, are important in facilitating the impact of its work

As described in Chapter 3 (see The primary route to impact of Health Technology Assessment-funded research is through guidelines, particularly National Institute for Health and Care Excellence and National Screening Committee guidelines and National Institute for Health and Care Excellence and the National Screening Committee are key users of Health Technology Assessment research), the relationships that the HTA, at a programme level, has with NICE and the NSC are important in facilitating much of the impact of the work that it funds. This was identified in the previous study into the impact of the first 10 years of the programme by Hanney et al. (2007)1 and remains the case. In particular, the structures around priority setting and the TAR programme are important in ensuring that research addresses questions of relevance to these key users and that the findings impact on policy and, ultimately, practice.

The STAs are an interesting example of this, which have emerged over the last 10 years. As described in the case study, this stream of research was introduced with the aim of producing independent evidence more quickly to support NICE decision-making, and, through speeding up that NICE decision-making process, they facilitate faster impacts on practice, with potential benefits for both patients and industry.

Although the specific impacts come through the individual projects, by having these structures and relationships in place, the HTA programme enables its work to have a more direct impact than might otherwise be possible.

The Health Technology Assessment programme answers questions that would not be answered by other funders and fills important research gaps

The HTA programme’s stated aim is to ‘research information about the effectiveness, costs and broader impact of health-care treatments and tests for those who plan, provide or receive care in the NHS’. Interviewees felt that the programme was not only successful in doing this, but also that it focuses its resources on topics that would not be supported through other means (I3, I6, I8, I16, I18). As one interviewee describes it:

[The HTA programme] answers questions that aren’t going to be answered by other means of research, which are the basic research imperatives that fund the universities and pharmaceutical company research, which doesn’t usually answer quite the questions we want to know.

I16

Another interviewee stated that the programme ‘plugs a really important gap in research we need to make recommendations’ (I18), whereas one commented that the HTA programme is ‘funding things industry won’t fund’ (I3).

The HTA’s prioritisation process plays an important role in this and was praised by one interviewee as ‘[ensuring] really important questions that need really good quality research are addressed rigorously and made publicly available’. Another commented that the ‘philosophy of the programme has been consistent: NHS relevance’ (I6).

However, some interviewees criticised a lack of transparency in the priority-setting process (I13, I14). For example, one interviewee noted that although the HTA programme has a good relationship with the NSC, it does not keep a good record of the work that it does for the NSC (I14), and, similarly, noted that looking at HTA programme-funded studies, it is ‘very hard to identify the source of the topic and particularly to identify how many of them came from the public’ (I14).

An important development over the last 10 years has been the growth of researcher-led research, particularly alongside the growth of the primary research stream. Given the clear priority-setting agenda in the HTA programme, initially the programme had been entirely commissioned-mode funding. However, this caused some challenges, as set out by one interviewee:

Researchers had good ideas that they might not want to share with [the HTA programme] because they might not get the contract. A lot of researchers were not giving [the HTA programme] their best topics. By creating the responsive mode, they can keep ownership of topic themselves.

I6

However, the researcher-led arm of the portfolio is still subject to the same prioritisation process. The topic is reviewed in the same way that a topic under the commissioned arm would be and the researcher still has to make the case around why it is important to the NHS. As one interviewee commented:

The research questions come from the practice community rather than the academic or research community. That remains the core of the HTA programme, even though it has now added on the researcher-led stream.

I9

The Health Technology Assessment programme has a track record of commissioning work that may be controversial

Evidence from the case studies suggests that the HTA programme is not afraid to commission controversial work or studies in areas for which many stakeholders already hold strong assumptions about the outcomes, which, when contradicted, can lead to controversy. Many of the case studies are noted as being in some way controversial. In particular, two of the studies41,43 were commissioned in directly controversial environments (ARTISTIC,41 IVAN43). The IVAN43 study in particular is noted as being politically high profile from the start, with a number of discussions taking place in the House of Commons about the difference in cost between Avastin and Lucentis, and the availability of both on the NHS.

Several other studies were commissioned in situations for which stakeholders such as industry, patients, and clinicians had existing expectations about the likely outcomes of the study. When these assumptions were confirmed, such as in the case of the EVAR46 study, this facilitated uptake of the findings. However, where the assumptions are contradicted, there seems to be reluctance to change practice (e.g. CESAR,49 CUtLASS51). Although specific groups with vested interests in the study results can lead to an unpleasant environment for researchers – for example, one of the researchers involved in the CUtLASS study suggested that it took around 2 years for the findings to be accepted in the field, during which time he described himself as being ‘kind of ostracised’ by the pharmaceutical industry – these studies also have a significant potential for impact on practice when implementation can occur, as they are challenging existing assumptions and practices.

In addition to controversy around study findings, several case studies comment on controversy around the methods used in the studies. These tend to fall into to two groups. First, there are cases when there is also controversy because the results are unexpected or not as desired (EVAR,46 RA,45 CUtLASS51). In other cases, there is some controversy in another country (typically the USA) because the methods do not correspond to existing practice there. This reflects the NHS focus of the work and study designs, as described in Chapter 3 (see Study design can influence whether or not research is implemented in the NHS).

The Health Technology Assessment programme has had important impacts on patients through health policy and practice, and impact on practice can be direct as well as via the National Institute for Health and Care Excellence and the National Screening Committee

Across the programme as a whole, the HTA’s impact on patient care is primarily an indirect impact, mediated through guidelines. The case studies provide evidence of the types of impact that the HTA’s research can have on patients and the NHS, including changes in NHS practice, health benefits for patients and increased patient choice. However, it is sometimes difficult to determine the impact of HTA programme-funded research on clinical practice because HTA studies are typically commissioned when the evidence is unclear, such that many studies show that existing practice is appropriate, which we observed in two of the case studies. 48,50

Given the close links between the HTA programme and NICE through, for example, the TAR programme, it is legitimate to look at the joint impact of NICE and the HTA programme on clinical practice, and it is often not possible to attribute any observed impact to either organisation, as both contributed. However, the case studies illustrated a number of examples for which NICE was not the most effective route to impact for HTA research. For example, because of the interests of other groups, NICE has not yet been an effective route of implementation for the treatments assessed in the CRASH-244 and IVAN43 studies because of the conflicting interest of other groups (notably industry). Despite this, the results of the CRASH-244 study have been widely implemented.

For screening studies, the relationship with the NSC provides another important route to impact on clinical practice. Where evidence from the HTA programme suggests that screening for a particular condition is both effective and cost-effective, the national screening programme often then undertakes a screening pilot programme. Two examples of this are ARTISTIC41 and Newborn CHD42 case studies. The NSC pilot itself has an impact on the patients involved in the study, and any resulting national screening programmes then have an impact on all of the patients in the target population.

There are also other ways through which the HTA programme can have impact. As the major funder of clinical research in the UK, the programme may have an impact on the quality of care provided in the NHS through supporting a high volume of research in the NHS by increasing the skills of clinicians. However, the impact of research on clinical skill has not yet been well researched. There is also some evidence that patients benefit from participation in clinical trials. However, the primary impact of HTA programme-funded research on patients is through producing high-quality scientific evidence that results in improved guidance for clinicians which, if implemented, should improve patient care.

Interviewees and the case studies suggested that impact on practice beyond the NICE and NSC routes might be limited by the HTA programme’s dissemination strategy. It was recognised that this challenge of dissemination and adoption is not unique to the HTA programme. However, the overall dissemination strategy was considered limited and too academically focused. This challenge in dissemination was reflected in the case studies, where several note a lack of support for dissemination and implementation, which may have limited the impact of HTA programme-funded research on NHS practice. The case studies illustrated that this challenge was particularly pronounced where other stakeholders have interests or expectations that run counter to project findings. However, the case studies also illustrated that some of these obstacles can be overcome by active and enthusiastic project teams.

The Health Technology Assessment programme has an impact on UK policy, primarily through its close links with the National Institute for Health and Care Excellence and the National Screening Committee

The HTA programme funds robust scientific research that is clearly and deliberately linked to policy-makers. Many interviewees viewed the evidence that the HTA programme produces on the effectiveness and cost-effectiveness of treatments as invaluable to policy-makers. Of the 12 case studies,41–52 10 studies41,42,44–49,51,52 indicated evidence of some impact on UK policy. The main routes to policy impact were through citation in clinical guidelines, typically NICE guidance and NSC pilots, which supports the finding that NICE and the NSC are key users of HTA research. The programme has close relationships with both NICE, particularly through the TAR programme, and the NSC. These findings regarding the impact of HTA programme-funded research on health policy are consistent with the findings of Hanney et al. (2007)1 on the impact of the HTA programme from 1993 to 2003.

However, we found that there are also other users of HTA research, including other guideline producers such as SIGN. The shift in focus towards primary research may also have an impact on the appropriate audience for HTA research, shifting it upstream from policy-makers towards specialist clinicians, other researchers and systematic reviewers. The case studies also provide examples where there are other routes to impacts on policy and practice. The impact of both the CRASH-244 and IVAN43 studies on NICE guidance has been hindered by the lack of incentive for industry to seek a marketing authorisation for the drug in question, but, in the CRASH-244 case at least, alternative routes to impact on NHS policy and practice have been found.

Health Technology Assessment research has an impact on policy and practice

Health Technology Assessment research is used outside the UK, particularly by other HTA organisations and those conducting systematic reviews. This is reflected in the case studies, with most of the studies having had an impact on policy and practice internationally. In the majority of cases, this impact has occurred directly as a result of the study itself, which was important in the field and hence is directly referred to by international sources.

The programme also has an international influence through its impact on HTA practice. The NIHR HTA programme is considered to be a thought leader in this area, and plays an important role in a wider international movement that increasingly recognises the importance of this type of research. This international influence is reflected in the case studies, with impacts on research internationally stemming from the sharing of results, tools and practices, and developing the field both through capacity building and the targeting of future research. In addition to the influence of the HTA programme independently, it also has an international impact through the reputation and influence of NICE internationally.

The work of the Health Technology Assessment programme is highly cited and considered academically rigorous, but the academic impact of the Technology Assessment Reports may be limited

The HTA programme has made a substantial contribution to health research through the publication of the monographs in the HTA journal, as well as by encouraging independent publication in other peer-reviewed journals. The work produced by the HTA programme is academically rigorous research, which is reflected in the bibliometric data both at the project level and across the programme as a whole, with citation levels more than double the expected level for the field. This finding regarding the academic quality of the research is also consistent with Hanney et al. ’s (2007)1 assessment of the impact of the first 10 years of the HTA programme from 1993 to 2003.

The HTA journal makes an important contribution to the academic impact of HTA programme-funded research, ensuring that all research is published in full, making much more detail available than would typically be present in academic publications, without restricting the ability to publish findings elsewhere. These observations about the HTA journal were reflected in the case studies, where most of the studies published academic articles outside the HTA journal publication, and both those articles and the HTA journal articles were typically highly cited.

One notable exception is the TAR stream of HTA research, for which publication opportunities are limited as a result of the nature of the research, in terms of the content, much of which is provided by industry in confidence, and in terms of the tight timelines for delivery of the work and the changing focus of researchers moving from one topic to another. STAs are not typically included in the HTA journal and finding other routes to publication can be challenging. However, these concerns should be balanced with the benefits that this stream of work offers, particularly in delivering timely evidence with a direct link into the policy-making process, which allows changes in practice to happen quickly.

Capacity building is not limited to members of the research teams, reflecting the importance of the programme as a funder of UK clinical research

The HTA programme is viewed as an important funder of clinical research, which has had a positive impact on both the careers of HTA researchers and on research capacity to carry out high-quality HTA research in the UK. The HTA programme has made a substantial investment in clinical research, and in HTA research in particular, which has played an important role in building and retaining HTA research capacity in the UK. This finding is consistent with the findings of Hanney et al. (2007),1 who described the programme as ‘an important and independent funding source’ (p. 73) and suggested that in several cases researchers felt that finding funding from other sources for particular trials would have been difficult.

Looking at the evidence from the case studies, all of the studies had some capacity-building impacts for the individuals directly involved in the study, although scope for this was sometimes limited because the researchers already had well-established research careers. However, the studies provided important examples of wider capacity building outside the study team by sharing methods and expertise in the conduct of clinical trials, the development of networks, and building up research capacity in a new or emerging area (e.g. by bringing more researchers into the field). Most of the case studies shaped future research in the field, either in terms of research priority setting or the development of new research methods.

The Health Technology Assessment programme has had broader impacts on the research system

As well as the direct academic and capacity-building aspects of the programme, the HTA programme has had an impact on the health research system more widely, through changing attitudes towards research, particularly HTA research, and through funding work addressing the needs of the NHS that would not have been supported by other funders.

The HTA programme has contributed to the cultural change in attitudes towards medical research, which has involved a paradigmatic shift towards evidence-based and, more recently, economic evidence-based medicine. This includes not just its role in providing an important part of the underpinning evidence required to support an evidence-based NHS, but also in changing attitudes within the research community towards conducting this type of research and its importance and validity. Interviewees also commented on the importance of the HTA programme in changing attitudes towards research in the NHS, largely because of the way that it addresses questions that are of NHS relevance. Interviewees commented on the role that the programme has played, alongside NICE, in the increasing focus not just on effectiveness, but also cost-effectiveness in medicine.

The significance of the HTA programme compared with other programmes in some of these changing attitudes is perhaps reflected in the nature of the research that it funds, which would not likely be supported by other funders. The research is not just NHS focused, but aims to focus resources on topics that, although valuable to patients and the NHS, would not be supported through other means, including topics that are not of interest to industry or academia.

Provide targeted support for dissemination

Findings from the interviews and case studies suggest that the dissemination of HTA programme-funded research could be improved. The case study evidence in particular suggests that a targeted approach to dissemination resources could be the most effective approach. Based on evidence from the case studies, an approach that allocates fundings after the completion of the research may be the most effective, as it would allow support for dissemination to be focused on projects for which it would provide most value. For example, if the results indicate that existing practice is appropriate, there is no need for the provision of dissemination resources to ensure that the findings are implemented in practice.

The resources provided for dissemination should also take into account the context of the study and the extent to which other stakeholders are likely to champion (or oppose) the study findings. The case study evidence suggests that when there is no champion for dissemination, a role often played by industry, research is more difficult to implement. Also, when results are counter to the expectations and/or desires of stakeholders, including patients, clinicians and industry, uptake may be slower.

By taking a targeted approach at the end of studies to allow the team to remain engaged in the active dissemination of the work, the HTA programme is likely to increase the impact of its work on health policy and clinical practice. However, there are some limitations to this approach. The inclination and skill of researchers in dissemination activities will vary and there may be legal and regulatory obstacles that researchers cannot overcome on their own. As such, the support needed from the HTA programme management may include their influence and input as well as their financial support.

Maintain close relationships with the National Institute for Health and Care Excellence and the National Screening Committee, but also consider working more closely with other policy-making organisations

The close relationships that the HTA programme has with NICE and the NSC contribute to the programme’s impact on health policy and clinical practice. However, the relationships operate quite differently in each case. For the NSC, the links seem more informal but nonetheless direct and effective in both the communication of research needs and the sharing and implementation of findings. For NICE, the relationship consists of many elements, including formal links to the TAR programme, for which the HTA commissions work on behalf of NICE to support its technology assessment process, but also more informal links with other parts of NICE, which inform wider HTA priority setting and commissioning, as well as parts of NICE involved in the preparation of clinical guidelines. These relationships are crucial to the impact that the HTA programme has on policy and practice, and should be maintained. However, it is worth considering whether or not the input of these two important policy-making organisations (and indeed other stakeholders, including members of the public) is sufficiently transparent, particularly with regard to priority setting. Better record keeping and disclosure might allow the programme to better demonstrate how and why it is generating its research priorities. This is discussed further below (see Consider funding research on the implementation of Health Technology Assessment-funded research).

It should also be noted, however, that NICE and the NSC are not the only routes through which the HTA programme can have an impact on policy and practice, as illustrated in several of the case studies. Being aware of other potential audiences – including clinicians, systematic reviewers and guideline producers – will allow the programme to ensure that it is also meeting the needs of these wider groups and thus increasing its potential impact.

Maintain the academic quality of the work and focus on NHS need

The combination of research that is both academically rigorous and of relevance to the NHS was noted as an important feature of the HTA programme. The work is academically rigorous and this is reflected in its bibliometric analysis, but it is also tailored to the needs of policy-makers and the NHS. This balance has been a feature of the HTA programme, as its inception and maintaining it is important in maintaining the impact of HTA programme-funded research. Maintaining the practicality of the work without the rigour would raise concerns regarding the use of the evidence in decision-making and would also lower the academic esteem for, and engagement with, the programme and potentially this type of research. Maintaining the quality without the practicality would significantly affect the programme’s ability to have an impact on health policy and clinical practice. Maintaining both the quality and relevance of the HTA programme’s work is therefore crucial to the future success of the programme and its impact.

Maintain good relationships with researchers and flexibility in the way the programme supports research

As well as being an important funder for academics in clinical research, and HTA research in particular, researchers consulted in the case studies were positive about the contribution of the HTA programme and praised the level of oversight, supportiveness and positive interactions with programme management. Some of the case studies also provided specific examples of how the interaction with programme management and oversight directly contributed to the success and impact of the research. Another important element for academics was the level of flexibility and academic freedom that the programme offers. However, one interviewee reported that the programme’s flexibility has decreased over time. Maintaining these good relationships and positive interactions with researchers is likely to be beneficial in terms of ensuring that communication is maintained and the programme is best able to facilitate the impact of the work it funds.

Particular consideration needs to be given to the TAR programme, which although beneficial in terms of providing a direct link to NICE and a clear route to impact on policy, has proved challenging for academics. Although working with the HTA and NICE is beneficial to academic institutions in terms of the prestige that it offers, interviewees expressed concerns about the fast turnaround of work, the range of topics and the limited opportunities for academic publication. Different TAR centres operate in quite different ways and it may be that there is potential for learning between them about how best to manage the demands of conducting this particular type of work alongside pursuing other academic research. For example, one of the centres described how they had found a route for publication for STAs, which perhaps others could emulate. Similarly, different centres had different numbers of staff involved in conducting TARs and thought could be given to the best balance there to allow staff to pursue other interests while also making sure that teams conducting TARs have the availability and skills to do so. The HTA programme could help facilitate this learning, and think about how the TAR programme could be better supported to allow academics to contribute sustainably to this important source of evidence for policy-makers.

Consider funding research on the implementation of Health Technology Assessment programme-funded research

It was noted in several case studies that even although an intervention had been shown to be cost-effective, there were still cost barriers to implementation. This is because the HTA studies do not usually consider practical issues around implementation in the NHS – such as training and infrastructure – and the economic analyses consider the operation of the approach as standard practice, not the upfront costs of implementation. This can be important, particularly where HTA research is serving as an important input to NICE decision-making. The programme may wish to consider whether or not, and how, the programme could more fully address these issues to make the findings of its work more complete from an implementation perspective. One way to do this would be by providing funding to expand studies in which the evidence suggests that there may be a need for a change in practice, or where these costs seem significant enough to affect policy and practice decisions. This would allow the programme to provide better information to decision-makers and better facilitate the uptake of findings where appropriate. Selection of the relevant studies for this type of extension could be on the basis of the importance of such an update for NHS policy and practice, using the same prioritisation approach as used throughout the programme, based on the suggestions of relevant stakeholders.

Improve the transparency of the priority-setting process and monitoring the impacts of patient and public involvement

The HTA programme is recognised as being one of the first public research funders to require PPI in research and has continued to be viewed as a leader in this area. Attitudes and approaches to PPI have changed over the last 10 years, but almost all of the case studies investigated had some PPI. However, what is not clear, either at the programme or the project level, is what the impact of that PPI has been. Views regarding the level of PPI and impact of PPI were generally positive, and although record keeping around what was done in terms of PPI is in place for more recent studies, there is little information about what changed as a result of PPI and, as a result, the evidence of the impact of that change. Increased transparency, monitoring and measurement of the effectiveness of PPI would not only allow the programme to better demonstrate its commitment to PPI, but also would allow programme management to improve and develop its PPI strategy to increase its impact on the effectiveness of the research that it funds.

A similar argument applies to the priority-setting process for the HTA programme. There is overall information provided at the programme level about what the priority-setting process consists of, and that input is taken from policy bodies, clinicians, researchers, patients and members of the public. However, it is difficult to trace the origins of a particular piece of commissioned research, and, indeed, little information is available about the relative importance of the different sources of project ideas. For example, it is not clear how often, if ever, suggestions from the public reach the point of being commissioned as a study. Increased transparency and monitoring here would, again, not only allow the programme to better demonstrate the quality of its priority-setting process, but also would give programme management the ability to better understand the effectiveness of that process and how it could be improved.

This recommendation reflects the findings of previous work. For example, Raftery and Powell (2013)11 note that given the programme’s needs-led research focus, it has always sought the opinions of clinicians, policy-makers and patients, and that the methods of doing this should be assessed and reviewed on an ongoing basis. Hanney et al. (2007),1 in the previous assessment of the programme, also suggested that there was a need to maintain more detailed records of the process for prioritisation and tendering of research topics, noting that the records kept were not sufficient for monitoring and evaluation of the successes and failures of the programme.

Consider ways to protect the future of the programme through improved recognition and planning for change

Looking forward, the HTA programme faces a range of potential challenges. In times when the NHS is facing increasing budgetary challenges, its budgetary pressures are likely to increase for any elements of the health system that are not delivering front-line care. Being able to provide evidence on the effectiveness and impact of the programme will be important as the programme looks to secure funding, as noted previously by Raftery and Powell (2013). 11 The need to clarify the role of the programme relative to other bodies, such as NICE and NIHR as a whole, will be important. As discussed in Chapter 3 (see Health Technology Assessment research is used outside the UK, particularly by other Health Technology Assessment organisations and those conducting systematic reviews), there is some evidence that the brand of the HTA programme is not always well known and understood outside those working closely with the programme, which may be a risk to its future status. There is also a need to consider succession planning. Several key individuals have played important roles in the success of the programme. If succession planning is not already under way, thought needs to be given to how the departure of key individuals will be managed. It will also be important to continue to change and adapt to the needs of a changing NHS to ensure that the research the HTA funds is timely and relevant to the needs of clinicians, policy-makers and patients. The HTA programme seems to have been successful at adapting to the changing needs of the NHS over the last 20 years, and the programme will need to maintain this adaptability to ensure that it meets the future needs of the NHS.

PPI questions

1. How effectively does the HTA programme engage and involve patients and the public?

2. At what stages are the public involved (prompts if needed: project design, execution, dissemination)?

3. Does the HTA address the right topics that matter to patients? What does it miss?

4. How has the HTA’s approach to PPI changed over time?

5. How does it differ from other research funders?

6. What does the HTA programme do well in terms of PPI?

7. What could it do to improve PPI?

‘Customer’ questions (just ask about their organisation – select as appropriate)

1. How does NICE/NSC/SIGN use HTA research?

2. How important is HTA research to NICE/NSC/SIGN?

3. How has this changed over time?

4. Is NIHR HTA evidence widely used in guideline development (e.g. for NICE – outside of TARs)?

5. What could be done to make it more useful?

6. What do you think the relationship between NICE/NSC/SIGN and the HTA programme should look like?

7. Optional: How widely are guidelines used in practice?

8. Is NIHR HTA research used in practice through other means (e.g. directly implemented on a local level)?

9. What does the HTA programme do well in terms of dissemination and communication of its research?

10. What challenges are there in the communication and uptake of HTA research?

NHS/health system questions

1. What impact does the HTA programme have on practice in the NHS?

2. Does the HTA programme address NHS priorities effectively?

3. Are HTA studies directly taken up in practice?

   1. If so, how does this happen? What barriers are there? What enablers?

4. How widely are guidelines and other NICE/NSC products (e.g. TARs) used in practice? What influences this?

5. What could the HTA programme do better to achieve its mission of providing information on effectiveness, costs and broader impact of healthcare treatments and tests to those who plan, provide or receive care in the NHS?

6. What does the HTA programme do well?

7. What challenges are there in the communication and uptake of HTA research?

Industry questions

1. What impact does the HTA programme have on the pharmaceutical industry (e.g. drug development, research priorities, etc. )?

2. What impact does the work of the HTA programme have on the research conducted by the pharmaceutical industry?

3. Does the work of the HTA programme inform your industry’s research agenda?

4. How well does the HTA programme interact with research programmes in industry? What are the challenges?

International questions (only for those with international Health Technology Assessment expertise)

1. How does the HTA programme compare to similar organisations in other countries?

2. What does it do well compared to other countries?

3. What could it learn from others?

4. To what extent do you think the NIHR HTA programme has had an impact outside of the UK, if at all?

   1. Does the HTA programme have an influence on international practice in this field? (Examples, which could be used as prompts if necessary and relevant: Do you require systematic reviews before conducting primary research? Are your protocols openly published? What proportion of your research leads to a publication, including negative results?)

   2. Is UK HTA research taken up elsewhere internationally?

   3. Any other impacts?

Internal/strategy questions

1. What is the intended impact of the HTA programme? How was that determined and developed?

2. What strategy does the programme take to achieve that impact?

3. How has this changed over the last ten years?

4. How far do you think the intended impact has been achieved?

5. What particular challenges does the HTA programme face? How have these changed over the last ten years?

6. What are the HTA’s priorities looking forward? How do you/does the programme plan to achieve them?

Academic questions

1. Have you ever applied for HTA funding? If so:

   1. When and what for?

   2. Can you describe the process?

   3. Was the process satisfactory?

   4. How did it differ from applications to other funders?

2. Have you ever been awarded HTA funding?

   1. What requirements were placed on that funding?

   2. What was your interaction with programme management like?

   3. How did it differ from awards from other funders?

3. Would you consider applying for HTA funding (again)? Why/why not?

Political questions

1. What are the perceptions around the HTA programme in the Department of Health and in government more widely? (Possible follow ups: Is it considered to offer good value for money? What are the intended aims of the programme from a government perspective? Has it made the case for its continued funding?)

2. What influence does HTA research, or the programme more widely, have on policy, if any?

3. If it does have an influence, how does this happen? (Possible follow ups if needed: How is the research communicated? To whom? How is it used?)

Other funder questions

1. What overlap is there between your work and the work of the HTA programme?

2. Do you communicate regularly with HTA programme management?

3. What influence, if any, does the HTA programme have on your research strategy and portfolio?

4. What influence, if any, does the HTA programme have on your research processes? (Examples, which could be used as prompts if necessary and relevant: Do you require systematic reviews before conducting primary research? Are your protocols openly published? What proportion of your research leads to a publication, including negative results?)

5. What could the HTA programme learn from you?

Summary

The ARTISTIC trial41 evaluated the effectiveness and cost-effectiveness of HPV testing in primary screening, as either an adjunct to cytology or as a stand-alone test, compared with the current screening programme, which relies on cytology alone. The results from the first two screening rounds found that HPV testing did not significantly improve the effectiveness of liquid-based cytology (LBC) and that it would not be cost-effective to screen with both HPV testing and cytology, but the trial also found that HPV testing is highly effective as a primary screening strategy. The follow-up study, which included data from three screening rounds, concluded that HPV testing, as a primary screening test, was significantly more protective over three screening rounds than current practice (cytology), and that the use of HPV testing in primary screening could safely allow the screening interval to be lengthened. The cost-effectiveness analysis found that primary HPV screening would probably be cost-saving in both vaccinated and unvaccinated cohorts.

The ARTISTIC41 trial resulted in the publication of two HTA monographs and eight peer-reviewed publications. It also had a positive impact on the career of the Principal Investigator (PI) and a number of other individuals involved in the study,41 which can be viewed as a proxy measure for an increase in skills and capacity building as a result of taking part in the HTA programme-funded trial. The ARTISTIC41 trial has had an important academic impact, particularly through its contribution to the pooled analysis of the four other European HPV screening trials, which showed that HPV testing as primary screening for cervical cancer results in a decrease in the incidence of cervical cancer. The ARTISTIC41 trial also resulted in the piloting of HPV testing as primary screening for cervical cancer, which may, in turn, result in national roll-out of HPV testing. Finally, the ARTISTIC41 trial, together with the other European trials, has had an impact on the development of new HPV tests internationally.

Introduction to case study

Stage 0: topic/issue identification

A number of key factors influenced the research team’s decision to work in this area, as detailed in Box 1 and described below.

Interface A: project specification and selection

Henry Kitchener and Julian Peto initially submitted separate applications to conduct primary research into the role of HPV testing in primary screening for cervical cancer. However, according to one interviewee, in the second round of the application, the HTA programme asked the two research groups to put in a joint application, which resulted in the funding of the ARTISTIC41 trial. According to one of the interviewees, Kitchener’s team had substantial clinical expertise, whereas Peto’s team had substantial epidemiological and statistical expertise. Henry Kitchener, Julian Peto, Stephen Moss, Robin Dowie, Gerald Corbitt, Mina Desai, Peter Maguire and Chris Roberts were all involved in the design of the study protocol. 65 It has not been possible to determine how many other applications were made to the HTA programme; however, as the HTA put out an open call to tender, it is likely that the application process was competitive. One interviewee noted that asking the two research teams to submit a joint application ultimately proved to be beneficial:

The success of the HTA, through combining [Julia Peto’s] bid, with Henry Kitchener’s, is perhaps underplayed by the HTA.

The ARTISTIC41 trial was a randomised trial that compared cervical cytology with cytology as an adjunct to HPV testing, over three screening rounds, which were 3 years apart. The first round of screening was intended to detect prevalent disease (the proportion of the population with HPV), whereas the second round was intended to detect a combination of incident disease (the proportion of the population that acquired HPV over the screening interval) and undetected prevalent disease from the first round (the proportion of the population with HPV in the first screening round that remained undetected until the second screening round). The third round looked at the effectiveness of HPV testing over a longer time period. The trial used LBC instead of conventional cytology, as LBC was set to be introduced into the NHS shortly after the start of the ARTISTIC41 trial. One interviewee noted that the NHS Cancer Screening Programmes advocated the use of LBC in the trial so that the results would be relevant to what was expected to be current practice in the NHS by the time of publication.

The research team did not have any interaction with patients or policy-makers in the design of the research. However, the ARTISTIC41 trial was designed to inform cervical cancer screening policy from the outset and the trial was embedded within the existing NHS Cervical Screening Programme.

Stage 1: inputs to research

Stage 2: research process

The ARTISTIC41 trial set out to evaluate the effectiveness and cost-effectiveness of HPV testing in primary screening as either an adjunct to cytology or as a stand-alone test, compared with the current screening programme, which relies on cytology alone. 65 HPV testing was undertaken in both arms of the trial,41 but concealed from those randomised to standard care with cytology only to allow the evaluation of HPV testing as a stand-alone test. The cohort data were then use to model the outcomes of three different screening strategies: cytology alone, HPV testing alone and cytology combined with HPV testing. The study41 concluded that HPV testing did not significantly improve the effectiveness of LBC and that it would not be cost-effective to screen with both HPV testing and cytology, but that HPV testing is highly effective as a primary screening strategy. According to one interviewee, a better trial design would have involved all trial participants being screened with both a HPV test and cytology.

The ARTISTIC41 trial used LBC instead of conventional cytology, whereas the four other European trials used conventional cytology. In LBC, cervical cell samples are collected in the same way as conventional cytology, but a brush-like device is used instead of a spatula. 202 The sample is then placed into a vial of preservative fluid so that most of the cervical cells are retained. The samples are then mixed in a laboratory to disperse the cells. The cellular debris (e.g. blood and mucus) is then removed and a thin layer of cervical cells is placed on a microscope slide, which is then stained. According to Kitchener et al. (2009),65 ‘the principal advantages of LBC are a major reduction in inadequate samples for reading and more rapid throughput of samples in laboratories’. At the time of the ARTISTIC41 trial, the NHS Cancer Screening Programme was piloting LBC for cervical screening. According to one interviewee, the ARTISTIC41 trial was run on LBC so that the results would be relevant to what was expected to be current practice in the UK by the time the trial would be published. National conversion from conventional cytology to LBC was completed in 2008. 203

According to one interviewee, the use of LBC complicated the interpretation of the trial results. As LBC had not yet been introduced in the UK at the time of the ARTISTIC41 trial, the cytologists involved in the trial had to be trained to use the new technology, which meant that they were over-cautious and found 16% of smears to be abnormal. The same interviewee also noted that the sensitivity of cytology to detect CIN3 (Grade 3) improved after the switch from conventional to LBC, which probably resulted in previously missed lesions being detected in the ARTISTIC41 trial. The other European trials that did not use LBC observed a larger difference between HPV testing and cytology. 204

Although the research team did not have any interaction with patients while conducting the study,41 they did have some interaction with policy-makers and the HTA programme. Henry Kitchener was a member of the Advisory Committee on Cervical Screening during the trial, and updated the Committee on the trial’s progress. According to one interviewee, policy-makers awaited the results of the trial and viewed the results as important. The research team also had some interaction with the HTA programme during the study,41 which involved the submission of interim reports, and meeting minutes from the TSC and the independent Data Monitoring Committee. According to one interviewee, ‘the HTA maintained some oversight, but it was light touch’.

The research team undertook a follow-up study, which started in 2008, and analysed the results of the extended follow-up of the trial cohort after a third screening round. 201 The objective of the follow-up study was to provide further insight into the duration of protection of a negative HPV test, through the use of long-term data, to determine whether or not it would be possible to lengthen the screening interval. In the extended follow-up study, all women underwent cytology, HPV testing and genotyping. According to one interviewee, the genotyping of all of the HPV-positive samples in the trial was also a unique feature of the ARTISTIC41 trial. The follow-up study concluded that HPV testing as a primary screening test was significantly more protective over three screening rounds than current practice (cytology), and that the use of HPV testing in primary screening could safely allow the screening interval to be lengthened. 201 The cost-effectiveness analysis found that primary HPV screening would be cost-saving in a number of different screening strategies for both vaccinated and unvaccinated cohorts.

Stage 3: primary outputs from research

Knowledge

The ARTISTIC41 trial resulted in numerous publications, including: two HTA monographs;65,201 a paper reporting on the results of the first two screening rounds;62 a paper reporting on the results of the third round follow-up;205 a paper on the efficiency of cervical smear readers;206 a paper on the well-being of cytoscreeners;207 an epidemiological study on the prevalence of HPV;208 an epidemiological study on the prevalence of different high-risk strains of HPV;209 and a paper on the psychosocial impact of HPV testing on women. 210 Figure 11 presents the results of the bibliometric analyses on the publications resulting from the ARTISTIC41 trial.

FIGURE 11.

The ARTISTIC41 bibliometric analyses.

The Kitchener et al. (2009)62 paper and the first HTA monograph65 report on the results of the first two screening rounds of the ARTISTIC trial. Those results indicated that LBC combined with HPV testing resulted in a significantly lower detection rate of CIN3+ lesions in the second round of screening than with LBC alone, but that the effect size was small. Over the first two screening rounds, LBC combined with HPV testing did not detect a higher rate of either CIN3+ or CIN2+ lesions than LBC alone. The economic analysis found that it would not be cost-effective to screen with both LBC and HPV testing, but that HPV testing as either a triage tool or an initial test triaged by cytology would be cheaper than cytology alone.

The Kitchener et al. (2011) paper205 and the second HTA monograph201 report the results of the third screening round. It was found that the additional sensitivity of HPV testing compared with cytology could permit the lengthening of cervical screening intervals because a negative HPV test would provide protection over a longer period of time than cytology. 205 The study205 concluded that a negative HPV test was significantly more protective that LBC over the three screening rounds of the trial and that, as a result, the primary screening interval could be increased from 3 years to 6 years if HPV testing replaced cytology. The economic analysis found that HPV testing would be more effective than cytology and save costs. 201

The Dowie et al. (2006)206 study on the efficiency of cervical smear readers using LBC found that LBC can improve laboratory efficiency. However, the study206 also noted that decision-makers should consider the costs and benefits of introducing LBC into screening programmes, including the upfront cost of capital investment and the workforce implications for cytoscreeners.

Dowie et al. (2006)207 looked at the well-being of cytoscreeners in the NHS. The study207 found that there was a strong negative correlation between cynicism, or an indifference to work, and overall job satisfaction. Cynicism among cytoscreeners increased over the period in which the Greater Manchester laboratories partially converted to LBC.

The Kitchener et al. (2006)208 paper reports on the prevalence of high-risk HPV in relation to age, cytology and histology upon entry into the trial. Of the 24,510 women screened, aged 20–64 years, the cytology results were 87% normal, 11% borderline or mild, 1.1% moderate and 0.6% severe dyskaryosis or worse. 208 Kitchener et al. (2006)208 also found that the prevalence of HPV decreased sharply with age, but increased with cytological grade. Lastly, the study208 found that the majority of young women in the Greater Manchester area had been infected with a high-risk strain of HPV by the age of 30 years, such that the introduction of HPV testing as a routine screening test in women aged < 30 years would result in a substantial increase in retesting and referral rates.

Sargent et al. (2008)209 report on the prevalence of type-specific HPV infection by age and grade of cervical cytology. Among the 24,510 women screened, aged 20–64 years, the most common strains of HPV were HPV16, HPV18, HPV31, HPV51 and HPV52. Those strains accounted for 60% of all HPV cases detected. Although the prevalence of HPV declined with age, the proportion of cases caused by each strain did not vary with age. Multiple infections were common in those aged ≤ 30 years. Lastly, the study209 found that catch-up vaccination would probably reduce the number of women with moderate or worse cytology, but that it would not substantially reduce borderline to mild cytology.

The Kitchener et al. (2008)210 paper reports on the psychosocial impact of HPV testing on women in primary cervical screening. The study210 found that a revealed high-risk HPV test result did not have a significant impact on women with a revealed HPV-positive test result compared with those women with a concealed HPV-positive test result. The study210 concluded that HPV testing does not add significant psychological distress to cytology in routine primary cervical screening.

Finally, Sargent et al. (2010)212 report on the optimal threshold for a positive Hybrid Capture 2 test, used in the trial for the detection of high-risk HPV. The study212 found that a relative light unit/cut-off ratio of ≥ 2 achieved an increased specificity in the detection of CIN2+ lesions compared with the manufacturer’s recommended ratio of 1, which was achieved without a clinically significant loss of sensitivity.

Interface B: dissemination

According to the interviewees, the research team primarily disseminated the findings from the ARTISTIC41 trial through academic publications (described above) and conference presentations. The team presented their research at academic conferences, including the European Research Organisation on Genital Infection and Neoplasia (EUROGIN), and the International Papilloma Virus Society Conference. The research team also presented the results to the Advisory Committee on Cervical Screening to ensure that the Committee was aware of the study’s41 findings. However, one interviewee stressed that ‘by far, the most prominent method of dissemination was the publications in good journals’.

Stage 4: secondary outputs

According to one interviewee, the Sargent et al. (2010)212 paper on the optimal threshold for the Hybrid Capture 2 Test for HPV detection had a direct impact on policy. The paper showed that changing the cut-off from the manufacturers’ recommendations would increase the relative sensitivity of the test so that it would be more clinically useful. This finding was then translated into practice in the NHS. However, the interviewee noted that the Hybrid Capture 2 Test has now been superseded by other diagnostic tests but that, nevertheless, at the time, the paper had an impact on NHS practice.

The ARTISTIC41 trial also led to the undertaking of a national pilot of HPV primary screening in the UK, which began in 2013. The trial41 was cited in the NSC meeting minutes in April 2012. 219 The minutes state that ‘Members also asked about the cost-effectiveness of HPV TaPS [HPV Testing as Primary Screening]. Professor Patnick said the ARTISTIC trial had looked at both clinical and cost-effectiveness but further modelling would be needed as part of the feasibility study’. Additionally:

The UK NSC agreed that there is enough evidence to suggest that HPV TaPS would be cost and clinically effective. It was agreed that the UK NSC should consult on a recommendation to approve HPV as a primary screen for cervical cancer and that the feasibility study should explore implementation issues including length of time before a re-screen following a HPV negative result.

Henry Kitchener chairs the Steering Group for the HPV Primary Screening Pilot. According to one interviewee, the ARTISTIC41 trial has been essential for designing the national pilot. The interviewee noted that policy-makers have found the main paper and the monograph to be most valuable:

[The Cancer Screening Programmes] have been able to crawl all over the monograph to get details and answers that aren’t in the publication. The publication alone would not have been enough. [The Cancer Screening Programmes] do need the detail that’s in the monograph. If the HTA ever think about stopping doing that, I would vote against.

The results of the pilot will then inform any future changes in the NHS Cervical Screening Programme. However, according to one interviewee, the bureaucratic process may pose a barrier to the adoption of HPV testing in primary screening because the changeover from cytology to HPV testing will be expensive in the short run, even although it may be cost-effective in the long run.

One interviewee outlined the uncertainties that the HPV pilot is trying to address:

In terms of HPV, it is unfeasibly complicated. The acknowledgement that HPV as a primary screening is better than LBC is almost the easy bit. What [the National Screening Committee] has then got to do, in the context of HPV vaccination, is work out how on earth to do it. You change the screening intervals, you change the call intervals, you up the colposcopies, looking at the cervix in the short term, but reduce it in the long-term, and you hopefully reduce the length of time between call and recall for women that are HPV negative. That will have a huge effect on the labs, because you suddenly have to do millions of HPV tests and has the possibility of completely getting rid of cytology for all intents and purposes, reducing the cytology workforce to next to naught. There will need to be some, but not in the volumes that we’ve got. And of course that changes year on year when you do HPV vaccination, because hopefully this will more or less go out of style completely. And then of course, there is the question of whether women fancy being screened for a sexually transmitted disease, what effect that will have on it. Women will come back at varying intervals – whether the system is capable at managing varying bespoke intervals or a much more complicated set of call and recall.

Stage 5: adoption by practice and the public

The ARTISTIC41 trial is unlikely to have had a major impact on clinical practice to date as HPV testing for primary screening is being piloted in the NHS only now. However, as the pilot covers approximately 10% of women screened, it is likely that, through its impact on the pilot, the ARTISTIC41 trial has had a substantial impact on practice in the areas where the pilot is being conducted and on the health outcomes of women in the pilot regions. As the pilot is ongoing, and its results have not yet been published, it is not yet possible to say precisely what the impact has been on clinical practice (e.g. the costs of implementation) or the precise impact on patient health outcomes.

According to one interviewee, the ARTISTIC41 trial has probably also had an impact on the teaching of clinicians:

I’m sure it has. It depends what clinicians you are talking about. If you’re talking about gynaecologists and so on, probably it has. I’m sure it has become part of the canon of knowledge and evidence that people would be taught about.

One interviewee also stated the ARTISTIC41 trial has also had an impact outside the UK, primarily through its contribution to the Ronco et al. (2014)204 pooled analysis. The interviewee noted that:

Worldwide, we are all considering where we go next on HPV primary screening and how fast we can move and how safe it is to take each step. We’ve got well-established screening programmes. It’s not like a new programme where you’re just introducing a new test. You’re actually de-commissioning something that has been proven to work. It’s different to: shall we introduce lung-cancer screening, shall we introduce ovarian cancer screening, etc. We’re talking about getting rid of something that works well. The Pap test has saved millions of lives across the world and we’re now saying that we’re going to stop doing it.

Another interviewee noted that the impact of the European HPV trials may, collectively, be particularly important in developing countries that do not have the infrastructure for cervical cytology, as the results suggest that developing countries could safely introduce cervical cancer screening programmes that use HPV testing instead of cytology.

Stage 6: final outcomes

As noted above, HPV testing for primary cervical screening has not yet been adopted nationally in the NHS. However, if there is a national roll out of HPV testing for primary screening and the benefits of that screening are similar to those reported in the ARTISTIC41 trial, there could be substantial health gains. Although it would be difficult to attribute all of these health gains to the ARTISTIC41 trial, it undoubtedly made an important contribution to the evidence base that led to the pilot, which, in turn, may lead to national roll-out. However, when considering the potential health gains from national roll-out of HPV testing, one must also take into account the national HPV vaccination programme, which will decrease the overall benefit of HPV screening, as fewer women will acquire high-risk HPV strains. One interviewee noted that the most immediate impact of the ARTISTIC41 trial, should it result in changes to the Cervical Screening Programme, would be a reduction in cervical cancer among women who were beyond the age of the effect of the vaccine.

Implementation of HPV testing as a primary screening strategy would also have an important impact on the labour force. It would result in a substantial reduction in the need for cytoscreeners. According to one interviewee, some existing cytoscreeners would likely retire or be made redundant, whereas others would be re-trained to do viral testing instead of cytology. However, viral testing is much less labour intensive than cytology.

The ARTISTIC41 trial, together with the other European primary HPV testing trials, has likely had an impact on the development of new primary HPV screening tests. According to one interviewee:

It helped to provide substantial data to support HPV primary screening. Therefore, these big companies could see that if primary screening was coming down the line with HPV there would be a big market for the diagnostic kits.

According to another interviewee, the European trials have already had a big impact on the medical device industry. At the time that most of the trials started in the early 2000s, there were only two devices on the market, both of which were DNA diagnostic tests: Polymerase Chain Reaction and the Hybrid Capture Test. Now, according to the interviewee, there are eight or nine validated tests, and their development has caused a substantial decrease in the price of HPV tests.

Table of payback

Payback details for this case study are provided below in Table 17.

Summary

The systematic review and cost-effectiveness analysis of the Newborn CHD study42 systematically reviewed the literature on outcomes of children with CHDs and compared the cost-effectiveness of different screening strategies. The study42 found that early detection of CHDs, through newborn screening, can improve the outcome of affected children, but that the current screening programme, which consists of clinical examination at birth and again at 6 weeks, is insufficiently accurate. The study42 concluded that pulse oximetry is a promising alternative screening strategy but that further research is required on its use for antenatal screening. The Newborn CHD study42 was the first to systematically review the evidence. It led directly to the HTA programme-funded PulseOx trial,220 which looked at the effectiveness and cost-effectiveness of using pulse oximetry to screen newborns for CHDs. The Newborn CHD42 study and the subsequent PulseOx trial,220 together with the increasing evidence on the effectiveness of pulse oximetry, resulted in the NSC undertaking a pilot project, the Newborn Pulse Oximetry Screening Pilot, which may result in a national roll-out of pulse oximetry screening for CHDs. Some neonatal units have already introduced pulse oximetry screening.

Introduction to case study

Stage 0: topic/issue identification

A number of key factors influenced the research team’s decision to work in this area, as detailed in Box 2 and described below. The Newborn CHD study42 was a commissioned call. Therefore, the key influencing factors outlined below include both the factors that motivated the initial commissioning of the study42 and the factors that motivated the research team to undertake the research.

Interface A: project specification and selection

Carol Dezateux developed the original project proposal with Catherine Bull and Rachel Knowles. 223 At the time, Dezateux had substantial experience in research into newborn and childhood screening programmes,234–250 as noted above. Bull was a Consultant at Great Ormond Street Hospital for Children NHS Trust, and had previous experience in research in cardiology and CHDs. 252–262 At the time of the Newborn CHD42 study, Knowles, a public health doctor by training, was on a secondment to the Institute of Child Health at the University College of London to do some public health training, which resulted in her involvement in the study. Knowles had a history of interest in child health. 263

The commissioning brief asked that the systematic review should include the natural history of different CHDs, the properties of different screening tests of CHDs, the clinical impact of the different tests, the effectiveness of different management options for children who test positive for a CHD and the psychosocial impact of the test and diagnosis on patients. 223 The commissioning brief also requested cost-effectiveness modelling with sensitivity analyses. The research team modified the research question after their initial literature review to include pulse oximetry, as the evidence supporting the use of pulse oximetry for newborn screening for CHDs was published after the commissioning brief.

The Newborn CHD42 project had five aims. 223 The first was to conduct a systematic review of the epidemiology, natural history, treatment and outcomes of CHDs and the effects and costs of existing newborn screening strategies. The second was to classify different types of CHDs for newborn screening. The third was to assess the effects, costs and cost-effectiveness of different newborn screening strategies. The fourth was to assess the quality of life of children with a CHD, from the perspective of both parents and health professionals. The fifth was to explore parents’ experiences of newborn screening for, and diagnosis of, CHDs. The methods used in the study42 included a systematic review, a cost-effectiveness analysis and a VOI analysis.

According to one of the interviewees, the intention of the research team from the outset was to involve patients, clinicians and policy-makers in the later research stages, even although patients, clinicians (beyond the two cardiologists on the research team) and policy-makers were not heavily involved in the development of the initial research protocol. According to the same interviewee, Carol Dezateux had previously done a lot of work with the NSC and was aware of the issues of interest to the NSC. Similarly, the two cardiologists on the research team, Catherine Bull and Christopher Wren, had a long history of interest in population health and, according to the same interviewee, knew from past experience that it would be necessary to engage with clinicians early on to understand the implications of screening for clinical practice. Lastly, from the outset, the research team planned to involve the parents of children with congenital heart disease in focus groups to understand parents’ experiences of screening for, and the diagnosis of, CHDs.

According to an interviewee, the research team received feedback from the HTA on their initial application, which was used to refine the final application. However, it has not been possible to determine how the research protocol changed as a result of the feedback.

Stage 1: inputs to research

Stage 2: research process

The study by Knowles et al. (2005)223 consisted of (1) a systematic review of the published medical literature on the epidemiology, natural history, treatment and outcomes of CHDs; (2) a cost-effectiveness analysis of alternative newborn screening strategies, which used a decision-analytic model and both deterministic and probabilistic sensitivity analyses, and a VOI analysis to explore the value of further research to policy-makers; (3) an empirical study on parental and clinician valuation of the quality of life of children with congenital heart disease, which used a visual analogue scale similar to that utilised by the European Quality of Life-5 Dimensions; and (4) an empirical study on parents’ perceptions of newborn screening for CHDs, which consisted of a systematic review of the literature and focus groups. 221

As noted above, the VOI analysis that was conducted for this study42 was a relatively novel method in health economics at the time. VOI analysis is used to estimate the potential benefit of collecting more information through increased research before making a decision. In the Newborn CHD42 study, the VOI analysis set out to identify the model parameters for which additional research would provide the most value. 223,226

The research team carried out focus groups in conjunction with Heartline (www.heartline.org.uk), which is a charity that supports the families of, and children with, heart conditions. 223 The focus groups were designed to include parents of children with CHDs, including both parents whose child had been diagnosed through an antenatal ultrasound scan, pulse oximetry, echocardiography and clinical examination, and parents whose child had become ill before receiving a diagnosis, as well as some parents whose child did not have a CHD. However, the final group predominantly included parents who had young children with more severe CHDs. In the focus groups, parents discussed their experience of having had their child screened for CHDs. Heartline allowed the research team to access the views of parents of affected infants. In the acknowledgement section of the report by Knowles et al. (2005),223 the authors note that ‘the authors would like to thank the national support group for families of children with congenital heart disease, Heartline, who provided us with invaluable assistance in recruiting and publicising the focus group’. According to one of the interviewees, this level of patient engagement was relatively novel at the time as it predated the current emphasis on the patient and public being involved in research, and the creation of organisations such as INVOLVE (www.invo.org.uk/).

A number of different individuals were involved in carrying out the research. Carol Dezateux, the CI, contributed to the development of the original project proposal, the review of the epidemiology of CHDs and the data extraction for economic model. 223 Rachel Knowles contributed to the review of the classification and coding systems for CHDs, the systematic review of outcomes of children with a CHD, the systematic review of parental views, the development and validation of the CHD classification for screening, the development of health states for the quality-of-life study, the organisation of the parent focus group, the compilation of the report on parents’ views, the systematic review for the model parameters, and the data extraction for the model parameters. Ingolf Griebsch conducted the review of cost parameters and contributed to the development of the decision analysis model and the expected VOI analysis. Jacqueline Brown contributed to the development of the decision analysis model and the expected VOI analysis. Catherine Bull contributed to the construction and validation of the CHD classification, the development of health states for the quality-of-life study, the organisation of the focus group, the development of the subjective probabilities for the model and the data extraction for the model parameters. Christopher Wren provided and analysed data from the Northern Region data set and contributed to the review of CHDs, the validation of the CHD classification and the development of the subjective probabilities. All members of the research team also participated in regular discussions and planning meetings, the administration of the questionnaire for the quality-of-life study, the development of the economic model, and the writing, proofreading and editing of the final report.

The research team did not have any specific international involvement in the study42 beyond the inclusion of research that originated from outside in the systematic review.

Stage 3: primary outputs from research

Knowledge

The Newborn CHD study42 resulted in the publication of two peer-reviewed papers and a HTA monograph. The publications cover the study results,223 the results of the cost-effectiveness model and VOI analysis,226 and the results of parents’ and professionals’ valuations of the quality of life of children with CHDs. 227 Figure 12 presents the results of the bibliometric analyses on the publications resulting from the Newborn CHD42 study.

FIGURE 12.

Bibliometric analysis for Newborn CHD42 case study.

The main results of the systematic review found that the prevalence of CHDs is 7–8 per 1000 live births, which accounts for 3% of infant deaths. 223 The study42 also identified a number of long-term conditions that result from CHDs, which included arrhythmias, infective endocarditis and pulmonary vascular obstructive disease. However, the review also identified important gaps in the evidence on long-term outcomes related to physical disability, the capacity to participate in normal childhood activities and neurodevelopmental, cognitive and psychosocial outcomes. The study42 also developed a classification for CHDs, which included three main types: life-threatening, clinically significant and non-clinically significant. Lastly, the review identified two alternative newborn screening strategies: pulse oximetry and echocardiography, in addition to clinical examination.

The economic analysis compared the cost-effectiveness of clinical examination to clinical examination with either pulse oximetry or echocardiography and found that the cost per additional timely diagnosis of life-threatening CHDs ranged from £4894 for pulse oximetry to £4,496,666 for echochardiography. 223,226 With the inclusion of clinically significant CHDs, the cost per additional diagnosis decreased to £1489 for pulse oximetry and £36,013 for echocardiography. The VOI analysis found that the maximum monetary value of additional research, at a cost-effectiveness threshold of £50,000 per timely diagnosis of a CHD, was £744,000 for the primary outcome, timely diagnosis of a life-threatening CHD, and £14,450,000 for the secondary outcome, diagnosis of a clinically significant CHD. The authors concluded that adding pulse oximetry to clinical examination is likely to be a cost-effective screening strategy for newborns, but that the VOI analysis supports targeting future research at reducing the uncertainty on the detection rate and false-positive rates for pulse oximetry.

The quality-of-life study compared the preferences of parents of children with CHD to those of health professionals for the longer-term health outcomes of children with CHDs, using a self-administered anonymous questionnaire with a visual analogue scale. 227 The study found that both parents and health professionals place similar values on the quality-of-life outcomes of children with a CHD, and agreed on the order of the ranking of different health states. Parents and health professionals both gave the lowest scores to health states with severe neurological disability.

The targeted review and focus groups found that parents support newborn screening for CHDs, prefer simple and accurate screening methods, experience negative psychosocial effects from poor management of the screening process and false test results, may experience short-lived anxiety in the event of a false-positive result until receipt of negative diagnostic test, and agree on the need for both universal screening standards and capable health professionals to discuss the outcomes of CHD screening. 223