Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Benefits of the approach

This approach has many benefits. Lookup rules can tag or annotate comments as belonging to quite nuanced literal themes, by combining gazetteers using Boolean logic. The functions of the syntactic applications include part-of-speech (POS) tagging, in which an individual word (X in the X–Y annotation used above) is tagged on the basis of the POS it represents (e.g. whether it is a noun, verb or adjective). This can, for example, distinguish between the different uses of the test in ‘the test came back negative’ and ‘I will test the new treatment’. Importantly, words (X) are not independent of each other, because if the previous word was an adjective, it is far more likely that the next word will be a noun than a verb. This enables new words to be more likely to be categorised correctly than not, given these various points of information. Such features of IR have been used to develop an analytical process that can cope with the fragmented phrases and sentences that typify survey free text. The approach also makes use of semantic (meaning-making) information, such as co-location. Thus, for example, if a comment says ‘The nurse was present. She did not speak’, the system will know that ‘She’ refers to the nurse because that is the relevant POS in the nearest sentence. Sentiment analysis has also been used, in which the label Y is used to reveal the sentiment (positive or negative) of each comment. These and other advantages of rule-based IR over template-based machine learning, and the rationale for choosing it, can be summarised as follows:

• External knowledge repositories [such as WordNet^®, version 3.1 (Princeton University, Princeton, NJ, USA)], can be used in categorisation, so that the system can cope with novel data linked in these repositories to existing labels.

• New words can be added to gazetteers in the time that it takes to type them.

• By writing rules that are not dependent on specific words, but that take into account the arrangements of words within syntactic and semantic contexts, the software can handle new data with good sensitivity and specificity.

• It is not required to annotate a large number of data to start the process, in contrast to memory-based machine learning. Thus, once the rules are written, the system can be used in virtual real time for future iterations of PESs with no modification to minimal modification (a matter of hours rather than months). This gives it potential speed advantages in future years.

• The system can be made domain specific, increasing the relevance of its outputs,45 as the rules are written by specific individuals.

• New rules could be written very quickly and added to the system in the future and, hence, the approach makes for a potentially more accurate, more adaptable and more future-proofed system than template-based machine learning.

• The system could be easily modified by others adapting the system in the future; transferability to other health-care surveys was an important element in the HSDR programme brief.

• Rules could be specifically written to tackle some specific issues with free-text comments (see Chapter 4).

• The approach is amenable to real-time data processing.

• It does not require training examples (using resource-consuming manual data categorisation) or redevelopment of the whole system each time novel data are added.

Previous research has shown that linguistics-based IR models such as this can outperform manual (human) categorisation of customer feedback reviews,46 an area in which IR is particularly used. Therefore, it shows promise in the analysis of patient experience feedback.

Stage 1: preliminary (scoping) work

Stage 2: main development phase

Stage 3: validation and evaluation phase

Theory

Normalisation Process Theory was used throughout the study. NPT is used to understand actions related to the implementation, embedding and integration of new technology or complex interventions. 52,53 It comprises four core ideas or constructs: (1) coherence, (2) cognitive participation, (3) collective action and (4) reflexive monitoring. Each of these constructs is subdivided into four and, thus, there are 16 constructs in total. The NPT toolkit website [www.normalizationprocess.org/npt-toolkit.aspx (accessed 9 October 2018)] contains example questions that can be used to explore these, and represents the results graphically in radar plots.

Identifying the research question

The research question was shaped by the larger study and was ‘How does existing empirical evidence suggest that a digital health-care dashboard or toolkit should be designed to optimise its usefulness and usability?’.

Definitions and scope

The PICOT (participants, intervention, comparison, outcomes, types of study) framework70 was followed to determine inclusion and exclusion criteria and develop the search strategy. Each PICOT element is described in the following sections.

Study inclusion and exclusion criteria

Table 1 shows the study inclusion and exclusion criteria, as a summary of the previous sections.

Assessment of methodological quality

A formal quality assessment of the included studies was not undertaken, as the main interest was in using the results to develop the health-care dashboard.

Charting the data and collating, summarising and reporting the results

A data extraction form was developed to ensure consistency in the review process across researchers. This included details of each study’s design, as well as the key constructs and other information needed to answer the research question. The focus was particularly on the format and audience of the dashboard, and results of the evaluation of its features. After two researchers extracted the data from all included studies (with cross-checks for quality control), one researcher (Daria Tkacz) undertook a narrative synthesis of the results. This was checked back against the original documents by Daria Tkacz and Carol Rivas for consistency and face validity. Results were also summarised in tabular form and discussed by the research team, steering committee and user advisory group (AG) members in relation to the PRESENT dashboard design. The DCE and structured walk-through results were mapped onto this table to inform both the final design and the general recommendations for health-care digital dashboard/toolkit design.

Access

Access was considered by 9 of the 43 studies. Dashboards and toolkits were reportedly more effective if provided as part of a workflow, integrated into an already existing system92 or otherwise easy or effortless to access or constantly in sight. 76,93–95 Stand-alone systems requiring the user to initiate access via a separate login were less likely to be used regularly. Users may be reluctant to register for access, particularly if the process requires effort. 89,90,94 Any registration process should therefore be clear and short, preferably limited to one screen. 96 The value of the system and its benefits for the user need to be clearly defined before registration, for instance through an engaging homepage. 96

The URL should be simple, memorable and clearly associated with the focus of the dashboard. 89

Having real-time access to a dashboard during clinics with patients was seen as a valuable feature for clinicians. 77

Flexibility and individualisation

The theme of flexibility and individualisation was reported in 11 articles. Users valued the ability to tailor a system to individual and organisational needs. 94,97–100 This functionality could include personalised alerts and reminders,77,79,93,100–102 the option to set default filters75,87 and the ability to pin particular information to the screen to appear at all times, enabling comparisons. 77 Benefits were also reported from being able to dynamically add, remove and adjust elements of the system to fit the user’s focus and workflow. 74,82,103

The appropriate style to use when presenting the data is highly related to the nature of the data set and its audience, and, hence, it was recommended in four studies that the user is presented with a choice of graphs, dials or tables that can be adapted to individual preferences. 74,82,102–104

Usability and navigation

Usability was considered in 24 studies (as reported in 26 papers). Study authors considered a simple-to-follow, clear layout that anticipates the user’s workflow as key to usability. 75,87,90,102,103,105 Faster interpretation of the data is enabled by applying colour to emphasise differentiations and by adopting ‘most common’ lists. 79 Scrolling should be avoided or limited according to study authors,89,96,106 and information should always be accessible in as few mouse clicks as possible. 79,100

Using bright, distinct and highly contrasting colours for the interface is recommended. 84,86,96,104,107,108 Warm colours have been found to be inviting. 109 Black fonts on contrasting, light-coloured backgrounds are favoured. 109 The font must be large and clear in style to ensure accessibility. 76,89,100,106,108

To ensure comfortable navigation, the browsing options should always be clearly visible on all pages. 76,96,110 The navigation buttons should be large and clearly labelled. 90,108 A left-to-right menu has been found to be more successful than a top-down menu. 95 It is also important to consider different screen sizes and browsers to ensure that the layout resizes well and looks suitable on different devices;73,76,111 in other words, the system should be responsive.

Stakeholder preferences for layouts and ways of presenting the data depended on their role. Simple, static dashboard screens were reported to be helpful for ‘at-a-glance’ overviews during busy clinics, whereas interactive views were described as being more helpful for in-depth analytics. 74,103 Correspondingly, simple layouts were found to be appropriate for most users, whereas more complex, highly interactive dashboards were more suitable for ‘power users’ with analytical responsibilities. 74,82,103 Overall, to support the effective interpretation of information on the dashboard, it was recommended that the pages are not cluttered or overburdened with information. 90,96,104,106,109 The information at the bottom of the page tends not to be found easily, thus key points should appear at the top. 82,87 Limiting the immediately visible textual information and giving users an option to view more detail has been found to be effective. 111

The language must be kept simple and abbreviations and jargon should be avoided. 76,89,90,93,104,109,111 One study suggested that the complexity should be kept below a seventh grade (in the USA – equivalent to year 8 in England) reading level, and first-person narrative should be used in the text for a more engaging style. 106 Coyne et al. ,104 who designed and evaluated a dashboard for young people, emphasised that the language should be empowering, not patronising. 104 To avoid misinterpretation, it is important to provide definitions for any dashboard elements that may not be immediately obvious. 74,100,103,112

Use of images and videos

Although images are considered to be effective at fostering engagement with dashboards,87,105 one study emphasised the need to ensure that their purpose was informative, not solely decorative. 109 The choice of pictures should reflect users’ realities regarding factors such as age, gender or ethnicity. 105,106 Videos were found useful both for providing instructions on how to use the dashboard and for providing content, for instance, in the form of patient stories. 88,106,107

Chart types

Four studies reported that line and bar charts were the clearest and most appropriate charts for providers. 76,84,87,103 Line graphs were considered to be particularly suitable for comparisons over time, and they can be used with tool tips. 76 By contrast, one study found that speed dials were more effective than bar charts or radar plots. 77 This might have been linked to the nature of the displayed data, which focused on lifestyle. A sixth study compared the effectiveness of graphs and tables and found that the former were more appropriate for tasks that required analysing relationships and the latter were more appropriate for extracting specific values. 101 This study also showed that graphical formats were suitable for providers with a lower level of analytical skills, whereas tabular formats were more accurate for complex tasks.

Decision-making users with highly developed analytical skills may benefit from innovative graphs showing different data simultaneously82,84 (e.g. with multiple graphs on one screen). 80 Users often stated a desire for historical comparisons. 76,83,84,87,93,97,98,112 When reference data points, such as national averages, are included, interpretation is facilitated.

Regardless of the type, graphs need to be clearly labelled. 74,76,91 Displaying proportions and sample sizes is important for informed judgement74 and using highly contrasting colours to differentiate data points on the graphs helps to avoid misinterpretation. 82,84 The red/amber/green (RAG) traffic-light colour system has been found to be universally recognisable across health care. 76,78,79,81,83,95,112

Data interrogation

The ability to filter the data in real time and to sort these by any level and quality indicator was found to be valuable. 73,76,112 Importantly, the filter parameters need to be practical, clearly defined and aligned with the user’s work. 73,76,80,112 The design needs to make it clear that the page has changed after filtering. 76

Search boxes were also found to be useful for interrogating the data, and a drop-down list or dictionary of suggested search terms was recommended to meet the needs of users who may not be sure what to look for. 90

Hartzler et al. 99 considered the use of pictographs on a dashboard designed for patients and found that weather pictographs were more effective at illustrating patient-reported outcomes than the overly simplistic ones based on smiley and sad faces or charged batteries.

Print and export functions

A feature highlighted in four studies,74,98,104,109 as desired by users, was the option to print or download information and data, for patients to use or share with friends and family, and for professionals to use or share with colleagues.

Community features

A number of studies found that fora, chat rooms or similar community features were highly desired by users, to encourage an exchange of experience and information-swapping and to establish communication between patients and other patients or clinicians. 74,86,89,97,99,101,107,108,111

Security and privacy

Given the sensitive nature of health-care data, several studies noted that security must be prioritised. 74,94,101,112

Offering recommendations and solutions

Dashboards are more likely to be adopted if, in addition to highlighting problems, they offer recommendations for solving them. 92,102 This may include signposting to other sources of information and support. 76,93,97,107

Strengths and limitations

This was a scoping review, and for this reason, although it was rigorous and systematic, it had a broad research question. Most studies considered the health-care professional user, and few evaluated dashboards that were designed for patients. Many of the features elucidated are no different to more generalised good dashboard design features (see Chapter 10). For example, many common recommendations from the review studies accord with La Grouw’s61 assertion that a good dashboard requires appropriate high-level summary data, display features that are appropriate for the message and design features that are customised to the requirements of the specific users. Nonetheless, sufficient papers were located to provide a clear picture of health-care dashboards as needing to be easily integrated within existing workflows and simple and effortless to use, but flexible. It was clear that health-care dashboard users were pressed for time.

Implications for research and health care

This scoping review provides a valuable summary of evaluation-based recent evidence on desirable design features for health-care dashboards. The findings have been used to develop a list of recommendations for evidence-based health-care dashboard design, combined with other work. This list of recommendations can be checked by dashboard researchers, commissioners and developers within health care when developing new dashboards, for their optimisation.

How this informed later stages of the study

The findings of this scoping review were fed into a report shared among research team members. The immediate implications for the design of the dashboard were discussed with the programmer. Many of the suggestions raised in the articles discussed in this chapter were considered in the topic guide for the workshops and follow-up interviews, when the opinions were sought of the stakeholders to whom the dashboard is addressed. These were then cumulatively discussed by the research team and considered in relation to the scope of the project. This informed the design decisions for the beta version of the prototype, which was subsequently tested in the proof-of-concept stage.

The scoping review was also essential for designing the DCE. The key design features identified in this chapter informed the list of attributes for the DCE (see Chapter 7). The cove research team met with the economist working on the project, Emmanouil Mentzakis, on three separate occasions, during which the list and variations of the attributes were further discussed and adapted. The final list of attributes was shared with the AG for comments before the DCE survey went live.

Themes from prior research by the team in more detail

Prior research by the wider PRESENT team, and included in the rapid review, is considered in more detail in this report for two reasons. First because this shows the team’s natural progression in the thinking about text analysis and second, it includes a study commissioned by Macmillan Cancer Support,21 which involved the thematic analysis of free text from the 2013 WCPES. The same data set was used independently of this analysis to develop the prototype system.

The WCPES analysis, reported by Bracher et al. ,21 was manual and took several months, using a four-stage process of coding and analysis. This involved literal coding (i.e. concrete coding based on respondent’s own words113) for areas of cancer patient experience (i.e. concrete coding based on respondents’ own words;113 literal coding for specific categories within different areas of cancer patient experience; identification of more abstract themes; and comparisons between closed questions and free-text responses). This research highlighted the need for a more automated approach to analysis for these data. The results from the original analysis by Bracher et al. 21 were compared with the PRESENT text-analytics process outputs in the study’s sensitivity and specificity analysis (see Chapter 8).

The main themes from the study by Bracher et al. 21 were:

• communication between patients and staff

• waiting for appointments

• waiting to be seen on the day

• communication between staff and/or institutions

• concerns about staffing levels

• out-of-hours and weekend care

• investigations and diagnostic services

• aftercare

• emotional, social and psychological support

• hospital environments

• travel-related issues

• food and catering

• financial concerns.

Wiseman et al. 14 undertook a framework analysis of free-text data from the 2012–13 National CPES subsample for two London Integrated Cancer Systems, covering 27 trusts. This comprised 15,403 comments from > 6500 patients. The dominant themes were poor care, poor communication and waiting times in outpatient departments, consistent with the analysis by Bracher et al. 21

Team members had also previously undertaken a memory-based machine learning analysis of the 1688 free-text comments in a population-based survey of colorectal cancer survivors. This necessitated the manual analysis of a large proportion of the data, to train the machine learning algorithms. The main themes from this study were similar to those for the WCPES analysis, relating to:

• emotional support

• diagnosis and referral

• preparation for treatment side effects

• signposting

• aftercare

• co-ordinated care.

Approach rationale

The way the target themes were named meant that rules could be written that did not require the subject, verb and object of the comment to be determined, although the code that Laurence Antao wrote is easily adaptable to do so. Nonetheless, syntax was still important for accurate rule-writing and for sentiment analysis. The study team wished to identify subclauses to break the comments down into meaning units, but with standard GATE parsing this was often inaccurate. Laurence Antao determined that using noun phrases, verb phrases (including object noun phrases) and adjectival phrases, and also prepositional phrases along with subclauses, made it possible to develop an effective rule-based approach to create syntactically viable lists of topics using domain-specific gazetteers.

Syntactic patterns

Apache OpenNLP, version 1.8.4 (The Apache Software Foundation, Forest Hill, MD, USA), a GATE parser plugin (i.e. a software component that adds to an existing computer program), is good at determining noun phrases and verb phrases, and annotates them hierarchically. This has two important (non-mutually exclusive) features that are of great use. First, noun phrases may contain multiple noun phrases within them (likewise, verb phrases may contain multiple verb phrases – as in the garden path example in Challenges in analysing survey free-text comments and key foci). Second, verb phrases may contain noun phrases. Whatever the structure of a sentence, a clause will always have reference to a verb phrase and a subject phrase, whether that be within the clause itself, in another clause within the sentence or in another sentence completely (although this is not true of single-word sentences or very short sentence fragments that stand alone, this study is considering surveys, in which there is always a question being asked). By pairing up verb phrases to noun phrases, with phrases that are able to be paired multiple times, syntactically correct semantic meaning can be drawn from highly complex sentences and non-sentences (Figure 3). The full meaning of a sentence can then be gleaned by using rules to interpret conjunctions and verb tenses (e.g. a conditional tense implies the negative of that action in the present).

FIGURE 3.

How a complex sentence can be broken down into its constituent noun phrases and verb phrases (prepositional phrases were also noted). These can be parsed in turn by the system so that it picks up each and all variations. NP, noun phrase; PP, prepositional phrase; VP, verb phrase.

The parser that was used outputs subclause category SyntaxTreeNodes (tree representations of the abstract syntactic structure of the programming source code), which means that rules can be created to differentiate between noun and verb phrases that are next to each other but not necessarily directly connected.

GATE pipeline

In Figure 4, the ‘processing resources’ (PRs) highlighted in yellow are those that interpret the clause structures and produce the syntactically feasible lists of topics contained in the verb phrase and noun phrase ‘pairs’. The PRs take, as input, the keywords produced by the blue elements, some of which contain a polarity (‘negative’ Boolean) and ‘degree’ (0 = average, 1 = very, 2 = extremely). The ‘themes’ JAPE transducer PR (highlighted in red), then assigns the different ‘pairs’ to their corresponding themes. The green highlighted PR merely annotates hospital names or trust names with their corresponding trust. The non-highlighted PRs are the default PRs of GATE known as ANNIE (‘A Nearly-New IE’ system) PRs (although the sentence splitter has been swapped with a more efficient one). The JAPE transducer works out whether or not the phrase is negative, taking into account modal verbs and multiple negatives, as well as ‘if’ statements; there is not a necessity to factor in negative conjunctions. For convenience (at the theme stage), if a keyword is determined to be negative, its majorTypes or minorTypes list entry will be preceded by ‘not’.

FIGURE 4.

The PRESENT GATE pipeline.

The GATE sentiment analysis JAPE was used and extended by adding relevant domain-specific entities to the JAPE so that the sentiment analysis can recognise specific entities from the domain-specific work to apply sentiment to.

Rule logic

Rules depended on a hierarchical approach upstream and an ‘if–then’ format and Boolean logic downstream and for gazetteer lookup.

An example of a rule that draws on gazetteers using Boolean logic is given below, where | means ‘or’. Rules can be easily tweaked by adapting or changing the gazetteer chosen.

Word use issues and WordNet

Although a rule-based system is potentially better able to cope with the unfamiliar than memory-based systems, it is not foolproof. The system has been built to take this into account and accommodate a wider possibility of previously unseen input words in a parsimonious way, to allow for the interpretation of unexpected vocabulary and reduce the reliance on human prediction. This has been achieved using the WordNet dictionary to tag words with synonyms and hypernyms and then annotating the most specific gazetteer entry with the flexible gazetteer PR. A flexible gazetteer was used rather than JAPE rules to make for easier thematic decisions, as it essentially presents the features to be added to the terms in a spreadsheet. This should be better in terms of legacy and accessibility, and should also make adding terms quicker.

One drawback of using WordNet is that it could lead to words being included in lookup that are inappropriate for the task and, therefore, lead to inaccuracies. Thus, it might become useful to put in delimiters. Users will need to balance the sensitivity and the specificity.

Issues of providing full comments publicly

One issue is that patients did not consent to have their comments publicly displayed when they completed the CPES. Therefore, the full comments cannot be shown on an open site.

Trust disambiguation

Respondents to a survey may be referred from the trust to which their survey is attributed to a centre in another trust for some aspects of their care. Previous work by some of the research team suggests that this occurs in 9% of cases and has the potential to cause significant problems. 14,21

Rules can be created that compare the trust name in the metadata for each comment with the name of organisations mentioned in the comment to see whether or not they correspond. The system includes such a rule, and also, in case the trust files need to be updated or edited for specificity of trust (names have several shortened forms), there is a Python program (version 3.6.5; The Python Software Foundation, Wilmington, DE, USA) to automatically create the gazetteer files from the Extensible Markup Language (XML) data file available on the NHS website containing trust names.

Redactions

Personal names were identified for redaction to ensure the privacy of patients and other individuals and confidentiality of patient information. This required disambiguation between the names of persons and of hospitals that happen to contain personal names, such as Jimmy’s for St James’s. Real names were replaced with [health-care professional name], [patient name], [hospital name] or any relevant alternative.

HealthWatch redactions, such as [xxxx] or [name removed], were replaced by the use of programming to indicate the entity type, such as [hospital].

Words with multiple meanings

Rules were developed to identify words that might be confused with a person’s name, such as X-ray or hospital ward.

Data volume

Loading large batches of data into the GATE graphical user interface (GUI), to play with visually, may cause problems for the GATE GUI. The developers of GATE advocate the use of their cloud system as an alternative. ‘GATE Embedded’ (a version of GATE usually used by developers) can handle much bigger data sets than the GUI can, but the GUI is more useful for looking at things when developing new rules. This means that refinements in development should be made on subsets of any data, but for the final outputs, large batch-loading is not a problem. Therefore, the option of choosing to load into a corpus up to 500 or 1000 rows only was included.

Themes extracted

The themes for the process were determined through co-design work with the different stakeholders in health care, but this work was started with a finite pool of themes determined from previous research (see Chapter 3), an initial survey (see Chapter 3) and the literature (see Chapter 2). This is a deductive approach. Using Python, a latent Dirichlet allocation (LDA) option was developed. This is an unsupervised clustering technique that can be used to group texts by discovering apparent topics that they share through looking at words that occur together in some of the texts but not all (words that occur in all of the texts are expected to be common words rather than words indicating a topic). LDA can be used as an inductive topic-modelling/topic-discovery approach; the PyLDAVis library (The Python Software Foundation, Wilmington, DE, USA) (a Python library for interactive topic model visualisation) was used to better visualise the results of the LDA. This is potentially useful to:

• see how close results come to what humans with domain knowledge have worked out the topics of interest are

• see if the LDA finds topics that the humans had not thought of, but that are of interest.

For future data sets, topic discovery could make a useful addition to the process of humans mindmapping which topics are of interest. This option is particularly useful for free text for which the topics have not been predetermined or when there is insufficient prior information for a useful list to be drawn up deductively. It could also be used for discovery within the existing data set.

Latent Dirichlet allocation/topic-modelling does have drawbacks that need to be considered with the data:119

• Topic discovery is a black-box approach and is prone to overfitting.

• Efficiency decreases when a very large number of topics are used to fit the LDA.

• The LDA performs better when the underlying topics are well separated (in the sense of Euclidean distance).

• ‘Topics’ discovered in an unsupervised way may not match the true topics in the data.

• It is theoretically impossible to guarantee the identification of topics from a small number of documents/comments.

• Short documents, such as free-text comments, may result in poor performance of the LDA.

In addition to LDA, the system could incorporate further unsupervised machine learning techniques in similar supporting roles to improve the automation of the overall system.

Case sensitivity

Existing PRESENT gazetteer entries in the pipeline are case insensitive (case sensitive = false). The assumption was made that words would not mean something else entirely if they were uppercase versus lowercase. This eliminates the need to make a duplicate of every lowercase word that might be at the start of a sentence within the gazetteers. It also helps to prevent errors arising when the case is used wrongly within a comment. This means that if there are any words that need to be case sensitive in the future, a second case sensitive = true gazetteer needs to be added to the pipeline for these words. This might be required, for instance, if acronyms are used that would otherwise be taken as normal words.

Benefits

Linked to these five levels, the main benefits of the dashboard/toolkit, as the study team perceived them at the start and aimed to uphold, were that it enabled:

• the routine systematic use of patient experience free-text comment data (at all levels, including the public level)

• enhanced decision-making in health-care improvements (strategic and operational use)

• facilitation of immediate targeted decisions to improve patient care (local use)

• improved quality-of-care benchmarking against other sites (operational use)

• reduced time and effort to assimilate information on the patient experience (all levels, including public use)

• increased staff satisfaction (and hence performance), as teams feel ownership of the local level of care quality delivered (local use)

• improved data quality as a result of information being displayed locally in a tested approach (local use)

• improved standard of care and experience for patients by aiding service planning and decision-making (all levels, including public use).

It should, however, be noted that it was always intended that its use would be qualified, given inherent limitations in the data (see Chapter 10), which was factored in from the start of the design and development process.

Given the dashboard/toolkit’s purpose in disseminating information from free-text comments on the patient experience that can be used to help to drive health-care improvements, it was determined that some of the key features the version should have are:

• easy access to the data in a visual and usable format

• data provided in a way that can be engaged with by members of the public and by professionals

• summary data that can be mined down

• locally relevant information displayed alongside relevant regional and national data

• information provision that is relevant across all levels of use and with multidisciplinary teams, enabling organisational working –

  • front-line staff delivering care at the ‘coalface’ may benefit from the fine detail and will also benefit from seeing change through the years

  • the toolkit should facilitate discussion between commissioners, the clinical team and others to improve care quality, potentially acting as leverage to drive change

• information provision in ways that can meaningfully inform service improvement within the contexts of local care

• a full account of the limitations of the data and its appropriate use

• information provision in ‘real time’, when the data are available for this, or (as is currently the situation with annual large-batch CPES free-text data) analysis and display within hours to days after the data become available.

Requirements elicitation

In the workshops (see Chapter 6), participants were asked how they would use the CPES free-text data in a dashboard. It was thought to be likely by patients and carers, who dominated the early groups, that there would be differences between what health-care professionals would want compared with what patients would want. Patient and carer use suggestions were:

• making hospitals/the government accountable for common problems for people with particular conditions – participants mentioned that the publication of data can lead to change (and social media was suggested as a way to drive that)

• informing medical students on the patient experience to help their education (some participants said that this might help doctors to support patients)

• links to external sources of support (groups, etc.) – this was also considered by patients as diplomatic, so as to not give an impression of trying to put too much pressure on health-care professionals

• the desire to see comments made by ‘patients like me’ to compare experiences

• audit trail of comments by patients so that they can check that their comment has not been misconstrued.

In the structured walk-throughs (see Chapter 9), similar questions were asked of health-care professionals. Participants identified that the primary aim is to see where improvements need to be made in a timely and easy-to-understand manner (instead of the years that qualitative analyses can take) and to be able to use this evidence to make a justification for funding. Many also wanted to be able to use the dashboard to find out what is going well, to let patients know and to boost the morale of staff. A few said that this could also provide useful evidence for appraisals. Some mentioned that they would like to know that patients see that their comments are being used constructively.

Dashboard/toolkit features development

Data were accumulated on the desired features in three ways (see Chapters 2 and 3). Here, the summary of these and the prioritisation table are not reported, but it is demonstrated how this was an iterative process. See Appendix 5 for a few examples of the feedback received and the actions subsequently taken in the early stages. In subsequent sections (see Chapter 9), the final decisions are discussed in more detail.

In the first workshops, two different dashboard designs were presented for comparison, labelling these the patient one and the professional one on the basis of what had been learned so far during the project. However, it became clear that professionals also preferred most features of the dashboard labelled as the ‘patient’ dashboard, so that over time the two versions morphed into one.

There was the feeling among patients that they should have access to the same data as professionals, as this was ultimately their data. As per the protocol, a version of the dashboard suited to health-care professional use was completed and tested. The study team believes that this should also be the template for the patient dashboard, as a result of study findings, with some changes (for a discussion, see Chapter 9).

The feedback received from the early workshops mainly pertained to patients. This was a limiting factor at that time. Another limitation was that in using WCPES data, only data that had been vetted for an absence of personally identifiable information and a lack of specificity in the patient experience could be shown. Therefore, the system was tested in the group concept-mapping workshops using only 35 comments. Participants noticed that many of the data visualisations showed very few data points (in some cases pointing out that this may make such comparisons unreliable). This was also picked up during an AG meeting. As a result, it was difficult to demonstrate the value of the data visualisations with this data set, so that feedback pertained mostly to the dashboard navigation and design, and not the data visualisations. Unfortunately, this meant that data representations did not see much refinement, and their suitability for clinicians and patients is less known. On the other hand, the issues of structuring unstructured big data into too-small groupings became an unexpected topic of general debate.

Participants in the first two workshops and the user AG (UAG) suggested an Amazon (Amazon.com, Inc., Bellevue, WA, USA)-style chart (Figure 5), as a way of showing the distribution of sentiment and, therefore, the subjectivity inherent in the data. This was tested in the subsequent workshops, but once implemented, there was unanimity that this was confusing and appropriate only for the commercial environment in which it was used by Amazon, rather than for something like health-care services.

FIGURE 5.

Example of an Amazon-style chart for the responses for one hospital.

The study team also experimented with another Amazon-style graphic: stars to show the level of sentiment (Figure 6).

FIGURE 6.

Comment with star ratings. Star ratings can be seen below the comments (when the sentiment is unknown, there will be no star rating).

Coalescing dashboard efforts

As the dashboards were being developed, the programmers realised the potential for generalising certain aspects of the source code so that they were not both solving the same problems separately. They decided that there were two core areas of generalisability: a data application programming interface (API), which would be used to interrogate survey response data, and a visualisation API for generating charts. A core advantage of generalising these APIs is that components of the dashboards can be reused to easily develop new dashboard designs based on future feedback; this may also be critical when modifying the dashboard for alternative PES data or CPES modifications.

Difference between indicators and metrics

A metric is a numerical measure of a specified attribute, such as prostate-specific antigen level or the miles per hour being driven shown on a car dashboard speedometer, and needs to be precisely reported. An indicator, as the name suggests, indicates something about a metric – usually quality or safe levels. Thus, a car dashboard speed gauge shows whether or not the speed is within the legal limits. Indicators are often developed from rates or proportions. The co-design work showed that participants believed that the free-text comments should not be transformed into pseudometrics but should always be shown as relative frequencies or indicators.

Data presentation for indicators

Comparative displays used in NHS dashboards typically include funnel plots, bullet or spine charts, or bar charts. 61 As explained previously (see Chapter 3), non-NHS stakeholder participants in the study often liked pie charts (Figure 7) despite their inherent inaccuracies, whereas NHS staff preferred bar charts.

FIGURE 7.

Examples of pie charts from an early prototype.

Another option, commonly used in dashboards to show changes over time, is a run chart of the score of something (such as the number of negative comments) plotted over time (hence shown as a rate or indicator). This needs at least seven data points to reliably demonstrate a change, and so was not possible with the current dashboard, which would need data from 7 years, although it could be incorporated as needed. At the moment, the bar chart approach would be able to show comparisons, but would become overcrowded within 3–6 years.

Spine charts for indicators are created within NHS dashboards using Spiegelhalter statistical process control (SPC). SPC charts are generally recommended by the NHS because they enable both a focus on the median or mean position and graphical control limits. Thus, one small image conveys a lot of important indicator information – the essence of a dashboard. A value that falls outside 2 standard deviations (SDs) is typically considered as an ‘alert’, and if it falls outside 3 SDs it is conventionally considered as an ‘alarm’ within NHS dashboards. This is usually shown using RAG colours. SPC charts tend to help to reduce service variability, as teams aim to keep their services within the limits. This form of presentation was not thought of as helpful for the data by stakeholders, as the original data were not quantitative.

Overall, the visualisation choices that were made depended considerably on the reliability of the data (e.g. the acceptability of displaying small numbers of results on a graph).

Channels or platforms

The main access point was expected to be via a web browser. It had to be assumed that some displays would be old and with poor resolution. The system was also made to be reactive, so that it would adjust to different resolutions, screen sizes and smartphone displays.

Extract, transform and load facilities

The basic functions that the system had to do were as follows:

• Extract data from source systems into a defined format. The source data are generally supplied in CSV format, although any data that GATE accepts could potentially be used. CSV is the de facto interchange format for this type of data, and tools such as Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) and SPSS [(Statistical Product and Service Solutions) SPSS Inc., Chicago, IL, USA] will readily export to this format. The first rows may contain header information.

• Split the data from the CSV file into chunks. The data are processed in chunks within the GATE application so that the rule-based IR system can work on each chunk separately to keep random-access memory (RAM) usage to a sensible level. When processing > 3000 responses (each with three free-text responses), it was found that standard workstations were unable to cope with the RAM requirement.

• Generate the data file needed by the toolkit. The output file contains the theme classifications from the rule-based IR process and also a reference to the respondent. The data also include basic demographic information on the respondents to drive the toolkit filters that relate to those demographics.

• Loading of the data into the toolkit display. Two approaches to this were undertaken: (1) a Ruby script (Ruby 2.3; Yukihiro Matsumoto, Japan) that loads data stored in GATE’s native XML format and generates the file needed for the toolkit; and (2) use of the GATE’s JavaScript Object Notation (JSON) support to generate the toolkit file directory, negating the need for a separate script.

• Toolkit presentation. The toolkit shows a thematic overview of the data at the site (hospital) level. The overview level uses infographics to provide a general view, and when a user selects a particular theme to investigate, the individual text comments are shown for that theme.

• Reporting. Functionality that allowed for the creation of specialised reports summarising the data in filtered specified ways had been planned. This could not be achieved with current NHS data-sharing constraints.

• Manual data entry. This is a functionality in which a health service professional would be able to upload their own CSV file for presentation in the toolkit. This functionality was present in earlier versions of the toolkit, but restrictions on how the NHS data could eventually be used meant that the information needed to relate it to the user-provided data could not be shown.

• Modification. The toolkit is packaged as Hypertext Markup Language (HTML) files for easy transferability and modification. The toolkit website itself does not have an active server component – the files are plain HTML, image files, Cascading Style Sheets (CSS) (World Wide Web Consortium, Massachusetts Institute of Technology, Cambridge, MA, USA) and JavaScript (Google Chrome, Joyent Inc., San Francisco, CA, USA). This approach was chosen to limit the attack surface as much as possible, and to reduce the cost of server maintenance, as the files can be moved to a new webserver without having to port/upgrade an active web server application.

Infrastructure and security

• Data storage: provision of secure data storage will be via the University of Southampton secure data management services or NHS secure sites, depending on the agreements reached.

• Servers: provided by the University of Southampton.

• Information security: managed by a University of Southampton-dedicated site.

• Directory services: a University of Southampton site dedicated to the sharing of tools such as ours will be able to provide support documentation.

Conceptual information flow

In summary, the data are extracted using common data interchange formats (XML, CSV, etc.) through the modified GATE pipeline, in which they are transformed for display in the presentation layer (dashboard). Screen displays can be printed using the usual browser print facility.

Toolkit interactivity and transferability

The website is built using HTML, JavaScript and CSS. These files are available in a Git repository and the website files are built using a Ruby script (build.rb). Each specific page (located in the source folder) has unique content, and the script adds the website structure menus and the header/footer areas based on a template (stored in the template folder).

The script requires the commodity software package ‘Ruby’ to be installed. This is installed by default on Mac OS X (Apple, Inc., Cupertino, CA, USA) and is available as free software for Microsoft Windows^® (Microsoft Corporation, Redmond, WA, USA) from www.ruby-lang.org/en/ (accessed 19 July 2017). To build the website, simply run the Ruby script as a command:

After the script, the website files are available in the www folder. These files can be transferred to a webserver as static files to drive the toolkit. The commodity webserver package Apache HyperText Transfer Protocol daemon (HTTPD) (version 2.1; Apache Software Foundation, Forest Hill, MD, USA) was used to serve the files, but any webserver system, including WordPress (WordPress Foundation, USA), could be used to drive the toolkit website.

Toolkit navigation structure

An excerpt from the menu structure is defined in the build.rb file as follows:

These are the first four entries for the PRESENT toolkit website. Each menu item has a label, which is the text used for the visual appearance of the menu item. Each top-level item may have subentries using the items entry.

Each item may have either (1) a href entry with the relative URL of a page local to the toolkit; (2) an ext entry with the absolute URL of an external web page resource; or (3) neither a href nor an ext entry, which means that it is not a link and is shown only as a placeholder.

The excerpt above was used to create the portion of the website navigation shown in Figure 9.

FIGURE 9.

Navigation tab for the toolkit.

Feeding data from GATE to toolkit

GATE uses an XML file format (known as GATE XML), which the user can export to using the GATE Developer interface. This is the file format that contains all of the information that is generated by GATE, including intermediate annotations made on the data during the rule-based IR process. There is a GATE JSON format, but when an attempt was made to use this format, it was discovered that it did not contain all of the data of the XML format.

JSON is the preferred data format for websites. Graphing libraries import data from JSON (and often CSV) with ease, unlike XML, in which there are often too many different ways to define the data and manual intervention is often necessary.

Non-toolkit view of the data

For the manual validation of the rule-based IR process, a script was written that takes a collection of GATE XML files and produces a CSV file suitable for a person to see a matrix of comments and each possible theme that the analyser can produce. The script is called generate_review_file.rb and is located in the present_dashboard Git repository.

Each ‘cell’ of this matrix is actually three cells in Microsoft Excel

1. The output from the rule-based IR process. A ‘1’ indicates a match and an empty cell indicates no match.

2. The response from the reviewer in the same format. A ‘1’ indicates a match and an empty cell indicates no match.

3. A correlation field to be added after the review process that simply compares the first two cells.

To use the Ruby script, supply the XML files as command-line arguments, for example:

Once complete, the script will generate an output file with the name output.csv. This file can then be reformatted, as necessary, for the review process. The review process spreadsheet, as shown in the screenshot (Figure 10), was prepared by the usual Microsoft Excel functions for formatting and presentation. The cells were locked except for the second column for each tag, so that the reviewer did not accidentally modify content areas other than the intended areas. The greyed-out areas represent the comments, redacted for this report for governance reasons.

FIGURE 10.

Microsoft Excel spreadsheet for reviewing GATE outputs without using the toolkit.

The review process has three columns for every comment/tag combination:

1. the output from GATE

2. the assessment by the human reviewer

3. the correlation of the first two (as a Microsoft Excel formula comparing the first two cells).

Difference between group concept-mapping workshop discussions and traditional focus groups

In the PRESENT protocol, the choice was made to call the consensus-mapping workshops ‘large focus groups’ to emphasise the discursive elements. This does not mean that they resemble the small focus groups typically used in qualitative research. 125 Concept-mapping groups discuss divergences and similarities to develop group consensus. 120 Small focus groups are not intended to reach consensus, but to reveal group norms, a different concept. 125 Although small focus groups require homogeneity for successful interactions that converge to the group norm, large focus groups (group concept-mapping workshops) work best with heterogeneity. 120 This is important, as revealed by the stage 1 results in which the different perspectives of health-care professionals and patients became clear (see Chapter 3, Impact of the scoping stage on the initial prototype). For example, sexual dysfunction resulting from prostate cancer treatment may be variously highlighted as a side effect of treatment, a relationship issue or a body change with age, depending on perspective. In order for the themes-based toolkit to work effectively, that is, for it to be understood by health-care professionals while having meaning for patients, such differences in perspective needed to be reconciled through negotiated consensus. The study team was committed to making the toolkit intuitive to use and useful in driving health-care improvement and, therefore, more likely to be implemented within regular practice, with sustained use.

Aims and research questions for this part of the overall study

The main aim of this part of the overall study was to determine the labels and definitions to use for themes in the text-analytics rules and the six themes to be used for the default view in the toolkit.

The secondary aim, in this part of the overall study, in a linked qualitative discussion (see Study design), was to explore the perceptions and preferences for different toolkit design and functionality features.

Using concept mapping and follow-on interviews, the study team therefore aimed to answer the following questions:

• According to the different stakeholders in the patient experience, how should priority areas for feedback be labelled and why is this?

• What is the difference, if any, between what these stakeholders consider to be most feasible and most important?

• Do patients/carers and other stakeholders perceive these labels and rankings differently?

• Do stakeholders in the patient experience of cancer and other chronic conditions perceive these labels and rankings differently to stakeholders in multiple sclerosis (MS)?

Study design

A decision was made to diverge from the normal approach in several ways, in addition to the decision to use face-to-face group work throughout:

• The whole process was to be undertaken in one workshop session (see Figure 11).

• As the study team already had a set of > 8000 free-text comments (from the WCPES 2013) that it had begun to use in the developmental work, the usual first phase of the concept-mapping process – in which the statements or items are produced from a mindmapping session – was excluded and a selection of existing comments were used as the items to sort and rate.

• To ensure that no potential participants were excluded because they lacked computer literacy, the individual sorting, labelling and rating task was undertaken by hand, with researchers then inputting results into the software for automated MDS and clustering.

• The study team could have discussed the results as they were input, for example, by asking people for confirmation of particular label names. However, a decision was made to use the time to hold a workshop discussion around early prototypes of the toolkit.

The plan for the last phase of the process did not deviate from the usual concept-mapping process,120 that is, an in-depth discussion of the concept maps was held, thus aiming to form a consensus by the end of the day. Semistructured qualitative interviews with some participants in the weeks following the workshops were used to further explore areas of contention in which reaching a consensus had been tricky or impossible.

In a mentoring event held in Sweden in March 2016 for users of the software to meet the developers and discuss their projects, it was discussed whether the process should be used to validate the themes developed from the text-analytics process or to gather data to inform the process. The latter was preferred by the PRESENT team, and this was also the opinion of the method mentors, although it was pointed out that both approaches made innovative use of concept mapping.

Choosing statements

A total of 100 ‘statements’ (mostly single sentences) were initially chosen from > 8000 comments in the WCPES 2013. These were selected to represent the working set of themes decided on from the stage 1 work to enable us to build up the JAPE programming rules. The two key criteria that had to be met in preparing the statements for the concept-mapping sorting task were:

1. 1. Each statement had to contain only one concept, as it could be sorted into only one theme or pile by each participant.

2. 2. Each theme had to be represented by at least two statements (selected on the basis of the previous qualitative analysis of the data),21 as the sorting task requires each pile to consist of more than one item.

In addition, the study team aimed to include:

1. 3. Statements that were considered to be unambiguous. This was to steer the sorting to some extent towards the predetermined theme and avoid the generation of new themes entirely unrelated to the original analysis.

2. 4. Statements that were relatively ambiguous – thus, in the original WCPES 2013 analysis, a statement that was grouped under communication might also have been grouped under resources.

The third criterion in particular produced an inherent, strong bias in the task, but was justified, as the focus was on the development of a functional toolkit rather than the generation of higher-order conceptualisations. A more conceptually focused analysis had already been undertaken and had informed theme selection. 21 The fourth criterion was intended to balance this bias and to generate discussion around the ‘fuzzy edges’ of existing themes to tease out differences between the groups when decisions were less clear cut.

It quickly became clear that to satisfy all four selection criteria using 100 statements, not all themes could be included. The themes that were the most common and the most transferable across conditions were therefore chosen.

To select individual statements, a random list of five statements per theme from the manual WCPES analysis was generated, and this random generation was repeated until sufficient statements were found to fulfil the criteria.

Two researchers independently went through the statements and made their selections, with 76% agreement; differences were resolved by discussion to form the final selection.

To ensure that each statement would be considered independently, each one was given a random number (it was also ensured that cards were shuffled before being given to participants on the day).

Piloting the day and final choice of statements to include

Two pilots were undertaken. At the first pilot, the team worked out efficient ways of entering the data and practised this to determine their data entry rate, using one set of statements sorted into themes by an independent volunteer from within the Faculty of Health Sciences at the University of Southampton. The second pilot involved four volunteers from the faculty, naive to this type of task, and three research team members (CR, DT and Mike Bracher) undertaking the sorting, labelling and rating tasks at their convenience and timing how long this took them. The study team then entered sorted piles, taking care to ensure that a team member’s own pile was given to a different team member to enter. From the second pilot, participant and data entry capacities were able to be confidently calculated. During the first pilot, the number of comments was reduced from the planned 100 to 76. After the second pilot, the number of comments was further reduced to 60. This involved the same two-researcher process as in the initial statement selection process. This decision took into account volunteer feedback that the sorting task took 45–60 minutes, which gave team members a maximum of 3 hours to enter all of the data from up to 15 participants, depending on the workshop schedule. Data entry rates were 30 minutes per 76-statement pile, consistently across the team; thus, each team member could manage to enter data from six participants in the 3 hours, and the study team wished to ensure that should any one team member need to miss the workshop, it would still be manageable with 15 participants.

Topic guide development for discussion part of the day

The topic guide for the workshop was informed by the scoping review of the dashboard design (see Chapter 2) and the two developers working on the dashboard.

Screenshots from the first prototype dashboards and interactive exploration of the live toolkit were used to support the discussion.

Participant inclusion and exclusion criteria

The inclusion criteria were that participants had to:

• be a cancer or, depending on the workshop, non-cancer patient/carer or relevant health-care professional

• have a confirmed diagnosis to ensure that they had sufficient experience of the relevant services.

Using convenience samples, the study team aimed for a sampling balance across professional roles (service providers, charities, policy-makers, budget holders and commissioners). The aim was to sample patients/carers by demographics, health and health-care-related features (gender, age, cancer stage and type and treatment type). Each of the five workshops was intended to contain up to 15 such stakeholders in health care, mixed for heterogeneity.

Participants were excluded if they:

• were aged < 18 years

• could not travel to the group location easily and with same-day return

• could not provide full informed consent.

Participants were invited from across the nation, but the venues were in three UK cities only: London, Southampton and Leeds.

Recruitment procedure for the workshops

The study team approached the AG, study steering committee (SSC) and UAG for support. The UAG group members disseminated the ‘call of interest’ among their patient networks, including other groups with which they were associated. The AG and SCC advised us on particular professionals to invite. Macmillan Cancer Support, a collaborator on the study with representation on the AG, disseminated the call via its professional and user networks. The study team approached the local networks at the University of Southampton and made clinical contacts within Wessex. Clinical Commissioning Groups (CCGs) in London, Wessex and Leeds were e-mailed; the Leeds co-applicant (Adam Glaser) and the study chief investigator (CR) both drew on their clinical and commissioning networks for further participants.

The ‘call of interest’ leaflet and participant information sheet included an Eventbrite (Eventbrite, San Francisco, CA, USA) link for registration. This enabled the study team to better prepare for the day by monitoring the number and type of participants and ensuring that no single group was oversubscribed. Reminders were sent 48 hours before the meeting, with an electronic copy of the consent form and information sheet to give participants the time to familiarise themselves with these documents and make a note of any questions they may have wanted to ask about the study.

Transferability

Although the first four groups focused on cancer, the study team invited people with MS or people involved in their care to the last session to explore the transferability of the system to a very different long-term condition. The patients who attended this workshop were members of a well-established user group that regularly convened at the Royal London Hospital.

Format of the day, including workshop discussions

The workshops lasted up to 5 hours, usually between 10.30 and 15.30, to take patient accessibility issues and travel times into account. Refreshments were available throughout the day and a relaxed informal approach was used. Each session began with team and participant introductions, and an outline of the context and objectives of the study and the workshops. Participants had the opportunity to ask questions. Each participant received a hard copy of the information sheet and two copies of the consent form and was asked to sign the forms and return one to the team if they were happy with all of the tasks involved. All of the participants who attended gave their written consent. A team member ascertained and highlighted housekeeping, comfort and safety rules, assistance needs and confidentiality rules, and explained the plans for the day.

Participants were asked to read through and consider the 60 free-text comment ‘statements’, sort them into thematic piles in accordance with the sorting rules and name each pile. The rules were that they had to create more than one pile, no pile could contain only one statement, no statement could be included in more than one pile and no pile could be labelled ‘miscellaneous’. Participants were encouraged to name their piles in ways that were meaningful to them and were supported when they felt unsure about the process or lacking in confidence. The study team pointed out that the approach required no prior expertise.

Participants were then asked to rate each statement on the basis of how important they felt the issue to be, relative to other issues and assuming that all were feasible, using a scale from 1 (lowest) to 5 (highest). Relativity was important to ensure a good spread of rating values. A second rating focused on how feasible the participants felt it was to address the issue, given the current state of the NHS. This was used to determine theme ranking and, hence, default themes for the toolkit.

Both tasks were completed individually with support from the team, with 45–60 minutes allocated to this part of the day (Figure 11). Two or three team members (depending on the number of participants attending) collected the sorted and rated statements and entered them onto the concept-mapping software.

FIGURE 11.

A summary of the process that was used. All but the first stage occurred within the same workshop, and each took approximately 45 minutes.

While they did so, another team member facilitated discussion on the design and usability of the toolkit prototypes. Drafts of both the patient toolkit and the professional toolkit were reviewed; a third toolkit, for commissioners specifically, was considered in the first workshop as per protocol, but was subsequently discarded as it was not considered to be helpful. Each version was discussed for 45 minutes with a half-hour lunch break.

In the final part of the day, several iterations of cluster repartitioning were discussed in the group, until consensus was reached. The software default representation of clusters is one-fifth of the number of statements, which in the case of the study was 12. However, through simple instruction, the software can repeatedly recalculate the cluster boundaries as the group actively negotiates consensus. The fewer the clusters, the bigger and more spread out they are. Outliers may be closer to proximal clusters than to central points in the same cluster, meaning that clusters have too much conceptual overlap, or else clusters may be seen within clusters. Conversely, having too many clusters may lead to overfragmentation with weak conceptualisations. The facilitator sought to achieve a balance in cluster size; a rule of thumb is to have not more than four more clusters than the largest number of sort piles produced by a participant.

As part of the process, bridging values were examined; these show how often a statement was sorted with other statements in the cluster, with lower values indicating a closer conceptual relationship between statements. Labels that people had given to the cluster were also examined; the programme shows those given to the piles closest to the representational clustering. Importance ratings for clusters and for individual points were considered to help the group to make decisions. Finally, the study team discussed ‘regions’ in the map on the basis that clusters closer together might form a higher-order conceptualisation and that clusters at the centre tend to be higher-order concepts linking several other clusters.

The data were cumulatively aggregated in each of the five groups, ensuring that a consensus was reached across groups. The data from individual groups were also stored for later analysis of the similarities and differences between the groups.

Toolkit design discussions

After the first concept-mapping tasks, while team members entered participants’ data into the software, one researcher facilitated a discussion on the design and usability of the toolkit. An interactive, online version of the prototypes was shared on a computer display, driven by the data from the WCPES 2013, and screenshot printouts were provided. The professional and patient (public) versions of the toolkit were discussed separately and cross-comparisons were made. The discussion focused on:

• what the participants would like from the system and what they would use it for

• how participants would browse and navigate the toolkit and how the prototypes could be improved

• what the preferred ways of displaying the data would be (e.g. type of graphs, sentiment analysis)

• what features should be included (such as the ability to personalise what is displayed)

• analytics (e.g. filtering, search boxes, comparators).

Interviews

Immediately after each workshop, two researchers scanned the transcripts to identify areas of disagreement in the group discussion. This informed an interview topic guide. Semistructured qualitative interviews126 were then undertaken to gain a more in-depth understanding of the different arguments underpinning the disagreements. Thus, participants were selected from the groups for a follow-up interview if it was felt that they might be illuminating in this regard. The interviews also gave us a chance to invite participants to explore points that they may have been reluctant to talk about in the workshops.

Initially, potential participants were contacted by e-mail, and received up to two reminders by telephone or e-mail; in the second reminder, they were told that the study team appreciated their involvement in the workshops and would not contact them further regarding a possible interview, although they could contact the team.

It was intended to invite participants for interview within 2 weeks of the concept-mapping group to aid in their recall of the discussion. However, delays earlier in the project meant that the groups were held mostly in November and, therefore, many participants were unavailable for interview in the run-up to Christmas. As a result, some interviews were undertaken in January 2017. Therefore, the interval between group discussion and interview was a mean of 48 days (range 16–82 days).

Analysis of importance and feasibility ratings

The software ladder graphs were considered and Pearson’s r statistic for mean cluster ratings was calculated. Differences of interest between these in the comparisons were analysed using Welch’s t-test, a two-sample unpooled adaptation of Student’s t-test for unequal variances. 127

Reliability of the data

The study team considered the ‘stress’ value, a goodness-of-fit statistic routinely used in MDS analyses, which summarises the degree to which the cluster point map is a reliable representation of the conceptual relationships generated by participant sorting. The lower the stress value, the better the fit, the more reliable the results and the closer they may be considered to be to a representation of a larger population. 128

The stress values for each group were calculated, as well as combined values, to better understand group inconsistencies, as the groups were very heterogeneous.

Qualitative analysis

The discussions leading to consensus and the follow-up interviews were transcribed and analysed by two researchers using NVivo software to manage the data. An initial coding framework was developed using deductive coding based on stage 1 of the study and inductive coding from the first two workshop transcripts. The researchers discussed any ambiguities and disagreements, and modified the wording as needed to create an operationalised set of codes. The remaining workshop and follow-up interview transcripts were then divided between the researchers for analysis with further coding changes developed through discussion as needed. Given that the analysis was intended to inform the toolkit design rather than conceptualisations, a kappa statistic was not calculated.

Concept-mapping workshop participants

Overall, 34 participants took part in the concept-mapping workshops (4–9 participants per group), 26 completed the feasibility ratings and 25 completed the importance ratings. Tables 5–9 show their breakdown by role, health condition most relevant to them or their work and demographics. Most were patients with no one condition dominating, but many were not represented; cancer was not the only condition represented in any one group (condition homogeneity was not required, but complicates comparisons with the MS group). Participants were typically female, in their 50s and self-reported as being from a cross-section of ethnic groups (see Appendix 6).

Recruitment patterns

The sessions were predominantly attended by patients; recruitment of health-care professionals was challenging because the sessions involved a whole day. Some health-care professionals cancelled at the last minute because of changes in clinic times and other commitments. The number of caregivers was also lower than anticipated. However, many patients identified themselves as carers too, and were able to share their experiences from both perspectives. Recruitment issues were partially caused by the relatively short time from invitation to the workshops (approximately 1 month), which was itself attributable to delays and uncertainties in the finalisation of the prototype dashboards. The turnout was 70–90% of those who booked. However, the original goal of 15–25 participants per face-to-face session could have compromised the quality of discussions; this was confirmed by one participant who said that the group size was perfect.

Interview participants

As most workshop participants were patients, the study team was able to recruit patient interviewees from both the cancer and non-cancer groups. One academic and three professionals were also recruited. The study team failed to recruit carers per se, although two patients discussed both perspectives. As the population was limited to those who took part in the workshops, the study team considered inviting more patients, but reached saturation of themes at 12 (hence not needing to recruit the per protocol 15–20 participants).

Cluster labels and stress values

The five workshops led to 224 unique labels for sorted statement piles. Labels suggested by more than one participant were facilities, complaints, travel, transport, aftercare, resources, funding, communication and administration. Several other labels were similar to these. The final seven-cluster map with consensus labels is shown in Figure 12. This provided the most conceptually meaningful and internally consistent solution, as determined through the consensus-forming process among participants. Clusters were explored with participants, resulting in six themes chosen for the dashboard default:

1. referral from primary care and initial diagnosis

2. travel (e.g. buses, no parking)

3. staff attitude and role (e.g. empathy from nurses who are expected to be caring – referred to as ‘clinical positive’ in the diagrams)

4. information for patients (e.g. on treatments, health issues, external sources of support)

5. patient’s finances (e.g. claiming benefits, affording parking)

6. hospital resources (e.g. broken equipment, mixed wards, lack of beds).

FIGURE 12.

Final negotiated consensus cluster map for all participants and all workshops. Numbers in bold are cluster numbers; other numbers refer to the statements.

The final maps by role (i.e. for patients and carers, health-care professionals and others separately, not provided) showed us how boundaries and theme names were similar but not identical across these.

Individual statements (items), their clusters (themes) and bridging values were calculated for the three different groups by role. Among the health-care professionals (pile count range 4–14), contact and hygiene were the most tightly bound according to their bridging values, with stress values of 0.25 and 0.09, respectively, which means that these are the most conceptually distinct; other clusters in this group were less well defined. Among the academics and others (pile count range 6–11), attitude, communications, transport and facilities were the most tightly bound, with stress values of 0.24, 0.22, 0.04 and 0.20, respectively. Among the patients and carers (pile count range 6–12), waiting times and NHS issues were the most tightly bound, with stress values of 0.23 and 0.09, respectively. Clusters in the patient and carer group that were not conceptually distinct were administration and patient experience. The MS participants had more distinct clusters than the cancer participants; the latter focused on treatment and treatment consent issues, neither of which appeared in the MS group data. The differences between the MS group and the cancer groups were considered by a t-test of the four cancer groups versus all five groups, to ensure the same number of themes in each comparison group.

Sturrock and Rocha129 determined that for two-dimensional MDS solutions in which 100 objects have been scaled, stress values of < 0.39 indicate a < 1% probability that the MDS arrangement is random; in other words, this indicates good internal validity or a match between the participant-structured input and the final mathematically generated output. Stress values in all three groups were 0.27–0.29, thus groupings were unlikely to be random; as the stress values were all similar, this shows that all groups performed the task to a similar level. The relatively high stress values reflect the heterogeneity within each role group.

Ranking by priority and feasibility

The importance of themes in the final consensus map is shown by the depth of the cluster polygons (Figure 13), with the following all ranking highest: referral from primary care and initial diagnosis; travel; staff attitude and role; and information for patients.

FIGURE 13.

Bridging values for the final consensus map.

Feasibility ratings were similar, suggesting that participants were unable to bracket this off when considering importance ratings, with a Pearson’s correlation coefficient of 0.95 (Figure 14).

FIGURE 14.

Ladder diagram for participants, with importance ratings presented on the x-axis and feasibility on the y-axis.

For the cancer-focused workshops, it was found that the top themes by importance for patients and carers were legal and safety issues; staff attitudes; teamwork; diagnosis and primary care; funding and resources; and facilities and environment of the hospital.

For health-care professionals, the top themes by importance were legal and safety issues; staff training and skills; facilities and environment of the hospital; staff attitudes; teamwork/communication; and including patients’ family members in treatment and decision-making.

For the MS-focused workshops, patient and carer ratings for top themes by importance were hospital cleanliness; staff expertise and attitudes; patients’ consent and decision-making; family support; treatment choices; and hospital resources.

For health-care professionals in the MS group, the top themes by importance were staff expertise and attitudes; treatment choices; diagnosis; availability of support; human contact and empathy; and hospital cleanliness.

This heterogeneity shows the importance of including theme choices in the toolkit.

In terms of feasibility, the patients and carers in the cancer-focused workshops decided that the six key themes were legal and safety issues; staff attitudes; service and clinical quality; diagnosis and primary care; facilities and environment of the hospital; and teamwork/communication.

For health-care professionals, the key themes by feasibility were facilities and environment of the hospital; including patients’ family in treatment and decision-making; legal and safety issues; quality of care; staff attitudes; and teamwork/communication.

For the MS-focused workshop, the key themes by feasibility for patients and carers were treatment choices; cleanliness; staff approach, knowledge and empathy; information for patients; general practitioner (GP) experience; and staff expertise and attitude.

For the MS professionals, the key themes by feasibility were treatment choices; hospital cleanliness; practical support following diagnosis; staff approach, knowledge and empathy; family support; and communication and teamwork.

Dashboard design discussion

The findings from the workshop dashboard design discussions focused on four main areas: (1) outcomes and uses of the system; (2) functions; (3) presentation and output of data; and (4) data integrity, security and interpretation. These were further explored in the interviews.

The findings are summarised in Table 3.

Disagreements in facilitated discussion

The key areas of disagreement related to:

• the potential uses of the toolkit and whether or not patients would benefit from access to it

• functions available on the toolkit, such as enabling user input through a community forum or the facility to add comments to the page; choice of filters; what labels should be used to indicate the sentiment of patient feedback; whether or not access should be restricted by user registration

• issues of data presentation, including whether patients’ comments should appear in full or be summarised; the type of graphs perceived to represent trends most effectively; how to represent the data numerically; and how to best support an intuitive navigation through the toolkit based on users’ experiences with other websites

• concerns regarding the representativeness, anonymity and security of the data.

Example comments were:

I think as what I might call an ordinary patient I think probably not [would not use the dashboard]. I’m not sure that they would go onto it. I think it’s more of interest to patients like me who are representatives of groups, and taking a greater interest in perhaps seeking improvement in the way that the hospitals or whatever respond. And I think that the ordinary member of the public, who’s a patient, I think they’re just concerned with getting their own health back to normal. I think as far as other patients are concerned it’s going to be those patients who are what I would call campaigning.

Patient 7, group 3

Because I’m just wondering what the media could do with it, and so I’m just thinking about that pie chart. That actually media’s interpretation of the work that we do is usually incorrect. And it just made me . . . it did make me think that actually if we are saying four of these patients have experiences and one of them has experiences, if the media want to they could just take it and turn it into a percentage.

Patient 2, group 2

Amendments to themes and rules

At the time the workshops were held, some tentative rules and 36 theme names for the rule-based IR process had been developed. As a result of the group concept-mapping work, several of these were changed, resulting in a smaller taxonomy of 19 themes. The six default themes for the toolkit were those considered to be most important by the participants overall.

Although the study team was mainly guided by the overall consensus map, participants sorted only 60 statements, and the themes were not exactly the same in their scope as those used for the text-analysis rules, so at times new labels had to be added to the themes uncovered in the workshops or the suggested ones had to be adapted. Whenever the team members did so, they tried to draw on the discussion transcripts and to use the participants’ words.

Amendments to the toolkit design

The team met to review results and make final decisions on changes to the prototype for stage 3. Considering larger decisions, as a result of the workshops and interviews, it was decided that the patient and public version of the toolkit would need to be different from the health-care professional version and that each dashboard would be expanded into a toolkit. This was determined from comments about registration, data security, graph preferences and aims of using the toolkit.

Patients were more likely to use the toolkit on a one-off basis to learn about treatment side effects and experiences of others suffering from the same condition, and to gain information on a site where they were to receive treatment. In contrast, health-care professionals declared that they would use it on a more regular basis and would therefore benefit from more tailoring options to adapt the settings to their workflow. The theme selection needed to be relevant to their work environment, making it easy to identify which teams are able to address issues highlighted by the patients.

Strengths and limitations

The concept mapping was undertaken to enable mixed stakeholder co-design of the theme names and scope, as well as the dashboard (which they subsequently expanded into a toolkit). These goals were achieved. However, as a wider piece of work exploring the conceptualisations of different stakeholders in cancer care, the work has several limitations. The study team considered only a small selection of statements, smaller than the number usually used (≥ 100) when the process is undertaken online. Participants worked with 60 statements, yet the data set contained several thousands of comments that inevitably included a wider selection of themes.

Nonetheless, these represented 26 of the 36 themes (72%) that had been determined deductively before the workshops or that became evident through further discussion within the workshops, and participants did not consider there to be major gaps in the themes used. The data had good construct validity, as statements were taken from a large data set and because the original comments from which they were derived were relatively simple compared with longer texts. 131

The comments were selected by the study team from the WCPES 2013 free-text comments to cover a range of themes in a variety of ways (positive, negative, ambiguous and explicit) for maximal variation of topics, while ensuring that no topic was represented by only one comment. This process necessarily involved a top-down filtering approach – comments were chosen in accordance with themes identified in a manual analysis of the same WCPES data,21 then piloted and the selection was refined to ensure that effective sorting was achievable. More inductive theme development might have better external validity. Nonetheless, the final list of themes had satisfactory external validity, as the stakeholders themselves determined theme names and rankings.

Even a sophisticated computerised text-analytics approach, such as the one that the study team developed, cannot make judgements about the underlying relationships between two different actions or events, such that these still need to be based on the judgements of the researchers and programmers who decide on the rules, exceptions and cut-off points. 132 Thus, the more people contribute to these judgements, the greater the external validity. Nonetheless, the published meta-analysis has shown that 20–30 participants are sufficient for this in group concept mapping120 and this number was met. Data were collected from 34 participants in total; the original aim was 60 participants, but it was realised on piloting the approach that the size of each group (and hence the overall number of participants) had to be restricted owing to the time it took the team to input the manually sorted data. Such a small sample size inevitably limits generalisability. However, the study team never aimed for generalisability, but was rather adapting the process to enable co-design work. Moreover, the mean cluster rating for any given cluster is the average of all mean factor ratings within that cluster that reduces the variance within clusters, enabling statistically significant findings from small mean differences and from the small sample size.

The participants were mainly patients; the team intends to collect further data online from health-care professionals, but this will not include the face-to-face negotiations that were an important feature of the work.

This study has also shown the feasibility of adapting the software-based approach to enable full participation that can include people who are not computer literate. The team has demonstrated how a simple technique could be used to obtain information to modify the rule-based IR toolkit system for different conditions, and how concept mapping can be used in iterative intervention development work more generally. Outputs were composite summaries grounded in what participants decided and determined through a combination of statistical analysis, participants’ personal judgement and researcher expertise. However, it is also advised that developers exercise caution; ultimately, the products of this approach might be used as guidance but not as a definitive list, even when combined with discussions. The results can be tested in the future in further research.

Statistical theory for the design of discrete choice experiments in more detail

Common experimental designs use the full factorial (i.e. the total number of combinations of attributes and levels is used). For example, with five attributes of two levels each, the number of combinations would be 2⁵ = 32 or L^A, where L is the number of levels and A is the number of attributes. The main advantage of full factorial designs is that they possess enough information to estimate the main effects of all of the attributes, as well as all their interactions. However, the number of combinations grows very large as the number of levels and attributes increases. A first attempt to control the size of the design is to redefine or exclude certain characteristics or reduce levels. With failure to reduce the full factorial to a manageable size, the analyst can resort to what is called a fractional factorial design. By using a fraction of the possible combinations, they attain a manageable size, but at the cost of lost information and estimation power. However, looking at linear models, Dawes and Corrigan136 found that 70–90% of the variance may be explained by the main effects and 5–15% by two-way interactions. Therefore, depending on the research question of the study, designs for main-effects estimation in most cases are sufficient to elicit the information required for the problem at hand.

The efficiency of the fractional factorial design depends on certain statistical properties, orthogonality, level balance, minimal overlap and utility balance. 137,138 Orthogonality requires that all attributes are independent of each other (zero correlations). Level balance requires that the levels of each attribute appear with equal frequency. Minimal overlap implies that the alternatives within each choice set do not have overlapping attribute levels. Finally, utility balance requires that the utility of each alternative within a choice set is the same. This final principle ensures that there are no ‘dominant’ alternatives within a choice set (i.e. alternatives that are ‘better’ at one attribute and at least as good on the rest of the attributes, which would force the rational individual to choose them without making any trade-offs between the alternatives’ characteristics). These four properties are combined to ensure maximum D-efficiency (a statistic based on the determinant of the covariance matrix), which itself is a relative measure compared with the efficiency of a fully identified orthogonal design.

Identifying attributes and levels

In the development of the PRESENT toolkit, 21 potential attributes, excluding cost, were initially identified from the rapid review of the literature (see Chapter 2) that were also relevant to the prototype toolkit design that had been developed. The cost-attribute values were determined through cognitive interviews and piloting of the survey. The study team elicited from potential users the maximum that they would be willing to pay for some of the combinations they liked most, and also whether or not this should be costed as an individual or team purchase.

The initial design used 12 choices per respondent. Each of these choices involved three hypothetical toolkits. Within each choice, the three toolkits varied in only 5 out of the 12 attributes. Two versions were developed, one for patients and one for health-care professionals. Some attributes were generic across these and some were suited to only one or the other; the final list contained 11 attributes per version and, therefore, one attribute had to be lost from each version, as the design requires even numbers. Following initial feedback on the DCE design, it was felt that it might be too difficult for patients to understand because they were not used to the type of toolkit being explored, and the study therefore continued with only the health-care professional version.

The study team ensured inclusion of some attributes that were deterministic, in other words, that all participants would be expected to either prefer or disprefer. For example, it was known with enough certainty that respondents prefer lower prices for a given product, and hence the sign of the price attribute would be negative. Similarly, it seems likely that whether or not users could save data would have a positive sign (i.e. for a given product, users would prefer to have the option of saving rather than not). This prior assumption about the influence of an attribute helped Emmanouil Mentzakis to create a more informative design for the task at hand; it was important to optimise the survey, as the study team expected to recruit only low numbers of respondents.

The final list of attributes and levels is given in Box 1.

Piloting

The attributes and levels were piloted online on a small sample of three health-care professionals who fitted the profile of potential users to gauge clarity and understanding of the survey and the experiment. Formal feedback was sought via a comment window incorporated in the survey, and a number of issues were raised:

1. Interesting survey and well explained but there are some complex instructions/ideas to get your head around which may put people off.

2. It’s interesting and very well explained but it is quite difficult, not something you can just dash off in 5 minutes! As long as people know that, I think they will be happy to work through it. Also, if your target health professionals are interested in using the dashboard, that will focus their minds.

3. It will be important to emphasise that this survey will take a while to complete.

4. At the start saying you will do 10 choices could be the difference between people giving up or not, if you have done six and know there are 10 you might be able to push yourself to finish it.

5. At first I was confused about the additional question that pops up at the bottom of a page once you have made your selection – it asks ‘Suppose you were given the option to opt out of subscribing to this service . . .’ but I did see why it was being asked once I’d gone through one choice set.

6. Very nice survey design.

As a result of the pilot user suggestions, the main change was to make the text on the welcome page and instructions page shorter, and including bold and underline to highlight the most significant information.

Further feedback was sought immediately after these modifications via cognitive interviews with two stakeholders. This was fewer than intended; the aim was to interview five stakeholders, but the survey took time, there was no compensation for helping us and the team was very short of time. However, further changes were made:

• The text on the welcome page and the instructions page was further shortened, broken into smaller paragraphs and the font size was increased.

• A graphic was added to the home page to make it more visually appealing.

• A link to a prototype toolkit, which had been included in the survey to enable respondents to get a feel for the product, was dropped and replaced with a screenshot. The interviewed participants had not seen any relation between the toolkit prototype and the survey, and were confused as to why they were asked to engage with it.

• A table explaining the different toolkit features was made more comprehensive and definitions of the features were made clearer. The link to this table, which was previously displayed vertically next to each choice set, was changed to display horizontally under the choice set.

• A clearer definition of the cost attribute was provided and the subscription prices were increased to correspond more closely to those currently found on the market.

• A conditional question about the pay bands/grades was added for the NHS staff respondents.

As many of these changes were made because of feedback that the survey was complicated to complete, the study team discussed whether or not training sets should be used. However, given that this would add to the cognitive burden, the team decided against this. The team also discussed a visit–revisit option. However, it was considered that this would reduce validity, as participants might forget the theme, topic and context.

Experiment design

Following the identification of attributes and their levels, a statistical theory was used to create a DCE with eight attributes of two levels and two attributes of four levels. To obtain enough information to identify the main effects of this size, each respondent would have been required to respond to a minimum of 15 choice sets of two alternatives each (i.e. 15 pairwise sets of two alternative hypothetical toolkits). However, with 10 attributes (potentially with different levels) in each of the hypothetical alterative toolkits, the cognitive burden placed on respondents to process this information and indicate their preferred alternative in each of these pairwise choice sets was deemed to be too large.

For this reason, a number of steps were taken to reduce such cognitive burden and render the DCE more manageable for the respondents. First, the number of hypothetical alternative toolkits within each choice set was increased to three (from two). This automatically reduced the number of choice sets required per respondent to eight. Second, given that all 10 attributes could vary simultaneously among the three alternative toolkits, implying that comprehending and evaluating the possible trade-offs among all attributes across all three alternatives for each choice set was impossible, an alternative type of design was used, namely partial profile design. Such designs vary only a limited number of attributes among the three alternatives in each choice set, but with different attributes across different choice sets. For this project, it was deemed that 4 out of the 10 attributes would vary in any given choice set (i.e. a different group of four attributes across different choice sets). This convenience and discount in cognitive burden, however, comes with the cost of a larger number of choice sets that each respondent must see. Finally, using one of the features of the experimental design software, three different versions of the exact same design were created (each respondent receiving only one version) that allowed more variation in the combination of attributes and levels presented to individuals.

To improve the power of the experiment, it was originally intended that the three versions would be distributed randomly to participants, with a balance of respondents across versions. However, the study team discussed the problems that a computer-driven random allocation would cause if people started but did not complete the survey, and decided against the randomisation.

Forcing individuals to indicate their preferred alternative out of the three in each choice set results in what is commonly called a ‘forced-choice’ experiment. Individuals are forced to trade off between the attributes of the alternatives without the possibility of choosing to decline the purchase. However, in such cases, it is impossible for the researcher to elicit true purchasing behaviour, given that the corresponding real-world choice would be first whether or not to purchase a toolkit service at all and, subsequently, which version one would like. 139 For this reason, a two-stage elicitation strategy was implemented, whereby individuals were first asked which of the three alternatives they prefer and, subsequently, whether they would rather keep their preferred alternative or opt out altogether if they were given the chance.

Construction of experimental designs from first principles was not necessary; JMP software, version 11 (SAS Institute, Marlow, UK), was used, which contains preprogrammed routines that are able to create tailor-made experimental designs.

Survey administration

The survey was platformed on the iSurvey toolkit (University of Southampton, Southampton, UK). Over 200 stakeholders were invited to participate through the existing professional networks and Macmillan Cancer Support and its digital team. The study team contacted all of the CCGs in England, as well as a number of cancer charities. The professionals who had participated in other stages of the study, had previously shown interest in the study or had attended the launch event were invited and encouraged to share the link with colleagues. The survey was also disseminated via members of the AG and steering committee, and shared within the Faculty of Health Sciences at the University of Southampton. The survey was open for 9 weeks, and there was one reminder.

Participant descriptive statistics

The mean age of the 32 completer responders was 49 years, with 81% being female and 38% working for the NHS (Table 4). Among the 38% of responders working for the NHS, the most common band was band 7 (50%), and management was the most common professional area (58%). This was the target group for the DCE, as the responders were likely to be able to influence the use of the toolkit and make use of it. Individuals with academic roles constituted 53% of the respondents; many were academic clinicians (incomplete reporting means that a final figure cannot be attributed to these respondents).

Discrete choice experiment data

The analysis of the DCE data was based on random utility theory140 and discrete choice econometric models. 141 Random utility theory assumes, as in neoclassical economic theory, that individuals have perfect information on their preferences and perfect discrimination capabilities. 140 On the other hand, the researcher has incomplete information and does not observe the full decision process of the individual. Hence, a component of uncertainty (with various potential sources) has to be included in modelling utility. The utility of the ith alternative for the qth individual is:

where:

is the deterministic part of the utility (with k attributes) and ε_iq is the error term capturing the uncertainty and the unobserved (by the researcher) part of individual heterogeneity. β_k are the utility parameters, independent of q (i.e. homogeneous across the population). Assuming that the error term is independent and identically distributed, extreme value type 1 gives rise to the McFadden’s conditional logit (CL),142 in which the alternative i is chosen out of j alternatives with probability:

The probability in the CL depends on differences among alternatives and, therefore, attributes that do not vary by alternative will not influence the probabilities. Hence, demographic characteristics cannot directly influence the choice within each choice set. Similarly, given the use of a partial profile design, any attributes that do not vary among the three alternatives in any given choice set (i.e. 6 out of the 10 attributes) also cannot influence the choice in the choice set (i.e. any variation in choices comes from differences in the preference over the attributes that vary within a choice set).

Taking the CL further, models have been developed that relax some of its restrictive assumptions, mainly with more flexible structures in the covariance matrix of the model. A common specification is the nested logit (NL) model, in which the alternatives are grouped into nests (i.e. purchase or not purchase), under the assumption that the independence of irrelevant alternatives property holds within each nest but not across nests. 143 The probability of the individual q choosing alternative i from nest g is the product of the probability of the individual choosing an alternative from nest g (i.e. choosing nest g) and of the probability of choosing alternative i conditional on nest g:

which is the product of two simple logits. Assuming that there are indirect utilities with both of these decisions and rewriting:

From above, the probability of the NL is:

where:

is the inclusive value, which links the two levels of the nested model and is used to establish the extent of dependence or independence between linked choices. 134 λ_g is the scale parameter to be estimated in the range 0–1. Every level in the NL has its own scale parameter, and for identification reasons, one of them has to be normalised to 1. Table 5 shows the DCE estimation results.

Willingness-to-pay values

Following the estimation of models and coefficients for each attribute level, WTP values can be calculated. Coefficients in DCE models depict part-worth utility (i.e. the contribution of that attribute level to the utility enjoyed from the alternative), and for this reason present the relative importance of attribute levels. By taking the ratios of coefficients, one can calculate the marginal rates of substitution between them [i.e. how much of one attribute level one is willing to lose to gain more of another (Table 6)].

Incorporating a price component (as a numeraire) into the attributes list allows one to translate the trade-offs and marginal rates of substitution between attributes into their monetary representations. These WTP values represent the amount of money an average individual is willing to pay to obtain a service/product that has a specific feature attribute level. For the CL, the WTP calculation would be:

Predicted probabilities

Further inference on the results can be drawn from calculating predicted probabilities. These probabilities represent the likelihood that an individual will or will not purchase a given product, or the change in the likelihood of purchasing or not purchasing a product when its features change. Specifically, three types of predicted probabilities are presented.

The first one is used to provide some better insight into the magnitude of the effect of each attribute level (i.e. how much it contributes to the attractiveness of the product). For this, the study team initially defined a baseline product with the features shown in Box 2.

Subsequently, the team created alternative toolkits that are identical to the baseline with the exception of one attribute level. For instance, such an alternative toolkit would be one that was identical to the baseline but with the capability to upload own data (i.e. attribute: upload own data – yes). The possible configurations (i.e. the number of possible alternative products) are numerous, so, without any loss of generality, the team members restricted themselves to changing only one attribute level at a time (see above), which resulted in a total of 12 alternative toolkits. The probability of purchase for each of these alternative toolkits was then calculated and compared with the baseline. This difference indicates the change in the predicted probability of purchase when one attribute level changes with respect to the baseline.

Given that the study team intentionally chose the baseline toolkit to contain the least desired level of each feature, with the exception of price when it was the cheapest, probability differences were expected to be positive, implying that a higher number of desired features increased the probability of selecting the toolkit.

The second type of predicted probability followed a similar logic, but gave an assessment of the importance of each attribute level in preventing an average individual from opting out of purchasing the toolkit. In this case, the probability of opting out for the baseline, as well as for all other alternative toolkits, was calculated and, again, compared. More desirable features were expected to reduce the probability of opting out.

Given that, in these calculations, the presence of the opt-out as an option was necessary, this was possible for only the second stage of the DCE choice. Recall that a two-stage elicitation strategy was implemented, whereby individuals were first asked which of three alternative toolkits they preferred and were subsequently offered the option to keep their preferred alternative or opt out altogether.

The third type calculated the unconditional predicted probability of purchase and of opt-out (i.e. decline purchase) for three hypothetical toolkits (i.e. the baseline, a fully featured low-price alternative and a fully featured high-price alternative). Such figures give a sense of the overall tendency to purchase from the sample of respondents and can aid in building market scenarios that can guide market placement and revenue streams.

Estimation results and product valuation

Out of 152 individuals starting the survey, 33 respondents completed it (i.e. completion/retention rate of about 22%). Although the identification of coefficients does not rest on sample size (i.e. the experimental design ensures that enough information is available to estimate parameters), the generalisability or robustness of the findings would therefore need to be verified.

Estimation results

Table 7 presents the estimation result from three models. The CL can be used to estimate the first stage of the DCE question (i.e. forced choice without an opt-out option) and the first and second stage of the DCE questions combined (i.e. three alternative toolkits plus an opt-out option to choose from). This latter combination (i.e. first and second stage combined) is also analysed using a NL. Results for all three are given in odds ratios, which indicate how much more likely one is to choose an alternative that features an attribute level compared with the baseline.

Common patterns appear across models (see Table 9). Being able to filter data (by gender AND age AND ethnicity AND condition/illness) seems to be the most influential feature, followed by the ability to search for a hospital from a drop-down list. All three search features are highly desirable for respondents, although the ability to upload one’s own data is almost equally important. Furthermore, filtering comments (by staff roles) and customisation of graphs also appear to be important, whereas the use of lay language and indicators display achieved limited significance. Interestingly, data resolution and predictive intelligence were not identified as important in any of the models. Finally, the cost of the toolkit was highly significant, suggesting that respondents were sensitive to price changes as predicted by economic theory. Overall, on average, individuals were very likely to decline to purchase the service (i.e. almost nine times more likely to opt out than to purchase it) at the highest price compared with the lowest price, and females were much more likely to do so, as indicated by the NL model.

Willingness-to-pay monetary valuations

Table 8 presents WTP calculations. Given that WTP calculations are essentially non-linear combinations of the estimated coefficients, statistical significance and relative size closely follow the coefficient results. All three models point to an identical list of important attribute levels, albeit there is some variation in the magnitude of the elicited WTP. This is largely explained by the different model assumptions and other model features and for the vast majority, WTP calculations were not statistically different across the three models.

Overall, respondents were willing to pay around £1674–2447 for their most preferred search feature (i.e. a drop-down list of hospitals), followed by the filter (by age, gender and condition), which was valued at £1644–2142. The other two search options, keyword and map/postcode, were valued at £1364–2062 and £1357–1461, respectively. The capability to upload one’s own data was valued at £1198–1377, and finally, filtering by staff role was valued at £736–941.

It should be noted that although the rest of the attribute levels do not appear to be as significant, this does not imply that they are not desired in the toolkit. As discussed previously, in Willingness-to-pay values, coefficients convey a relative piece of information, namely, how much more a feature is desired compared with the baseline or the baseline level. For this reason, statistically insignificant WTP values imply that respondents did not systematically value the toolkit with these attribute levels more than they valued a toolkit with the respective baseline attribute level. For instance, in the case of graphs [with two levels: (1) fixed graphs or pie charts and (2) user can choose and change graphs], an insignificant WTP suggests that respondents were not willing to pay more for a toolkit with the user-specified graphs than they were for a toolkit with fixed graphs. This is vastly different from the notion that graphs are altogether not important to the toolkit users.

Predicted probabilities calculations

The results from the predicted probabilities calculations are given in Table 9. Recall that the values in the table are changes in predicted probabilities and, for this reason, are interpreted as percentage-point changes. Positive values imply an increase in the chances of purchasing an alternative toolkit (i.e. identical to the baseline toolkit in all aspects, apart from the specific attribute indicated) compared with the baseline toolkit. Correspondingly, negative values suggest that the attribute level evaluated is less desired than the respective baseline attribute level and, for this reason, the chances of purchasing the alternative toolkit are lower than those of the baseline toolkit.

All three models point to a similar ranking across attribute levels, with the search feature of a drop-down hospital list increasing the chances of an alternative being selected by 32 percentage points in the forced-choice model and 28 percentage points in the NL model. Equally large changes in likelihood were obtained by the filter (by age, gender and condition) feature, increasing the chances by 32 percentage points and 25 percentage points, respectively. The remaining search levels and data-uploading capabilities follow. Looking at the price effect, an increase in the annual fee from £250 to £1500 drops the probability of purchase by around 18 percentage points in the forced-choice model and 10 percentage points in the NL model.

Moving on to the second type of predicted probabilities through which the change in the predicted probability of opting out was calculated, negative values imply that an attribute level will result in a lower probability to opt out (i.e. a desired feature), whereas positive values suggest an increased chance of opting out and not purchasing (i.e. an unattractive feature). For both the CL and NL models, it can be seen that the changes in the opt-out probabilities were relatively small, with around a 13-percentage points and 3.5-percentage points drop in the CL and NL, respectively, for filtering (by age, gender and condition). Search features follow, with values between 7 percentage points and 9 percentage points for the CL and 4.2 percentage points and 2 percentage points for the NL model, suggesting that all features are desirable and reducing the chances of opt-out.

It is noted that such drops in the opt-out probability potentially appear to be small. However, this is to be expected, given that there is a large tendency of respondents to opt out of purchasing the baseline toolkit, which, in essence, is the least desirable toolkit that can be configured, given the attributes specified. To gain insight into the overall attractiveness of the digital toolkit and the potential market demand, the third type of predicted probabilities is presented in Table 9. Individuals in the vast majority of cases (i.e. 91%) declined purchase for the baseline toolkit, but would choose to purchase when a fully featured alternative is offered, with 89% for the lower priced options and 80.5% for the higher priced options. From this, the effect of price on a highly desirable toolkit can also be seen, whereby an increase of £1250 (i.e. from £250 to £1500) reduces the probability of purchase by around 8.5 percentage points.

Basic cost–benefit evaluation of the online toolkit

Given the results of the DCE, a basic cost–benefit simulation exercise was performed to evaluate the profitability of a digital toolkit venture. This may also be useful when considering transferability. An evaluation for the three toolkits from Table 9 was performed (i.e. the baseline, a fully featured low-price alternative and a fully featured high-price alternative). Furthermore, it was assumed that the toolkit was offered to 100, 500 and 1000 potentially interested clients and the product uptake purchasing behaviour was calculated from Table 9. For the calculation of costs, only the initial development cost was taken into account (i.e. the value of the grant: £412,242) and a varying annual maintenance cost was assumed, depending on the number of clients (i.e. £50,000 for 100 potential clients, £100,000 for 500 potential clients, £150,000 for 1000 potential clients). Finally, a product life of 5 years and an interest rate of 3% were assumed.

From Table 10, it is apparent that the baseline toolkit is not a viable option for any number of potential clients, largely because of its very low chances of uptake (i.e. even for 1000 potential clients, only 90 purchases are predicted). Moving onto the full-featured option with the low-pricing strategy, the product becomes profitable over a 5-year period only for 1000 potential clients (i.e. 890 purchases per year). Finally, for the full-featured option in the high-pricing strategy, the product becomes highly profitable for 500 potential clients (i.e. 403 purchases per year) and 1000 potential clients (i.e. 805 purchases per year).

Statistics

A standard contingency table was used to classify performance (Table 11). For a binary classification problem, the table has two rows and two columns. Across the top are the observed class labels and down the side are the predicted class labels. Each cell contains the number of predictions made by the classifier that fall into that cell. Using this table, standard calculations were made for accuracy, precision, sensitivity and the F-score.

Classification accuracy is a common performance statistic. It is the number of correct predictions made divided by the total number of predictions made, expressed as a percentage. However, accuracy statistics alone should not be relied on, because, for example, it may be desirable to compromise accuracy to achieve greater predictive power. This is known as the ‘accuracy paradox’:

Precision is the number of TPs divided by the number of TPs and FPs. It is also called the positive predictive value. Low precision (< 1) can mean that the system throws up a large number of FTPs:

Sensitivity is the number of TPs divided by the number of TPs and the number of FNs. This is also known as the recall or TP rate:

The F-score shows the balance between precision and sensitivity. Thus:

When calculating these statistics for the WCPES data, the study team used the half data set that had been reserved for this purpose (having built the analytics from the other half).

Adapting the approach for the analysis of Life After Prostate Cancer Diagnosis study data

The LAPCD study is a UK-wide patient-reported outcomes study that has generated information to improve the health and well-being of men with prostate cancer. 151 Prostate cancer survivors (18–42 months post diagnosis) in all four UK countries (≈70,000) were identified through cancer registration systems and were sent postal surveys. A survey instrument was developed, covering a range of generic and cancer-specific patient-reported outcome measures, with additional items covering treatments received, sociodemographic characteristics and the patient perspective on their disease, treatment and experiences. The survey was structured into sections covering specific issues facing prostate cancer survivors [e.g. treatment decision-making (TDM) and decision regret; emotional well-being; and coping and self-management]. In addition to the closed-response items, eight free-text response questions were included at the end of each survey section for respondents to add further detail or to capture other relevant issues not covered in the section.

The adapted GATE software was used to identify themes within the responses to two free-text questions. First, at the end of section 3 of the questionnaire, which elicited 10,358 comments, a free-text question asks respondents the following:

• Please add anything else that you would like to tell us about your diagnosis, treatment and the decision-making process.

Second, a final ‘catch-all’ free-text question, to which 6682 respondents provided comments, asks the following:

• Is there anything else you would like to tell us about what life has been like for you following your prostate cancer?

This catch-all question was analysed as a direct test of transferability.

With regard to the first question on TDM, the LAPCD data research team involved four members of the study UAG to help inform the development of a gazetteer. Following a teleconference between the research team and the UAG on 30 March 2017, a random sample of 400 free-text responses to the TDM question were equally divided between the UAG members. The UAG members then read the comments and identified words and phrases that indicated comments describing the TDM process experienced; the treatment options with which they were presented; the level of involvement they had in the decision-making process; the amount of information and preparation they received concerning potential side effects; and their subsequent satisfaction with the treatment received. UAG members returned these TDM lists of words/phrases to the research team members, who then collated them and categorised them under subheadings.

Immediately preceding the TDM free-text question in the LAPCD data questionnaire is a closed question asking respondents to indicate their level of involvement in the decision-making process that determined their treatment (Box 3).

Comments provided by respondents were categorised on the basis of the level of involvement that patients indicated they had in the TDM process. This was done to identify common themes within the comments pertaining to levels of TDM involvement.

This information was used to develop gazetteers and rules to test for transferability.

patientopinion.org data

The study team obtained 10,000 comments from patientopinion.org (now known as Care Opinion) to test the approach on a different form of patient feedback free-text comments with often lengthy narratives.

Recruitment

The study team chose participants on the basis of geographic and sociodemographic variation, and poorer and better performance in the CPES 2014. Participants were selected from cancer services, and also other health areas, to enable the study to consider the transferability of the intervention to other specialties.

Setting

Walk-throughs took place at participants’ workplaces, and lasted an average of 2 hours per participant. An iPad (Apple Inc., Cupertino, CA, USA) was used to video-record the session. The iPad was set up on a flexible arm attached to the participant’s desk, with their hands and computer screen in view. An audio-recorder was placed on the desk, the participant was asked a final time if they consented to proceed (participants were reminded that they could stop at any time), then recording and the evaluative process began. Typically a scribe attends walk-throughs with the moderator (facilitator), to take notes and produce a list of required changes for the developers. After the first three sessions, the study team realised that, as team members were video-recording the session, the scribe was not needed. Therefore, the remainder of the walk-throughs were continued with just one researcher present who facilitated, scribed and recorded.

Heuristic evaluation (standard usability principles)

The starting point was to rely on the classic principles of (1) usability engineering and (2) the support of psychological flow in information systems, and to define these as benchmarks to be evaluated by users. 152,153 The first stage of the evaluation was therefore a heuristic evaluation. 154 This helps to identify usability problems in a user interface through potential users (i.e. evaluators) who examine the interface and judge its compliance with recognised guidelines and usability principles (the heuristics). This evaluation method takes less time and uses fewer resources than a full systemic evaluation, but is widely seen as being successful in uncovering usability problems during the design phase. For this reason, it is widely used in the industry in the design of user interfaces and especially in the design of new media. The guidelines that were selected to ask participants questions about were identified by Jakob Neilsen155 and Jill Gerhardt-Powals. 156

Participants were shown the website with the toolkit for the first time (naive) and asked to use the toolkit on their own without any prompts. They were then asked to read themselves the following questions, answering in an instinctive ‘think-aloud’ manner (facilitated by the researcher, with questions and responses spoken aloud and audio- and video-recorded):

• Does the toolkit keep me informed about what is going on through appropriate feedback within a reasonable time?

• Does the toolkit speak my language using words, phrases and concepts familiar to me?

• Do I find that I make errors when using this toolkit?

• If I commit an error, is the help that the toolkit provides clear?

• How do I correct something if I make a mistake?

• Do I have to remember a lot of information, or are prompts, etc., visible and accessible?

• As I become more familiar with the toolkit, can I take shortcuts and achieve the same result?

• Is the visual effect of the toolkit confusing?

Severity ratings

Severity ratings are useful in making clear to designers the magnitude of usability problems, and to allocate resources to address those problems. It is not advisable to release a system that has severe usability problems that are persistent, but one might decide to go ahead with the release of a system if the problems are cosmetic. 157

Participants were asked for information that enabled the research team to rate the usability problems encountered by participants:

• The frequency of the issue: is it common or rare?

• The impact of the issue: how easy is it to overcome it?

• The persistence or recurrence of the issue: will users overcome the problem after they encounter it once, or will it be a continuing problem?

Participants were also asked to rate the severity of the usability issues that they encountered:

• 0 = I don’t agree that this is a usability problem at all

• 1 = cosmetic problem only – does not need to be fixed unless extra time is available on the project

• 2 = minor usability problem – fixing this should be given low priority

• 3 = major usability problem – important to fix, so should be given high priority

• 4 = usability catastrophe – imperative to fix this before the product can be released.

Goal-oriented evaluation

Following the heuristic evaluation, a goal-oriented evaluation was conducted. Task- or goal-oriented evaluations focus on the extent to which a system supports the goals or tasks set by users. This is a logical extension of a heuristic evaluation, but although a heuristic evaluation evaluates the usability from a user-interface perspective, a task-oriented evaluation examines whether or not the system actually helps users to achieve their goals or complete their tasks. 158

Participants were asked to examine the toolkit and ask themselves the following questions (facilitated by the researcher, with questions and responses spoken aloud and audio- and video-recorded):

• What is my goal of using this toolkit?

• What actions do I need to take to achieve this goal?

• Is it obvious to me how to take this action?

• What feedback do I need from the toolkit to achieve my goal?

• Am I getting appropriate feedback?

To help the participants with this and other tasks in the session, the team members suggested that they imagine real goals that they might try to use the toolkit to achieve.

Implementation (Normalisation Process Theory) and diffusion of innovation

After the walk-through, the study team determined:

• the relevant knowledge and skill set of the evaluators

• the evaluators’ roles and reasons for potentially using the process

• the evaluators’ ability to support others to do so.

For this, the team developed walk-through topic guide extensions to the main questions in the various walk-through components, including questions drawing on diffusion of innovations theory and network contagion theory, as well as NPT.

Participants

The plan was to recruit 15 participants, but because of time pressures, the study team stopped at 13 participants, hence not following up the six who had agreed to participate and were booked, but then cancelled. All participants were nurse directors and managers, and they came from across Wessex, London and Leeds. Table 12 contains information on those recruited.

Heuristic evaluation of the toolkit

A summary and solutions implemented in the study-end toolkit prototype are shown in Table 13. In terms of usability, all participants found the toolkit simple and ‘easy on the eye’, and liked the visual summary, with the further ability to see actual comments sorted into themes. But most participants found the infographic dial in the prototype uninterpretable. This is being changed in the final prototype to bar charts. Although visual simplicity was considered an asset, all participants wanted more navigation tools (e.g. select all, back, clear all, keyword search and help/prompts available on the dashboard, not on a separate page).

Two senior clinicians and a Macmillan Cancer Support lead observed that it might be hard to reconcile the differences between performance targets and patient experience – being good at one does not necessarily equal good feedback in the other. They commented that a manager can be ‘sacked’ for poor performance but not necessarily for poor patient experience, and that major improvements may require long-term, whole-culture changes.

When participants were asked to rate the severity of problematic issues they found during the heuristic evaluation, most agreed that the prototype was not market ready; infographic (dial) re-design was needed for simple interpretation and the navigation tools needed improvement; otherwise, the system was highly desired. Some of these issues were subsequently addressed.

Wider website design comments were also made. Most participants suggested the following: change the images on the ‘Welcome’ page; reduce the wordiness of the ‘About’ page; the ‘Background’ page could include a summary of data collected from the CPES that was used to populate the toolkit and key findings; instructions on how to use the toolkit are needed; further links could be to the CQC, CCGs, National Institute for Health and Care Excellence (NICE) and other key organisations, along with CPES qualitative summaries, Cancer Research UK (CRUK), NHS and other key cancer resources that may be useful to service improvement. Many of these had already been identified by the concept-mapping workshop participants (see Chapter 6), and most were implemented in the final prototype.

Goal-directed enquiry: can the toolkit fulfil the working needs of health-care professionals?

All participants considered that the toolkit had the potential to help them achieve goals in service improvement, justify requests for funding/designing new initiatives, gather data to show successes (and boost morale), support appraisals and support the reporting of service data (although an export/report output function is needed). Once these were addressed, all participants agreed that the system would ‘revolutionise’ the way in which patient free-text comments could be used in service improvement. Results and solutions are summarised in Table 14.

Implementation (based on Normalisation Process Theory toolkit questions)

The majority of participants do not currently have an efficient or reliable method for using patient experience free-text comments and would support the implementation of the present toolkit. Equally, the majority believe that their teams/organisation would also support the use of such a system, and would see the purpose and value of it. Better use of patient-derived free-text experience needs to be made, and the toolkit could enable this. Most participants could imagine, and had examples of, service-improvement initiatives as a result of qualitative data from patients and were aware of the methods for assessing the impact of such initiatives. Many participants are overwhelmed by the many local systems to use patient experience and would like to see one system that could process all free text from several different sources.

The implementation data are summarised in Table 15. There was some overlap in concept between the coherence construct and goal-directed questions summarised in Table 14, in which several more explicit actionable points are listed. The main implementation barriers and issues identified were time, absence of run charts, being overwhelmed already with online systems, password protection if used, data misuse problems, the possibility that the system would lead to ‘easy wins’ and neglect more deeply rooted problems, and a lack of funding for the system. Enablers were mentioned less frequently, but included the use of local champions, the timely use of data and the possibility of comparing data year on year. The study team did not explore deeper implementation issues that might be important to ensure uptake and sustainability, as participants were not presented with an implementation plan to discuss; this was the result of governance issues mentioned earlier, which made this premature. As a result, the study team was not able to maximise the potential of the questions here.

Usefulness of the Normalisation Process Theory

The NPT was helpful in determining how the toolkit might be used in practice and what adaptations it might need to enable this, but most participants felt that the NPT tool or e-HIT were not helpful in evaluating the potential workability and integration of interventions developed from the toolkit. A few participants felt that they would ‘add unnecessary extra work’. However, some participants thought that either might be useful for big projects and teams; they had seen NPT before, and were aware of organisations using it, although they would prefer bar chart output for ease of interpretation. One participant thought that the NPT radar output was similar to the 360-degree appraisal tool (see Appendix 7 for more quotations).

Diffusion of innovations

In the consideration of the diffusion of innovations, the study team focused on typing participants, as the use of NPT was intended to cover many of the other constructs in this theory. The intention was to follow-up the current study with an evaluation of implementation into practice once an implementation model had been developed. Most participants assessed themselves as innovator/early-adaptor types of manager (Figure 15). One participant assessed their organisation as having individuals who can innovate-adopt: the majority of participants worked in organisations where there was overarching authority or collective decision to innovate-adopt (Table 16).

FIGURE 15.

Diffusion of innovations types. Tech enthusiasts – the extreme tail of the curve (n = 2); bordering tech enthusiast and visionary categories (n = 4); range all the way from tech enthusiast to pragmatist – depending on what the work is – one of these said that they were also conservative at times (n = 3); bordering visionary–pragmatist (n = 2); visionary (n = 1); and pragmatist (n = 1).

How this contributes to knowledge

Heuristic evaluation and goal-directed design are validated structured methods that are widely used to uncover problems in usability (user interface design), and whether or not systems achieve user goals. They have not been used systematically in a health-care context before to the study team’s knowledge, and the study has produced evidence that the method is relevant and useful in this context. This is important because one of the major faults in system design, and especially in implementation in health-care settings generally and in the NHS in particular, is that although standard requirements analysis informs design, the experience of the user and the relationship of the system to the goals that the user wants to achieve are often, if not always, missing. This design and evaluation process is fast and inexpensive, and it uncovers potentially fatal flaws before they ever reach the workplace. Overall:

• The process of evaluation with users when the designers were not present increased the users’ capacity to highlight errors and, in effect, co-design the system. This was a low-cost way of doing this, and it achieved a lot in just a few hours.

• The discussions, added to the data from the group concept-mapping workshops and associated interviews, provided novel information about big data text processing and NLP, and how that information might be applied in future projects.

Improving the use of patient feedback data

The NIHR call was intended to improve the use of patient feedback data within the NHS. The study team looked at PES free-text comments specifically, with the National CPES as the study’s first case. There is no systematic analysis and use of CPES free-text comments (or indeed, any free-text patient feedback) either within the trusts from which they arise or at a national or regional level. This makes it impossible to respond systematically to emergent themes from the comments or to understand how they relate to variation in care quality across NHS trusts. This study has made significant strides towards the development and validation of an approach that could automate the analysis of summary free-text data and present summaries in a visually engaging and useful way for the 150 NHS trusts that participate in the CPES. It has been shown how various challenges have been met to develop a model of the features a NHS toolkit should have to optimise the use of patient experience free text. This is the first evidence-based attempt that the study team knows of to represent PES free-text comments in ways that are useful to health-care professionals and that the public can understand, and the team members are eager for this to be further developed. As one outcome of the study, proof of concept is thus provided.

Assurances of acceptability and value

With its embedded co-design (using surveys and group concept-mapping workshops) and validation with stakeholders using established preference-based usability and acceptability techniques, the study also provides substantial reassurance that the approach has value for NHS staff. Through these approaches, the study has also shown keen patient interest once governance issues are addressed; until then, the value to patients is indirect, from improvements in the patient experience. The toolkit generated considerable excitement when it was shown to health-care professionals, who know of nothing like it and welcome its implementation into practice. They have declared that they would use it regularly if it has tailored options to adapt the settings to their workflow and themes relevant to their specific work environment and teams. This has been enabled by the study team.

Commissioners have also declared their needs independently, in commissioner events run by Macmillan Cancer Support,13 which are a good fit with what the study team has provided, confirming that the approach meets current demands. These include:

• Awareness of the lack of data for some marginalised groups – the data summary tab (Figure 16) helps to show where data are missing. Summary demographic details of people with relevant comments can be viewed on the main toolkit page in a different format (Figure 17).

• Annual and local comparisons as leverage at the board level.

• Granularity, including access to raw CPES data sets.

• A more systematic, easier and quicker way to analyse CPES free text, having had experience of its value when sorted into themes.

• Online access rather than a report via e-mail.

• Emphasis on priority areas [which the ranking of themes can achieve (Figure 18)].

FIGURE 16.

Part of a screenshot showing demographic details of respondents.

FIGURE 17.

Part of a screenshot of the toolkit highlighting how demographic details of patients making comments on a particular theme may be determined through filters.

FIGURE 18.

Overview of themes. As well as developing six high-priority default themes, a quick overview of all themes is provided. Note that the wording has changed slightly since this draft.

Commissioners in the Macmillan Cancer Support study13 also suggested an online forum for CCGs to compare and share examples of interventions for improvement, as well as peer support networks. The study’s own evidence confirms the value of such additions to the site [and a patient forum for the public-facing version (see Chapter 9)]. Neither of these has currently been included, as this would require some form of moderation or management, or a suitable filtering system to be set up, which needs further funding.

Patients were more likely to use the toolkit on a one-off basis to learn about treatment side effects and experiences of others suffering from the same condition, and to gain information on a site where they were to receive treatment.

Meaningful presentation of the data

The study has also addressed the point in the NIHR HSDR programme brief that there was still uncertainty as to how to present patient experience data in a meaningful and granular way that stimulates local action. An important result and advantage of the study’s approach is that it draws together very large and complex data sets into a thematically driven, simple visual display without loss of the nuances that other manually based methods can have, and it can still allow for exploration of the original text.

The study team is of the belief that:

Information is only useful if it is translated into knowledge and knowledge is only useful if it is used to improve the health of individual patients. One of the main reasons why big data have not fulfilled its[sic] full potential in health care is that small data are not adequately systematized to generate useful knowledge for future patients (research) and that big data are not used to improve health outcomes for individual patients (care).

Reproduced with permission from Sacristán and Dilla. 47 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

The study team has achieved the potential systematisation of patient experience free-text comments in a way that has the potential to drive health-care improvements.

Good toolkit design has the following three requirements:

1. identification of what is being measured

2. connection to current strategic objectives

3. consideration of how the first can achieve the second, and how actionable outcomes can be achieved.

This third requirement was also the ethos of the study. The study team aimed to achieve this by establishing which dashboard or toolkit features were important from empirical evidence, and which were meaningful to individual users within the patient experience setting. This also indirectly satisfied the first and second requirements.

Transferability

The study’s approach is transferable with further work; it has been designed to enable this and has supporting documentation and code. The study’s testing on three very different health-care free-text data sets has elucidated some of the challenges with this and possible solutions.

Toolkit

The dashboard-scoping survey and scoping review showed that the focus of any clinical toolkit designer should be on ensuring minimum user effort and making role-specific tasks simple and quick to achieve. The survey suffered from sampling bias (see Chapter 3, New surveys). The review also had limitations. Although it was rigorous and systematic, given that it was a scoping review, it had a broad research question. Most studies considered the health-care professional user and few evaluated toolkits designed for patients. Many had top-down designs, so it is perhaps no surprise that many of the features elucidated are no different to more generalised good toolkit design features.

Nonetheless, the study team was able to use the findings from the dashboard-scoping survey and scoping review as the foundation for the prototype toolkit development and exploration in the group concept-mapping workshops, DCE and walk-throughs. Overall, these confirmed the scoping review and dashboard-scoping survey data and added further features. For example, the toolkit had to make patient care needs and service failures clear, for leverage in funding bids, care planning and – if a patient were to use it – treatment considerations. In addition, the toolkit had to enable comparisons between sites and years for these purposes. Searches and filters were important, but the suggested form that these should take differed depending on the approach that was used to explore them, as did the colours to use in the charts (RAG or a more neutral scheme) and the use of maps to indicate patterns in the data. Contradicting the dashboard-scoping review, web-page scrolling was deemed to be acceptable by participants across the study; possibly, this was because users did not envisage checking the toolkit on anything except for a desktop PC.

Differences need further exploration (see Transferability to other applications). However, there were more similarities than differences across methods and groups, enabling us to develop a list of recommendations for the evidence-based clinical toolkit design. This can be checked by researchers and users within health care when developing new toolkits, for optimisation.

These recommendations should not replace gestalt principles of design, which are constructed from preattentive processing (i.e. subconscious cognitive processes) and apply regardless of the use to which a toolkit is put. In the preliminary and developmental parts of the study (stages 1 and 2), the study team effectively asked stakeholders for their considered impressions of what they believe they are doing in preattentive processing, and this was explored in the scoping review of clinical digital toolkit design (see Chapter 2). It could be argued that the DCE work attempted to get at preattentive processing more objectively, although the caveats were also discussed (see Chapter 7). The checklist does not aim to replace gestalt principles, but the criteria that were developed should make it easier for designers who use these principles to appreciate health-care stakeholder needs for a digital dashboard or toolkit.

Themes

The group concept mapping was undertaken to enable mixed stakeholder co-design of the theme names and scope (drawing on the term and theme mindmapping survey and rapid review), as well as the toolkit. These goals were achieved, resulting in a draft evidence-based taxonomy of themes suited to driving improvements in the patient experience.

The study team was also able to record the conceptualisations of the different stakeholders through the way in which they sorted statements into themes. However, these data have limitations. The team considered only a small selection of statements, smaller than the number usually used (≥ 100) when the process is undertaken online. A total of 60 statements were purposively selected from the several thousands of comments in the data set, which inevitably included a wider selection of themes. The choices were influenced by findings from the term and theme mindmapping survey and rapid review of themes, and so carried across their limitations. For example, the rapid review of themes was undertaken by one researcher and was limited in scope, the sample sizes for the surveys were small and there was considerable sampling bias across the study.

Nonetheless, it is also a strength of this output that it drew on several data sources across the study, involving multiple stakeholder groups, as it is likely to represent the most significant themes in the health-care experience. The 66 comments considered in the workshops represented 26 out of the 36 themes (72%) that had been determined deductively before the workshops or that became evident inductively through further discussion within the workshops. Participants did not consider there to be major gaps in the themes used. The data had good construct validity. Larger numbers of workshop participants might have enhanced external validity, but the minimum number recommended for robust results was exceeded. 121

Overall, the study team is confident that the final taxonomy of themes had satisfactory external validity, as the stakeholders themselves determined theme names and rankings. Therefore, the taxonomy has meaning across the groups (being defined by them); moreover, themes were ranked by both importance and feasibility for health-care change. The taxonomy can be widely used within health-care studies and practice. The taxonomy was further refined in the last stage of the study and it will probably develop organically as it is used. It would be interesting to revisit it over time to see if priorities and needs change, in other words, to see if the patient experience changes and improves.

Governance and big data

The study has served to open up or contribute to some significant debates, which was not anticipated. One of the most important is the current debate around the open use of big data and tensions with governance. This is explored further in Some remaining challenges.

Data misuse

A second debate is around data misuse. Each decision that is made from the moment a survey is designed, through the data collection phase, to its final display in digital form, influences the next. This means that users need to be aware of the different decisions that are made and the way in which they affect usage of the data. The study team has attempted to address this with a tab on the toolkit that describes the issues. For graphics from this tab, see Figures 19 and 20. This is good practice, as highlighted by the Government Statistical Service. 175 In this regard, it is to be noted that many commercial data-mining companies do not follow this practice, but offer NHS commissioners dashboards that have not been co-designed, to show data sets, such as the Friends and Family Test, in ways that they claim can be used as quality assurance measures in their own right. In fact, these data should always be used to contribute to the wider picture – albeit in a particularly useful, fine-grained and practical way – rather than as hard data. There are sufficient studies to show that each survey has limitations. 176–191 Potential users are often aware of these limitations already. A study by Macmillan Cancer Support13 showed that commissioners valued CPES free-text data, especially as a means of capturing broad information from patients and opening up conversations about patient experience between commissioners and providers. However, they would consider the qualitative data only alongside quantitative data and raised a number of issues with the CPES survey sampling and case mix, as well as the problem that transitory local difficulties, such as staff sickness, cannot be taken into account in national summaries. 13 The study’s participants were of the same mind.

FIGURE 19.

An example of how users are being alerted to consider the limitations as well as the strengths of all data sets, including the study’s own data set. Users can mine down to find more detail on each point. This graphic relates to PESs in general and not the CPES specifically.

FIGURE 20.

A draft of how the limitations of qualitative data may be highlighted. Such highlights may need to be customised to different uses of the system.

Realistic expectations

A third area in which we wish to open up debate is around managing expectations in a thematic analysis of large data sets and the reification of manual analysis. It has been shown how the system missed some theme annotations that seem obvious to a human, but often picked up themes that humans missed. When qualitative researchers undertake inter-rater reliability checks for manual thematic analyses, they often come up with a concordance (similar to the accuracy metric) of around 76%, which is considered to be acceptable. The accuracy figures, which are comparable (indeed slightly higher), are therefore a good achievement, although the study team will continue to work on improving them. Whether human or computer, there will always be some themes that are not accurately categorised, and thus the expectations of computational methods should be realistic.

A public-facing site

Although the study team developed HTML pages for a public-facing site, these are not currently useable as a result of governance limitations. These pages could be developed by the University of Southampton at a later date, with some more funding, once patient consent rules change. When this report was written, data regulations were in flux. Long-awaited results of the government consultation on the new health data security standards and consent/opt-out model for patient data had not been published. These were disseminated a month later in July 2017,193 and confirmed the recommendations of the independent review of data security, consent and opt-outs by the National Data Guardian, Dame Fiona Caldicott. 194 This review recommended:

• new data security standards

• a method of testing compliance with these standards

• a new consent model for data sharing in health and social care.

The NHS still needs to implement these recommendations, which will take time.

In parallel, in late December 2015 the European Council, Commission and Parliament reached agreement on new data protection rules. A final draft of the new European Union General Data Protection Regulation came into force in mid-2018, and the UK’s withdrawal from the EU (Brexit) is unlikely to change this. The British government has announced that it will opt out of Article 43a of the General Data Protection Regulation, which requires there to be a mutual legal assistance treaty in place with the relevant country before a transfer of personal data can be made to that country to comply with a court order.

Given these changes, and the tightening of governance rules following NHS data breaches (unconnected to the project), NHS England restricted use of the CPES 2015 data. This is appropriate, as patients who completed the survey did not consent to new use of their data on public display. Although most study participants found this frustrating and believed that patients would be happy for this new use, a few agreed with us that it should not be done. It is important to record here that patients often felt that this was tantamount to concealment and said that they would not trust the NHS until data were fully shared and they could follow the journey of their own comments in the survey. This is something that has been acknowledged as being important in a Cabinet Office White Paper of 2013. 195

Use of Cancer Patient Experience Survey 2015 data

The same governance issues that precluded the launching of the public-facing site for CPES data also delayed the acquisition of these data, which before the study – and indeed until May 2016 – had been promised to us immediately on release. The study team did not get these data in time for the draft report. Although this did not compromise the study, it reduced the immediate benefits within the NHS. However, this should not be a problem in future years or with different data (it is also noted that these data can be obtained separately from each individual trust by local agreements, at the time of release).

Use of Normalisation Process Theory themes

In the first prototype, radar plots generated from completion of the NPT toolkit were included on the main dashboard page. It was felt that if professional users completed the NPT toolkit, this would enable them to reflect on the feasibility of implementing small-scale interventions as a response to the free-text data summaries. For example, if there were a great many negative comments about teamworking, and in the toolkit the score for interactional workability was poor, this might suggest both that the unit had an inherent problem with teamworking and that it was not practical to address this. Alternatively, it could be decided that the large number of negative comments relating to teamworking could be used as a lever to address deeper issues. However, the NPT toolkit was not designed or developed to drive change, but rather to understand the implementation, embedding and integration of new technology or complex interventions. NPT has, however, been shown to be very useful when interventions are developed,52,53 and it is likely, therefore, that the use of NPT was attempted too early in the potential intervention development process. This was, therefore, a high-risk application that did not work. Nonetheless, NPT has been retained as a link for users of the site to explore, given that the dashboard has now been developed into a broader toolkit. The toolkit explains that once health-care professionals have designed an intervention to improve the patient experience, they might explore its implementation using the NPT toolkit.

Final toolkit features

Some toolkit features were prepared by us, but could not be implemented either because of current governance restrictions or because the study team did not have > 1 year of data.

As an example, the current toolkit shows dials, even though there was consensus that these were not helpful, and currently additional data from other sources cannot be added, nor data reports exported or printed (although screens can be) – see Chapter 5, Back-end decisions and Software for a discussion. However, the study team has developed HTML pages to enable the chart style to be changed, and these other features to be included, once they can be used (Figure 21).

FIGURE 21.

Some alternative features that were developed for the dashboard. Note that as these were developed using WCPES data, health boards have been specified rather than trusts.

Use for the Cancer Patient Experience Survey

Further steps are needed between this proof-of-concept study and real-world implementation into routine management.

At the start of the study, the study team discussed with Insight NHS England the possibility of embedding the approach within the management of the National CPES survey. However, discussions were frozen while governance issues are being attended to by Insight NHS England, and the full use of the system is not possible until CPES consent is changed. Implementation also requires more qualitative research and exploration of logistics, structured by implementation science theory, once an implementation model has been worked up. This should be the first research priority. If national roll-out is not embedded through Insight NHS England, and perhaps in any case as a preliminary to this, the first step is an exploration of local (small-scale) implementation into practice and potential sustainability of the rule-based IR toolkit approach for CPES data. One route forward could be to involve local strategic clinical networks13 in piloting the approach, once the necessary refinements to the work have been undertaken. It would be possible to use the rule-based IR process and toolkit for CPES in this way with only modest further financial input to undertake rule and toolkit refinement to suit the implementation model. The potential effect of the approach on practice itself could be examined, for example; a focused ethnography might be undertaken at a pilot site using actor–network theory. A multimodal evaluation of human–computer interaction would also be informative.

Any further study will need funding. Moreover, as indicated above, small refinements of the dashboard and IR for these studies require a small amount of funding for approximately 6 months of programmer and researcher time. This funding is being sought in 2018, with potential sources already identified. Further maintenance and updates, including the incorporation of an interactive forum as desired by stakeholders, are dependent on more funding or activation of a business model. A handover strategy has been developed, as the chief investigator has left the University of Southampton, and this is in the process of being worked through. See Transferability to other surveys.

Transferability to other surveys

The results have been less promising with non-CPES data than with CPES data, although the proof-of-concept work shows that transferability is possible with minor tweaking of the rules and the toolkit, which has been designed to be possible to do for someone with only basic technical know-how. Thus, a secondary priority is to further explore the use of the system with other health-care data sets to drive health-care improvements. More thorough testing of the technology on different data sets is critical if the approach is to be used extensively – as has already been shown, what works with some types of free text may work less well with other types, for example, with a more diffuse style of writing or with the conflation of clinical and personal experiences, as with the LAPCD data. If this further testing is not undertaken, the approach would still be useful for CPES data and similar free-text data, but it would be disappointing to limit it in this way.

On this basis, and with suitable documentation, the toolkit and the rule-based IR package will be placed on a new online repository targeted at health-care professionals, which the University of Southampton Faculty of Health Sciences is in the process of setting up. This will enable potential users to adopt and amend the toolkit freely, whereas the use of the rule-based IR package will probably need to incorporate a small maintenance fee depending on how it is intended for use; this is partly to ensure the integrity of the system, as rule-based approaches may cease to work well if they become overly complex. The processes in manuals and documentation have been operationalised to be accessed via this repository, whereby the system and the toolkit that have been developed could be transferred to other surveys or modified by other users. The study team also intends to use the system for further work of this nature with other data, subject to funding.

Transferability to other applications

The rule-based IR process as a stand-alone feature has considerable application in the processing of data for research and education use. The approach results in literal themes, the first stage of a thematic analysis, freeing up analyst time to focus on more conceptual theme development. Transferability has been built into the system, but further research needs to be done to explore how this works in the different settings of health care, research and teaching.

Engagement activities

More work needs to be done to determine how to engage members of the public with such approaches and how to enable single trusts or commissioners to use them. This might be facilitated through the involvement of charities and advocacy groups or the setting up of a dedicated group. Such a group could, for example, hold regular national training and discussion events developed locally within Wessex and then spread to other areas.

Methodological research

The co-design processes that were used could be further developed and refined to improve their usefulness, with a further exploration of the degree of concordance between the different approaches.

Information retrieval is a rapidly developing area, but for uses such as ours, more sophisticated processes are not needed. However, a scoping review of what is being undertaken and its underlying theoretical frameworks might be a useful way forward, so that guidelines on good practice can be developed. This might also inform the refinement of the analytical approach for better transferability.

Cost–benefit analysis

A detailed cost–benefit analysis is dependent on further research of the type detailed above.

PubMed

PubMed

Web of Science and Cumulative Index to Nursing and Allied Health Literature

Date range searched: 2007–17.