The design of a survey questionnaire to measure perceptions and behaviour during an influenza pandemic: the Flu TElephone Survey Template (FluTEST)

Article history

The research reported in this issue of the journal was funded by the Health Services and Delivery Research programme as project number 11/46/21. The contractual start date was in August 2012. The report detailing the set up phase and initial outcomes began editorial review in July 2014 and was accepted for publication in October 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The Health Services and Delivery Research editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report. Should the study progress further, the full report will be published in the Health Services and Delivery Research journal.

Declared competing interests of authors

none

Copyright statement

© Queen’s Printer and Controller of HMSO 2014. This work was produced by Rubin et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Identification of key outcome and predictor variables

Outcome variables were selected through discussion with senior representatives from the following groups of end users of the survey data: the pandemic flu team for the English Department of Health; the two official advisory groups for the Department of Health and the UK Health Protection Agency (now part of Public Health England) that deal with the behavioural and communication aspects of pandemic planning; a team from the Health Protection Agency responsible for modelling the spread of a pandemic; and academic colleagues with a particular interest in pandemic flu planning. We also included lay members in this process to include a broader public perspective. A central component of this was identifying the range of behaviours that members of the public might be advised to engage in, or which they might engage in even in the absence of any official recommendation. We also sought to identify what other data would assist these groups in their work in the event of a pandemic.

Predictor variables for the behaviours that were selected were then chosen, based on their theoretical or empirically demonstrated relationship with the behaviour. The main theoretical model we used to guide the selection was Protection Motivation Theory,10 which proposes that people are more likely to engage in health-protective behaviours if they perceive that a health threat is likely to affect them; the consequences of the threat are severe; the protective behaviours are effective; any costs associated with the protective behaviours are small; and they have high ‘self-efficacy’ for the behaviour, i.e. if they are confident in their ability to perform the behaviour should they wish. We also used the results of two systematic reviews of factors associated with behaviour change during a pandemic to inform our selection. 11,12

Testing and refinement of questions

In order to measure each variable we adapted a previously published item or scale where it existed or generated new items where required. Because telephone surveys usually last for no longer than 15 minutes, we kept the number of items used for each scale to a minimum, and we used single items rather than scales where possible. 13 Each item or scale was reviewed by the research team to rectify any obvious problems, such as the use of double-negatives. Where applicable, items were phrased to allow closed responses (‘yes/no,’ ‘true/false’ or ‘strongly agree/agree/neither agree nor disagree/disagree/strongly disagree’) or open-ended responses, which were coded into closed categories by an interviewer.

We tested the 208 items generated in this way for their comprehensibility, face validity and usability in three rounds of interviews (n = 30, n = 20 and n = 28). Participants aged ≥ 18 years and who spoke English were recruited by e-mail from a university database of volunteers drawn from the general population. Demographic characteristics for the participants are given in Table 1. We did not attempt to obtain a demographically representative sample for these interviews. Instead, participants were sought who would allow us to test our questions with people from different sections of society.

Consenting participants were read each item, in turn, and asked to provide their answer to it, and explain the reasoning for their answer. Where required, we also asked them to explain what they believed the question was asking and/or to suggest an alternative wording. The interviews were conducted over the telephone to reflect the way that our items would be used in practice during a pandemic. We reworded items identified as difficult to understand or answer by two or more participants, and retested them in a subsequent round of interviews. These interviews, and the surveys that followed, were approved by King’s College London’s Psychiatry, Nursing and Midwifery Research Ethics Subcommittee (20 July 2012, reference PNM11/12–139).

Reliability of questions

Between 16 and 30 January 2013 (time 1), Ipsos MORI, a UK-based market research organisation, carried out a telephone survey in England, Scotland and Wales, using random-digit dialling of landline telephone numbers. Proportional quota sampling ensured that respondents were demographically representative of the general population, with quotas derived from the most recent Census data and based on age, sex, work status, region and social grade. Respondents were required to be ≥ 16 years and to speak English. Participants were initially asked for consent to take part in a survey on ‘issues currently facing the UK’ and were informed that the survey related to flu only after initial consent was obtained. Surveying continued until at least 1067 people had been interviewed. This allows any future prevalence estimates made from the survey data to be made with a confidence interval of ± 3%. 14 The design was identical to that used for the national surveys conducted by the Department of Health during the 2009–10 pandemic. 6

Out of 17,044 calls made by Ipsos MORI which resulted in contact with someone whose demographic quota had not already been filled, 15,684 (92.0%) were to people who declined to participate, 273 (1.6%) were to people who asked the interviewer to call back later but who subsequently failed to keep their appointment, seven (< 0.1%) began their interview but did not complete it and 1080 (6.3%) completed it in full. This rate is usual for this type of survey and similar to the rates achieved in Great Britain for the national pandemic flu telephone surveys. 6 The demographic characteristics of the sample are given in Table 2.

Interviews typically lasted 15 minutes. Because of the time limitation, we included only a subset of our questions (101 items: full wording and top-line responses are provided in Appendix 1). We excluded questions if they would make sense only in the context of a pandemic (e.g. questions relating to antiviral use, which is not normally recommended in the UK for seasonal flu) or if the basic format of a battery of questions could be checked by asking one or two example questions. As well as answering questions about themselves, parents who had children aged ≤ 17 years living at home with them were also asked a subset of vaccination-related questions about one child, who was selected using the ‘most recent birthday’ method. 15 To assess the test–retest reliability of the items, Ipsos MORI attempted to recontact all of the participants between 28 January and 4 February (time 2). Those who could be reached were asked to complete an identical set of questions. A total of 621 (57.5%) participants completed the time 2 survey. Table 2 shows the demographic characteristics of these participants.

We assessed the internal reliability of a six-item scale assessing the severity of flu that we adapted from the Revised Illness Perceptions Questionnaire (IPQ-R)16 and of a measure of worry about the flu outbreak that we adapted from the six-item state version of the State–Trait Anxiety Inventory (STAI-6)17 by checking for adequate Cronbach’s alphas (between 0.7 and 0.9), item-total correlations and inter-item correlations (between 0.2 and 0.9). 18 Because both scales resulted in skewed data, we dichotomised their scores, based on a median split for the time 1 data.

We assessed the test–retest reliability of data from scales and individual items using kappa coefficients and by assessing the percentage agreement in responses between the two time points. Owing to an administrative error, interviewers randomly selected a child to ask about vaccine-related questions at both times 1 and 2, rather than referring to the same child at both times. For the relevant items, we therefore restricted our analysis of test–retest reliability to those children who were of the same age and gender at each time point (n = 71), on the assumption that these were probably the same children. We treated kappa coefficients of 0.21–0.4 as ‘fair,’ those of 0.41–0.6 as ‘moderate’ and those of 0.61–0.8 as ‘substantial’. 19

Non-response bias

The survey data were also used to test for non-response bias. We tested this using chi-squared tests to compare participants who responded at time 2 and participants who did not respond at time 2, in terms of their scores at time 1.

Identification of key outcome and predictor variables

The key outcome and predictor variables that we selected are summarised in Tables 3 and 4. These tables also show the original source for the items, where applicable. In brief, the priority outcomes we identified were (1) preparatory behaviours (e.g. stocking up on over-the-counter medication or making plans); (2) the presence of flu-like symptoms among respondents; (3) the perceived presence of flu among respondents; (4) performance of respiratory, hand hygiene and avoidance behaviours; (5) intended and actual behaviours when ill, relating to health-care use or avoidance of other people; (6) intended and actual vaccine uptake for self and for any children; and (7) intended and actual antiviral use for self and for children.

Testing and refinement of questions

In general, participants provided interpretations of our questions that matched our own interpretation, although questions that required them to consider a hypothetical circumstance, such as being prescribed antivirals, provoked more hesitation and uncertainty. All three rounds of interviews highlighted minor issues regarding ambiguity, technical jargon and lack of clarity within items. Most were straightforward to resolve. However, three difficulties were noteworthy. First, problems with wording persisted for items assessing social norms, which asked participants to state what ‘people who are important to you’ thought the participant should do. At the end of the third round of interviews, some participants still felt that these were convoluted and difficult to answer. Second, some participants were uncomfortable giving ‘true’ or ‘false’ answers to statements that were intended to assess knowledge or perceptions. This was resolved by changing the response options to ‘probably true,’ ‘not sure’ or ‘probably false.’ Third, it appeared that the five-point ‘strongly agree’ to ‘strongly disagree’ scale might pose challenges in a telephone interview. Participants often asked us to remind them of the options or hesitated when we asked them to clarify whether their agreement or disagreement was ‘strong’ or not. This was resolved by using the same three-point ‘probably true’ to ‘probably false’ scale. A complete list of all questions produced following our pilot testing is given in Appendix 2.

Reliability of questions

Removal of two items from the severity scale adapted from the IPQ-R (‘if I catch flu, it will cause difficulties for people who are important to me’ and ‘if I catch flu, it will have serious financial consequences for me’) brought the Cronbach’s alpha (0.73), inter-item correlations (0.30–0.50) and item-total correlations (0.42–0.57) to acceptable levels. The four-item scale was used in further analyses. The adapted STAI-6 showed acceptable Cronbach’s alpha (0.75) although one item, ‘content,’ showed poor inter-item correlations with the items ‘tense’ (0.15) and ‘worried’ (0.11). Deleting this item to produce a five-point scale that retained acceptable Cronbach’s alpha (0.72), inter-item correlations (0.21–0.58) and item-total correlations (0.45–0.53). The five-point scale was therefore used for all further analyses.

Test–retest reliability was fair for 33 variables, moderate for 36 variables and substantial for 12 variables (see Appendix 3). Two variables, relating to the perceived ability of someone to thoroughly and regularly wash hands if they wanted to (kappa coefficient = 0.16) and believing that flu is spread via coughs and sneezes (0.06), had low kappa coefficients. Both displayed ceiling effects, with > 95% of participants reporting high self-efficacy or believing the statement to be true, and both showed high agreement between the two time points (93% and 97%). The kappa coefficient, as a measure of chance-corrected agreement, is not useful in these circumstances.

Non-response bias

For the large majority of items, there was no difference in terms of responses to questions at time 1 between those who did and those who did not go on to respond at time 2.

Table 5 shows the difference between responders and non-responders in terms of demographic variables. The only significant effects were that responders tended to be older and better educated than non-responders.

Appendix 3 shows the difference between responders and non-responders in terms of non-demographic variables. The pattern of significant differences generally suggested that non-responders may have felt more vulnerable to flu than responders. More specifically, non-responders were more likely to report that they had recently had flu; believe that other people expected them to cough and sneeze into tissues; believe that catching flu would have financial consequence for them; believe that antibiotics are an effective treatment for flu; intend to be vaccinated; be willing to pay to be vaccinated; feel they had insufficient information about the vaccine; feel confused about the vaccine; and have high anxiety concerning flu. Non-responders were also less likely to believe that catching flu would cause difficulties for their friends or loved ones and to take over-the-counter remedies if they caught flu.

Future use of the survey template

The questions listed in Appendix 2 are freely available for anyone to use or adapt as they see fit, providing that appropriate reference is given to this paper. Within England, the questions will be kept under review and will be proposed for inclusion in any future survey work that is required during a flu pandemic or similar public health crisis. Funding and ethical approval is already in place for our team to assist with the analysis of any such surveys. The protocol for this future work, and for the current study, is given in Appendix 4. The questions are not specific to Great Britain and colleagues from other countries may wish to consider whether or not the items in Appendix 2 are applicable to their own needs and contexts. Use of identical items across countries would be of value in building an evidence base systematically and efficiently during the next pandemic. Further work on identifying a minimum data set that could be collected internationally would be worthwhile.

We do not recommend that future users attempt to adopt the items wholesale or uncritically, however. Most obviously, there are too many items for this to be feasible and priorities will need to be made. These are likely to change, depending on the needs of the survey end user and also on the stage of the pandemic. We also plan to conduct factor analysis with some of our data set to explore options for further reducing the number of items used. Future users should also be aware that the questions reflect current recommendations and needs. When these change, the questions will need to be adapted. For example, we used current official definitions to help develop some items, such as what flu-like symptoms to record24 and how to describe appropriate hand-washing. 28

Limitations

Five limitations should be considered regarding this work. First, our use of a database of research volunteers for our pilot interviews may have made our sample unrepresentative. In particular, it is possible that members of the database were familiar with research jargon and procedures owing to their participation in previous studies, making them less inclined to detect or comment on unusual wording in our questions.

Second, our items relating to social norms remained difficult for some participants to understand. The confusion appeared to relate to being asked to anticipate what someone else might think or feel about one’s behaviours. Additional work on these items is required.

Third, although generally acceptable, the test–retest reliability scores for some items suggested room for improvement. Some caution is required in interpreting our statistical measures of test–retest reliability. Participation in the initial survey and knowing that the interviewer would be calling back may have prompted some participants to read about flu-related issues between the two time points, artificially lowering the apparent reliability of their responses. Indeed, participating in the time 1 survey may itself have been sufficient to alter how people thought about flu, with other questions or interviewer prompts changing the way participants perceived certain issues. Nonetheless, our use of single-item measures almost certainly contributed to genuine low reliability in many cases. Although adding more items and producing scales for each variable might be one option to improve reliability, this would be at the expense of reducing the number of variables that could be measured in any given survey. We therefore chose to accept suboptimal reliability for some variables as an acceptable trade-off for increased information per survey.

Fourth, our items are based on self-report. Responses may be affected by recall or social desirability biases. Although this is less of an issue for some variables (e.g. recall for having had a flu vaccination) it may be more problematic for others (e.g. reports of how many times the participant has washed their hands). As ever with survey data, caution should be exercised when interpreting some of the results. Future work should explore how to improve the validity of self-report items in this context, for example by linking behaviours to a concrete activity or point in time to help make them easier to recall (e.g. washing your hands before your last meal).

Fifth, our measure of non-response bias relates to the effects of non-response only among people who had already elected to take part in the survey at time 1. Whether or not that sample is representative of the general adult population of Great Britain is a separate matter. The choice of a random-digit dial proportional quota sample for this study was primarily driven by our desire to replicate the official surveys used within Great Britain during the 2009–10 pandemic. These strategies inevitably give rise to questions concerning their low response rates, although it should be noted that such surveys can still perform well when compared with other, more traditional, epidemiological techniques. 29 Despite this, given the current decline in landline telephone use and the drive to explore alternative survey methods, a telephone survey using quota sampling may not be appropriate during a future pandemic. 30 The decision as to how to deploy the questions described in this paper is an issue that requires consideration in its own right.

Contribution of authors

G James Rubin (Senior Lecturer in the Psychology of Emerging Health Risks) and Susan Michie (Professor of Health Psychology) had the original idea for the study following discussion with policy and public health colleagues in Europe and the UK, and developed the study design with Richard Amlôt (Scientific Programme Leader in Behavioural Science), Nicola Fear (Reader in Epidemiology) and Henry WW Potts (Senior Lecturer, Statistics).

Savita Bakhshi (Post-Doctoral Research Worker, Psychology) conducted the interviews to test question wording and carried out the statistical analyses with James Rubin.

G James Rubin and Savita Bakhshi co-wrote sections for the first draft of the manuscript.

All authors contributed to further drafts and had full access to all of the data.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health.