Neuraminidase inhibitors for influenza: a systematic review and meta- analysis of regulatory and mortality data

Background

Neuraminidase inhibitors are stockpiled and recommended by public health agencies for treating and preventing seasonal and pandemic influenza. They are used clinically worldwide.

Objective

To describe the potential benefits and harms of NIs for influenza in all age groups by reviewing all CSRs of published and unpublished randomised, placebo-controlled trials and regulatory comments.

Search methods

We searched trial registries, electronic databases (to 22 July 2013) and regulatory archives, and corresponded with manufacturers to identify all trials. We also requested CSRs. We focused on the primary data sources of manufacturers but we checked that there were no published RCTs from non-manufacturer sources by running electronic searches in the following databases: the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, MEDLINE (via Ovid), EMBASE, EMBASE.com, PubMed (not MEDLINE), Database of Abstracts of Reviews of Effects (DARE), NHS Economic Evaluation Database (NHS EED) and Health Economic Evaluations Database (HEED).

Selection criteria

Randomised, placebo-controlled trials on adults and children with confirmed or suspected exposure to naturally occurring influenza.

Data collection and analysis

We extracted CSRs and assessed risk of bias using purpose-built instruments. We analysed the effects of zanamivir and oseltamivir on time to first alleviation of symptoms, influenza outcomes, complications, hospitalisations and adverse events in the intention-to-treat (ITT) population. All trials were sponsored by the manufacturers.

Main results

We obtained 107 CSRs from the EMA, GSK and Roche. We accessed comments by the FDA, EMA and Japanese regulator. We included 53 trials in stage 1 (a judgement of appropriate study design) and 46 in stage 2 (formal analysis), including 20 oseltamivir (9623 participants) and 26 zanamivir trials (14,628 participants). Inadequate reporting put most of the zanamivir studies and half of the oseltamivir studies at a high risk of selection bias. There were inadequate measures in place to protect 11 studies of oseltamivir from performance bias due to non-identical presentation of placebo. Attrition bias was high across the oseltamivir studies and there was also evidence of selective reporting for both the zanamivir and oseltamivir studies. The placebo interventions in both sets of trials may have contained active substances.

Authors’ conclusions

Oseltamivir and zanamivir have small, non-specific effects on reducing the time to alleviation of influenza symptoms in adults, but not in asthmatic children. Using either drug as prophylaxis reduces the risk of developing symptomatic influenza. Treatment trials with oseltamivir or zanamivir do not settle the question of whether or not the complications of influenza (such as pneumonia) are reduced, because of a lack of diagnostic definitions. The use of oseltamivir increases the risk of adverse effects, such as nausea, vomiting, psychiatric effects and renal events in adults and vomiting in children. The lower bioavailability may explain the lower toxicity of zanamivir compared with oseltamivir. The balance between benefits and harms should be considered when making decisions about use of both NIs for either the prophylaxis or treatment of influenza. The influenza virus-specific mechanism of action proposed by the manufacturers does not match the observed clinical evidence.

Description of the condition

Influenza is mostly a mild, self-limiting infection of the upper airways with local symptoms, including sniffles, nasal discharge, dry cough and sore throat, and systemic symptoms such as fever, headache, aches and pains, malaise and tiredness.

Occasionally, patients with influenza develop complications such as pneumonia, otitis media and dehydration or encephalopathy with or without liver failure, which may be due to the effects of the influenza virus itself or associated secondary bacterial infections and/or adverse effects of drugs such as antipyretics [including salicylates and other non-steroidal anti-inflammatory drugs (NSAIDs)]. 15

Influenza is not clinically distinguishable from ILI. 16 Epidemic influenza in humans is caused by influenza A and B viruses. Currently, influenza A/H1N1, influenza A/H3N2 and influenza B cause most influenza infections worldwide. 12

Description of the intervention

Neuraminidase inhibitors comprise inhaled zanamivir (Relenza^®, GSK), oral oseltamivir (Tamiflu^®, Gilead Sciences and F. Hoffman-La Roche), parenteral peramivir (Rapivab^®, BioCryst Ltd), inhaled laninamivir (Inavir^®, Daiichi Sankyo Co. Ltd)17 and others still under development. 18 The use of NIs has increased dramatically since the outbreak of A/H1N1 in April 2009, partly because of the rise in amantadine (Symmetrel^®, Endo Pharmaceuticals Inc.)/rimantadine (Flumadine^®, Forest Pharmaceuticals Inc.) resistance and, in the early stages of the outbreak, the lack of a vaccine, which meant that NIs became a widespread public health intervention. WHO had previously encouraged member states to stockpile and gain experience of using NIs. 19–21

How the intervention might work

Although NIs may reduce the ability of the virus to penetrate the mucus in the very early stage of infection,6,22–24 their main mechanism of action is thought to lie in their ability to inhibit influenza viruses from exiting host cells. 23,25 The manufacturers state that oseltamivir does not prevent infection or affect antibody production,26 but it reduces symptom duration probably by reducing viral load, spread and release of cytokines,8,27 diminishing the chance of complications and interrupting person–person viral spread. Oseltamivir phosphate (Tamiflu) is the prodrug of oseltamivir carboxylate, the effective form. Oseltamivir phosphate dissociates in the gastrointestinal tract to form oseltamivir, which is absorbed and metabolised into oseltamivir carboxylate by hepatic carboxylesterase. Oseltamivir may have a central depressant action15 and may also inhibit human sialidase,28 causing abnormal behaviour. Inhaled zanamivir reaches a far lower plasma concentration than its intravenous administration. 29

Any treatment that reduces the complications of influenza (e.g. pneumonia) and the excretion of the virus from infected people might be a useful public health measure to contain an epidemic by limiting the impact and spread of the virus. In addition to symptomatic treatment, prophylactic use for interrupting the spread of disease has informed pandemic planning over the past decade.

Why it is important to do this review

There are three major reasons for conducting this review, in addition to questions of efficacy associated with the clinical use of NIs for influenza:

1. Influenza antivirals are a commonly used and stockpiled drug against past and future pandemics on the basis of international and national recommendations. These recommendations are based on the claimed and assumed ability of the drug to reduce complications and transmission. 2,3 In theory, containing the spread of influenza allows time for an organised response with longer-term interventions (such as vaccines), which take time to produce. 3

2. The risk of reporting bias and publication bias leads to uncertainty about the effects of NIs and the results of previous Cochrane reviews of NIs in adults6,10,30,31 and children. 32

3. Oseltamivir is now on the list of WHO essential drugs. 33,34

Process

Review A159 is an amalgamation of two long-standing Cochrane reviews on the effects of NIs for influenza in healthy adults14 (also published in the BMJ31) and children,35 and it is based on the assessment of trials through their CSRs and other regulatory information; a decision we made after finding substantial reporting bias in the journal publications of the relevant trials.

For the full rationale for this process, see Appendix 1.

Examples of discrepancies and reporting bias

We identified that 60% (3145/5267) of patient data from randomised, placebo-controlled, Phase III treatment trials of oseltamivir have never been published. This includes M76001,36 the biggest treatment trial ever undertaken on oseltamivir (with just over 1400 people of all ages). Exclusion of unpublished data changed our previous findings regarding the ability of oseltamivir to reduce the complications of influenza. 8,31 In some cases, mistakes in the attribution of adverse events were discovered only through matching summary tables with individual participant listings. 37–39

A modified approach

We have modified the routine Cochrane processes to improve our previous methods, which we now consider to be inadequate. To resolve inconsistencies and under-reporting, we changed our approach by no longer including trial data as reported in papers published in biomedical journals. Instead, we treated CSRs as our basic unit of analysis. CSRs are often sent to national drug regulators, such as the FDA and the EMA (formerly the European Agency for the Evaluation of Medicinal Products), which require far more stringent standards for completeness and accuracy of reporting than biomedical journals. Journal articles can be regarded as a very succinct synthesis of a CSR. In addition to seeking CSRs, we decided to read and review regulatory documentation. The FDA in particular (and the EMA to a far lesser extent) makes many of its scientific reviews available on its website. Unlike Cochrane review authors, regulators can have access to the whole data set and their comments can provide useful insight, helping to achieve a better understanding of trial programmes.

Clinical study reports generally remain hidden from public view and are not readily available for wider scientific scrutiny, despite the wealth of information that they contain for those willing and able to spend the time reading them and despite calls to make all relevant trial data public,11,40 as well as the known problems with reporting biases. 41,42

Implications

This modified approach to a Cochrane review aims to provide patients, clinicians and policy-makers with the most transparent and independent information possible about NIs for influenza. In addition, it should contribute to improving a European regulatory and pharmacovigilance legal framework, which commentators consider weak. 40,43 We believe that as NIs have become public health drugs, recommended and stockpiled globally, independent scrutiny of all of the evidence relating to harms and effects on complications is necessary to provide patients, policy-makers and physicians with a complete and unbiased view of their risks and benefits.

Criteria for considering studies for this review

Search methods for identification of studies

To identify trials in the manufacturer-funded clinical trial programmes for NIs, as well as non-manufacturer-funded clinical trials of NIs, we used a variety of methods that were applied to a variety of sources from the literature, manufacturers and regulatory bodies. These methods, as well as our methodology for identifying and obtaining relevant CSRs, are detailed in Appendices 1, 4 and 5.

Data collection and analysis

Collection and inventory of the evidence base was facilitated by the tools that were specifically developed for the review (see Appendix 5). The overall risk of bias is presented graphically in Figure 1 and summarised in Figure 2.

FIGURE 1.

Risk-of-bias graph: review of authors’ judgements about each risk-of-bias item, presented as percentages across all included studies. ‘Other bias’ includes potentially active placebos.

FIGURE 2.

Risk-of-bias summary: review of authors’ judgements about each risk-of-bias item for each included study. ‘Other bias’ includes potentially active placebos. +, low risk of bias; –, high risk of bias.

Selection of studies

For this 2013 review, two authors (PD and TJ) reapplied the inclusion criteria for the oseltamivir CSRs and resolved disagreements by discussion. Two review authors (ES and IO) applied the criteria for the zanamivir CSRs, whereas one review author (CH) arbitrated.

For the procedures followed in the 2012 review, see Appendix 7.

Data extraction and management

The sizeable quantity of available data led us to subdivide the extraction, appraisal and analysis of the data into a two-stage exercise. In stage 1 we assessed the reliability and completeness of the identified trial data. We decided to include in stage 2 of the review (full analysis following standard Cochrane methods) only data that satisfied the following three criteria:

1. Completeness CSRs/unpublished reports include both identifiable CONSORT (Consolidated Standards of Reporting Trials) statement-specified methods to enable replication of the study. Identifiable CONSORT statement-specified results (primary outcomes, tables, appendices) must be available.

2. Internal consistency All parts (e.g. denominators) of the same CSRs/unpublished report are broadly consistent.

3. External consistency Consistency of data as reported in regulatory documents, other versions of the same CSRs/unpublished reports and other references, to be established by cross-checking.

This was a different approach to that used in the previous version of the current review,9 as we had only incomplete information at that time and applied only the second and third criteria.

Use of regulatory information

We used regulatory information to assess the possible correlation between citation frequency of oseltamivir treatment trials in the FDA regulatory documents and trial size.

Post-protocol analyses

After publication of the A159 protocol in December 2010, but before validation of our CONSORT-based extractions in the spring of 2011, we decided to carry out analyses (which we called post-protocol analyses) to test five null hypotheses that we had formulated while reading, summarising and reconstructing the CSRs. The hypotheses originated from our observations of discrepancies and other unexpected observations in the CSRs’ data, and were informed by reading regulatory information. Appendix 8 reports the rationale, methods to formulate and test, and the results of the hypotheses.

The hypotheses reflect the uncertainty prevailing in the evidence base at a time when full CSRs were not available for all studies.

Assessment of risk of bias in included studies

Previous studies comparing regulatory with published or internal company sources of evidence have reported a variety of different biases that affect medical knowledge. 41,42,56,101 We will report in detail elsewhere our comments on using the Cochrane risk-of-bias tool102 to appraise CSRs and for trial programmes, and our efforts to construct an instrument for assessing risk of bias in complete CSRs. A full description of the methods used to quantify biases will be published in another paper.

Measures of treatment effect

To estimate treatment effects we first calculated the risk ratios (RRs) and used the average (mean) control event rate and the pooled RRs reported in the figures to calculate the RDs. For consistency, we adopted this method for both the ‘Summary of findings’ tables and for the RDs reported in the text. For the analysis we chose to report the RRs, as they are more consistent across the studies, and we have reported the heterogeneity for the pooled RR. We reinterpreted the results using the RD, as this result is applicable to clinical decision-making. We calculated MDs for time to first alleviation of symptoms. For time to first alleviation of symptoms we also estimated the treatment effect as the percentage reduction in the average time to first alleviation of symptoms in the placebo group. Most zanamivir CSRs stated treatment effects only in terms of medians in each treatment group, as well as p-values from a hypothesis test comparing the time-to-event distributions. These data are insufficient for conducting a meta-analysis. However, often sufficient time-to-event data were reported to allow us to estimate restricted means and standard deviations. Restricted means are based on the maximum time reported where alleviation occurred. There were some patients for whom alleviation was censored at the maximum follow-up time; therefore, restricted means are underestimates of the true means. However, the proportion of patients who were censored was generally low and similar in both treatment arms, hence this limitation is unlikely to have led to bias. The length of follow-up varied across trials and this has led to high variation in the estimated means and standard deviations across trials.

A post hoc analysis was undertaken after we discovered seven zanamivir trials that provided data on time to first alleviation of symptoms with and without relief medication. Each patient in the studies may or may not have taken relief medication [e.g. paracetamol (acetaminophen)] during the trial. Alleviation of symptoms may have occurred while the patient was taking relief medication and the ‘standard’ comparison was made using this scenario. However, an additional analysis used a stricter definition, for which alleviation of symptoms could be achieved only without the use of relief medication. For example, a patient may have achieved alleviation using relief medication after 5 days but took 7 days to achieve alleviation without the use of relief medication. The comparison we reported is for all patients for whom we used the stricter definition for the zanamivir group (alleviation without relief medication) and the less-strict definition for the placebo group (alleviation with relief medication).

We planned to use the tridimensional dose-relatedness, timing and patient susceptibility methodology to assess the likelihood of harms causality,103 but the quality of the data available did not allow for this.

Unit of analysis issues

Problems with unit of analysis are described in ‘Post-protocol hypotheses’ (see Appendix 8).

Dealing with missing data

We developed a comprehensive strategy for dealing with data that we know are missing at the trial level, that is, unpublished trials (see Search methods for identification of studies, above, and Appendices 1, 4 and 5) and unreliable published records, which are a very concentrated summary of CSRs. For example, in the oseltamivir trial programme, some trials’ CSRs (e.g. WP16263104) consist of 8545 pages. This has a 1000-fold greater length than its published version,105 which consists of seven pages. The purpose of this review is to provide as complete a picture as possible of trial programmes, without reliance on the published literature. Appendix 9 reports an example of the content of a typical Roche CSR.

Assessment of heterogeneity

We used tau-squared (inverse variance method) and the I² statistic to estimate between-study variance as measures of the level of statistical heterogeneity, and the chi-squared test to test for heterogeneity.

Assessment of reporting biases

We carried out assessment of reporting biases (comparing CSR with the relevant publication) only in the first publication of A159. 106 For this version, as we had complete CSRs for the trial programmes of the two drugs, we expected to find all of the relevant information in these documents and adopted a binary assessment (high risk, low risk or unclear bias).

Data synthesis

We used the random-effects approach of DerSimonian and Laird107 based on MDs for analysis of time to first alleviation of symptoms. For all of the other outcomes we used the random-effects approach for binary data of DerSimonian and Laird107 where tau-squared was estimated using the inverse variance method.

Although overall symptom reduction is well documented, our interest was particularly focused on complications and adverse events, as this is where evidence is currently scarce or inconclusive. 31,32,35 Our preliminary examination of CSRs identified that some symptoms and sequelae of influenza (such as ‘pneumonia’) had been classified as either a ‘complication of influenza’ or as an ‘adverse event of the treatment’, or both. We called this somewhat confusing classification ‘compliharms’. We decided to deal with compliharms as follows. We identified complications of particular clinical interest as ‘pneumonia’, bronchitis, otitis media and sinusitis. We tabulated the type of data capture used for each complication (‘secondary illness’) by study, including the following variables: definition of what events are termed complications; which part of the CSR captured data on complications; who reported and captured the data; which diagnostic method was used; whether or not, and where, the diagnostic pathway was (usually a form); and whether or not prescriptions for treatment were captured. We then aimed to stratify our analysis by method of diagnosis with three possible criteria: (1) laboratory-confirmed diagnosis (e.g. based on radiologically or microbiologically confirmed evidence of infection); (2) clinical diagnosis without laboratory confirmation (diagnosed by a doctor/investigator after a clinical examination); and (3) other type of diagnosis, such as self-reported by patient. We also conducted analysis of any complication (such as ‘pneumonia’, bronchitis, otitis media and sinusitis) that was classified as serious or led to study withdrawal.

We tested the effects of oseltamivir in prophylaxis of influenza and ILI. However, the CSRs of prophylaxis trials do not define ILI but report eight different definitions for influenza with laboratory confirmation (see web extra influenza definitions, Appendix 11).

This is a complex and confusing set of definitions, in which, for example, the definition for upper respiratory tract infection with systemic disturbance is the same as one of the definitions for asymptomatic influenza. After discovering the absence of a definition for ILI, and the complex and confusing definitions for laboratory-confirmed influenza, we classified ILI as having two or more symptoms from the following: nasal congestion, headache, chills/sweats, sore throat, cough, fatigue, myalgia and fever. These were the symptoms reported in the efficacy listing of individual patients in module 3 of the prophylaxis trials CSRs.

In two oseltamivir treatment trials8,58 and one prophylaxis study59 there were three treatment arms comparing placebo, standard dose and high dose. For time to first alleviation of symptoms, we restricted comparison to placebo against standard dose (as this is how it was reported in the original report). However, for all other outcomes we combined the standard and high-dose treatment arms. There was little apparent difference in the incidence of outcomes between the standard- and high-dose arms, and combining the arms did not appear to cause heterogeneity. However, in two cases there was some evidence of a dose–response effect. These cases are described more fully below (see Results, Analysis of harms).

The majority of zanamivir trials compared placebo with inhaled zanamivir. However, some trials also included an intranasal zanamivir treatment arm and a combined arm of inhaled and intranasal treatment. The multiple zanamivir arms were generally combined for meta-analysis, as effects appeared similar and did not appear to cause heterogeneity.

Subgroup analysis and investigation of heterogeneity

We investigated the robustness of complications outcomes using subgroup analysis by method of diagnosis. We investigated high estimates of heterogeneity, where possible, using subgroup analysis. For example, we conducted a subgroup analysis of time to first alleviation of symptoms in studies of oseltamivir treatment in children by partitioning studies into those of otherwise healthy children and those of children with chronic illness (asthma). Based on a referee’s comment, we conducted a subgroup analysis on time to first alleviation of symptoms by infection status for zanamivir. We could not do a similar analysis for oseltamivir because we did not have data on the non-influenza-infected patients, and we could not correctly identify the patients with influenza infection as a result of the effect of oseltamivir on antibodies.

In the trial programmes for both oseltamivir and zanamivir there was large variation in treatment effects for pneumonia across the populations studied (i.e. adults and children, as well as treatment and prophylaxis), hence we conducted a metaregression to investigate this heterogeneity. We included all of the studies that reported pneumonia (32 studies in total) and investigated the four binary factors: age group (adults vs. children), drug (oseltamivir vs. zanamivir): indication (treatment vs. prophylaxis) and method of diagnosis. For oseltamivir studies, the method of diagnosis was based on either data collected on non-specific adverse events or secondary/intercurrent illness forms, or data collected on specific ‘diagnosis of secondary illness’ forms that included objective criteria such as X-ray confirmation. For zanamivir, two trials included X-ray confirmation of pneumonia. We conducted the metaregression in Stata/SE, version 13 for Windows (StataCorp LP, College Station, TX, USA) using the metareg command. There were some studies where one treatment group had zero events, therefore we added 0.5 events to all treatment groups for all studies prior to analysis. The dependent variable in the regression was log-relative risk. A further post hoc analysis was undertaken after we discovered that seven trials provided data on time to first alleviation of symptoms with and without relief medication. Each patient in the studies may or may not have taken relief medication during the trial. Alleviation of symptoms may have occurred while the patient was taking relief medication, and the ‘standard’ comparison was made using this scenario. However, an additional analysis used a stricter definition, for which alleviation of symptoms could be achieved only without the use of relief medication. For example, a patient may have achieved alleviation using relief medication after 5 days but took 7 days to achieve alleviation without the use of relief medication. The comparison we reported is for all patients, for which we used the stricter definition for the zanamivir group (alleviation without relief medication) and the less strict definition for the placebo group (alleviation with relief medication).

Sensitivity analysis

Sensitivity analyses applicable to our post-protocol analyses have been covered above (see Methods). We used the fixed-effect method of Mantel and Haenszel as a sensitivity analysis to supplement our primary analyses using the random-effects method of DerSimonian and Laird. 107 Random-effects meta-analysis is known to be overly conservative with sparse data. Hence, we conducted sensitivity analysis using Peto’s method on two occasions for which we had sparse data and borderline statistically significant results (prophylaxis with oseltamivir: renal body system on-treatment and psychiatric body system on-treatment).

Description of studies

We searched trial registries, electronic databases and regulatory archives, and corresponded with manufacturers to identify all trials and requested CSRs. Although this review focuses on the primary data sources of manufacturers, we checked that there were no published RCTs from non-manufacturer sources by running electronic searches in the following databases: CENTRAL 2013, Issue 6, limited to year published 2010–13 (20 search results); MEDLINE (January 2011 to July week 2, 2013) (56 search results) and MEDLINE (via Ovid) from 1 January 2011 to July week 2, 2013 (56 search results); EMBASE (January 2011 to July 2013) (90 search results) and EMBASE.com from 1 January 2011 to July 2013 (90 search results); and PubMed (not MEDLINE) with no date limit (21 records). We searched PubMed to identify publisher-submitted records that will never be indexed in MEDLINE and the most recently added records not yet indexed in MEDLINE. To identify reviews that may possibly have referenced further trials we searched DARE 2013, Issue 2 of 4 April (four search results); NHS EED, Issue 2 of 4 April 2013 (two search results); both resources parts of The Cochrane Library (accessed 22 July 2013); and HEED (searched 22 July 2013) (three search results).

Results of the search

Effects of interventions

Reconstructing trial lists and indexing regulatory comments

Calls for incorporating unpublished data to supplement published trial data in systematic reviews and meta-analyses highlight deficiencies in the current methods for obtaining the most complete understanding of a drug’s effects. 121 Our methodological approach entailed comprehensive searching of unpublished sources, with a particular emphasis on obtaining unpublished and internal reports from drug manufacturers, intended for regulatory submission, and comments from national regulatory bodies. Our decision not to use published evidence as a basis for trial appraisal and data extraction meant that we had to reconcile and synthesise information from multiple unpublished sources. We had to devise a new method of searching, indexing, retrieving and reviewing trial data, and to combine this understanding with regulatory comments to produce an informative review. The first step in this process entailed the need to develop our own reconstruction of the trial programme without initial help from outside sources. The reconstructed list of trials and then programmes took a whole-time equivalent researcher 20 days to compile. Owing to the complexity of the task, we suggest that, in the future, some of the essential phases, such as checking the trial identification number across multiple documents and databases, be conducted in pairs.

One of the comments received on our protocol suggested that discrepancies between published and unpublished versions of the same data set could be due to mistakes in the non-peer reviewed, unedited CSRs (which may be corrected by the time of publication). Our experience, especially with the non-reporting of serious adverse events, points to the opposite being the case. 122 Considering the fact that unintentional errors can occur, we believe that the response should not be a resort to published papers as ‘most accurate’ and best unit of analysis, but rather that CSRs – as by far the most comprehensive record of a trial – remain the key unit of analysis, with the expectation that they be amended and kept as accurate as possible over time, with complete documentation of reasons for any amendments. We believed that the results of our review would be undermined without accessing a more complete body of evidence that we knew to be outside the public domain.

In theory, trial registers would be expected to provide a comprehensive picture of a drug’s trial programme. However, registers were not our primary instruments to reconstruct zanamivir and oseltamivir trial programmes. Both drugs’ programmes were mainly run in the late 1990s, before trial registration became the norm. In addition, registers may suffer from some of the problems that we were trying to address. Bourgeois et al. 123 audited entries for 546 trials of five major classes of drugs on ClinicalTrials.gov, the biggest prospective register of clinical trials, and found evidence of risk of reporting bias and delay in reporting of results. Another review of 152 trials found that the description of 123 (or 81%) of the trials in the sample had been changed in at least one key element in the time between registration and publication. The most frequent changes regarded outcomes. 124 Despite the current limits of registers, both specifically to this review and in the way they are run and updated, we believe that registers are an obvious first choice to start reconstruction of trials programmes. Searching for unpublished material has not yet become standard practice in conducting Cochrane reviews,125 and is currently variably reported. 126

The indexing and review of regulatory files was also a very laborious task. It took a whole-time equivalent researcher 3 days to review the FDA regulator’s comments and gain a basic understanding of the content. Four additional days were needed to read and annotate the FDA zanamivir files and 28 days for reading and annotating the oseltamivir files and building the table of contents-evidence (see Appendix 11). The exercise had to be repeated several times to cross-check content and expand annotations. Construction of the TOC was laborious. A first attempt at electronic mapping the TOC content took 12 and 8 hours, respectively, for the FDA and National Institute for Health and Care Excellence (NICE) regulatory documents. This was carried out using the Adobe Acrobat (Adobe Systems; https://acrobat.adobe.com/us/en/acrobat/how-to/ocr-software-convert-pdf-to-text.htm) optical character recognition search facility, which enabled mapping of citation counts by document and by trial ID. Initially, we used the trial prefix followed by the serial number (‘WV15670’) as ID. This procedure, however, had one major drawback linked to the nature of regulatory documents. As regulatory documents consist of notes, correspondence and reviews, the same trial is cited in a non-standardised way. For example, trial WV156708 is cited as ‘WV15670’ 15 times, as ‘WV_15670’ 12 times and simply as ‘15670’ 19 times. Thorough searches must be conducted using all of the different terms. As this can be very time-consuming, we decided to compare an Acrobat search with a Boolean string strategy containing all of the possible citation formats (e.g. WV15758 OR WV 15758 OR Trial 15758 OR Trial15758 OR Trials 15758 OR Trials15758 OR 15758 OR study 15758 OR study15758) (this is logically equivalent to ‘WV 15758 OR WV 15758’) with a term-by-term search (i.e. separately searching for WV15758 and then for WV 15758 and so on). We reasoned that if the yield were comparable, the Boolean strategy would have been faster. The yield of citations of the two strategies was the same for six of seven ‘tracker’ studies, but use of a Boolean string was considerably faster (an average of 3 vs. 14 hours) than the term-by-term strategy. The NICE submission citations took 2 hours to list in a TOC using a Boolean strategy. We adopted the Boolean search strategy to construct our TOC. Ultimately, it is possible that a search with the trial numerals (‘15670’) may be sufficient to identify the vast majority of citations. To validate this method of searching further, our methods should be repeated on other sets of regulatory documents.

Once we had reconstructed the trial programmes, we submitted the results to GSK and Roche for their input. We received detailed feedback from both but, into 2011, Roche’s list of trials was still incomplete. Despite the laboriousness of the methods, we believe that we ended up with a far more comprehensive and less biased set of evidence than that available through the current system of journal-based publications. This shift in our data synthesis paradigm was made necessary by the numerous and documented discrepancies between regulatory and published evidence and by the sizeable risk of publication bias of the oseltamivir trial programme. The importance of reconstructing the trial programme by first generating a complete trial list was further reinforced upon discovering bias and oversights in regulators’ handling of the trial programme. Regulators focus on a few mutually agreed ‘pivotal’ trials, the data analyses of which are replicated by the FDA but not by the EMA. Both largely ignored trial M76001,36 the largest oseltamivir treatment trial that was conducted prior to initial registration of the drug (and still unpublished). Although the manufacturer may not have offered it as a ‘pivotal’ trial, far smaller and even ongoing studies were included in the evidence base to support Roche’s year 1999 NDA number 021087 (treatment of uncomplicated acute illness due to influenza infections in adults who have been symptomatic for no more than 2 days). The history of the EMA scrutiny is harder to assess as we could find no reports of trial site visits or of data analysis replication, but we identified a pooled analysis of treatment trials, very similar to the Kaiser 2003 analysis,4 which formed the basis for the EMA conclusion that oseltamivir affected complications reported, for example on EMA’s 4 October 2012 Summary of Product Characteristics (www.bmj.com/tamiflu/ema). We requested modules 3–5 (individual listings, demographic data and the statistical analysis report) from the EMA. However, for most oseltamivir trials, the EMA does not have the relevant documents and neither apparently does the National Competent Authorities (e-mail from the EMA, 24 May 2011; e-mail from Dutch regulator Medicines Evaluation Board, 20 July 2011). This means that the modules do not appear to have been either submitted to or requested by regulators, raising questions as to the extent of scrutiny of the clinical trials during the regulatory review of oseltamivir in Europe.

Our new method

Reviewing large quantities of complicated data and linked comments is a very difficult and delicate process. The main problem is not so much the appraisal following standard rules and possible synthesis of data (as when we review published information), but the reconstructions and logical threading of a trial programme generating huge amounts of data needing appraisal. Also the manufacturer’s full regulatory submission, which may have even more information than a full CSR, remains confidential. Most of the essential data required are available in CSRs, together with masses of less important data, but, as we have explained, even in this case there may be important omissions, such as mislaid diary cards (Figures 10 and 11) for follow-up. Manufacturers are under obligation to provide regulators with all of the data requested to enable them to reach a decision: in doing so they produce vast submissions. None of the authors (all experienced systematic reviewers) had any experience of reviewing regulatory information, but we could not find any workable shortcuts. We believe that providing a critical overview of a trial programme rather than minute dissection of each trial is necessary. This can be done by identifying the important topics in the trial programme (such as the effects of the drug on symptoms, infection, complications, transmission and well-being) and following them throughout the programme, putting the evidence together coherently. This includes carrying out a high-level overview of the mode of action of the drug in different populations for different indications. Understanding a drug’s mode of action underpins correct reporting of its strengths and limitations. In addition, a large part of the regulatory submission is made up of chemistry, microbiological, animal model pharmacodynamic and pharmacokinetic studies, which are important for shedding light on the trial programme but which seldom feature in systematic reviews. We are unsure whether or not this information could be considered as core information, but an exhaustive review of a trial programme should include reviews dedicated to such topics.

FIGURE 10.

Example diary card from CRF for zanamivir trial (1).

FIGURE 11.

Example diary card from CRF for zanamivir trial (2).

These methods revealed possible problems in trial conduct and validity, including the lack of comparability between arms induced by subset analysis and by the randomisation analysis fork, high positivity rate of influenza, high gastrointestinal events in the placebo arms, possibly active placebo content and possible procedural breaches in several trials. The ideal option is to carry out analyses on the basis of the ITT population, in which units of randomisation and analysis are the same and many of the potential problems listed are either not present or minimised. We are continuing to develop further methods for using such data.

Regulatory comments

Reviewing regulatory comments was essential to expand our understanding of the trial programme. We expected that detailed reading of regulatory material would allow us to understand discrepancies between US and European regulators’ conclusions regarding the effects of oseltamivir, particularly (but not limited to) their putative effect on complications. 8 We were interested in what led the FDA to have far more cautious and conservative statements – as witnessed in the Tamiflu product label and FDA letters – in comparison with European regulators. Our access to huge amounts of FDA regulatory data allowed for many insights but gave us little visibility of manufacturers’ responses.

Some of the statements (such as NIs reduce bacterial complications) made by the manufacturer in the CSRs, and, subsequently, in contemporaneous publications and advertisements, appeared unsupported by the evidence provided at the time. The FDA drug regulatory reviewers’ comments, although laborious to summarise and contextualise (because of the non-availability of the whole pharmaceutical submission), were confirmed by our reading of the CSRs. However, we were unable to find a statement explaining how the FDA reviewed each NDA. FDA reviewing methods appeared to be a mixture of spot checks, re-run of statistical analyses and on-site inspections. A FDA methods volume or standard operational procedure may be among the documents not available from the web but accessible through a FOI request. Neither the FDA nor the EMA have inventories of held documents, making it very difficult to know what to ask for under FOI rules. We concentrated on downloading or asking for specific CSRs and related documents or reviewers’ comments on a particular NDA. The quantity of information held by regulators is likely to be large. For example, NDA 21–246, regarding the use of Tamiflu in the treatment of influenza in children, submitted to the FDA on 15 June 2000, consisted of 137 volumes of study documents and possibly several electronic files. Although we do not know exactly how long a volume was, we have seen references to up to hundreds of pages in each volume.

Requesting specific documents and packages of information is especially important to allow a more efficient and timely reviewing process when confronted with a large volume of evidence, most of which could be of peripheral value. A request for a specific document is likely to be dealt with far more efficiently than a generic request for ‘all documentation relating to oseltamivir’. This is one of the reasons why developing a TOC for any drug or family of drugs (no matter how time-consuming) is an absolute prerequisite for any serious attempt at reviewing regulatory evidence. This introduces another very difficult problem: how to handle huge quantities of structured information and the ethics of drawing conclusions from what is still a fragmentary (albeit sizeable) evidence base.

Overall, the FDA assessment of the performance of oseltamivir was ‘modest’. This adjective appears six times in a 50-page review document. 109 For example, in the Division Director Memorandum dated 25 October 1999, under the heading ‘Public health role of antiviral treatment’ the FDA109 states: ‘The clinical relevance of the modest treatment benefit is a highly subjective question’ (p. 3). The FDA refused to accept claims of oseltamivir’s effects on influenza complications as ‘false or misleading’ statements in promotional materials. 122 A FDA warning letter seems to imply, for example, that oseltamivir’s mode of action is ‘proposed’ or ‘possibly’ (that proposed by the manufacturers) (i.e. not certain). 127 However, FDA reviewers appear to have missed important problems in Roche’s clinical trials (such as the imbalance in the numbers of individuals classified as influenza infected in oseltamivir treatment trials). Importantly, there appears to have been no investigation into the coherence of the evidence with the proposed mode of action of the drug.

Summary of main results

For the first time, a Cochrane review is based on all relevant full CSRs of a class of drugs integrated by regulatory comments. Also for the first time, all CSRs of trials in a manufacturer’s programme (regardless of their relevance to the review) are available to readers without any restriction (apart from minimal redactions to protect anonymity further). The role of Roche and GSK in making this possible should be recognised, as well as that of the BMJ, which kept the issue in the public eye until it was resolved.

The evidence we have presented and synthesised shows that both of the NIs in this review have symptom-relieving effects, especially for self-reported outcomes. They appear to have symptom-relieving properties that make people with ILI and self-reported, investigator-mediated, unverified pneumonia feel better by shortening symptom duration and reducing the frequency of symptoms, such as cough. For oseltamivir, this effect perhaps extends to cardiac symptoms, despite the short duration of treatment (5 days). We are unsure what to make of this finding but we think it deserves further investigation.

The issue which triggered our change of evidence-seeking methods is partly resolved: no definitions of secondary illnesses were given anywhere in the CSRs [e.g. ‘pneumonia’ was defined as ‘pneumonia’ in the CRFs (see Table 1) and diagnostic criteria were not given]; clinical diagnosis in the absence of criteria and without X-ray has only a moderate chance of being correct.

We could not decide the level of diagnostic ascertainment of diagnosis of pneumonia and other complications, as it is unclear from the CSRs. Definitions of pneumonia were not given and the algorithm for classification of an event as pneumonia was not supplied. In oseltamivir trials, the CRF trigger for recording of adverse events and secondary illness was a question to the participant posed by the investigator. A typical phrasing is as follows: ‘Secondary illness reminder: Has the patient reported any sinusitis, otitis, bronchitis, other chest infection or pneumonia since baseline?’ This was followed by a yes/no box to be ticked and an additional form was to be filled out by the investigator for collecting details on the secondary illness. A record of medications outside trial allocation was elicited in addition to the participant’s diary card. The original and Medical Dictionary for Regulatory Activities terms suggest diagnoses for all secondary illnesses and adverse events but there is no indication as to how the original and preferred terms were assigned. We therefore considered these outcomes to be ‘self-reported, investigator-mediated, unverified’ outcomes. For a subset of trials, secondary/intercurrent illness and adverse event data were collected on a single, one-page form. In our meta-analyses, we called this subanalysis ‘Trials which collected data on non-specific adverse events or secondary/intercurrent illness form’. For a different subset of trials, CRFs contained space to record diagnostic tests, such as chest X-rays, tympanometry and sinus X-rays for all secondary illness but there was no reporting of such variables in the CSRs (Figures 12–15). In our meta-analyses, we called this subanalysis ‘Trials which collected data on specific “Diagnosis of Secondary Illness” form’. None of the complications was defined as primary outcomes in any trial, which may explain the poverty of data definition.

FIGURE 12.

Sample ‘Adverse event or intercurrent illness’ form (oseltamivir study M7600136).

FIGURE 13.

Sample ‘Secondary illness’ form (oseltamivir study WV156708).

FIGURE 14.

Sample ‘Diagnosis of secondary illness’ form, page 1/2 (oseltamivir study WV1597867).

FIGURE 15.

Sample ‘Diagnosis of secondary illness’ form, page 2/2 (oseltamivir study WV1597867).

In a metaregression of all of the 32 included studies that reported on ‘pneumonia’, we found evidence that treatment effects for pneumonia are statistically different depending on the method of diagnosis. Unclear objective diagnosis was associated with an apparent 46% reduction in pneumonia as a result of treatment with NIs, whereas the use of objective criteria in the data collection showed no evidence of effect, with a RR of 1.0. Age group (adults vs. children), drug (oseltamivir vs. zanamivir) and indication (treatment vs. prophylaxis) showed no evidence of association with treatment effect.

Meaningful conclusions on the effect of either NI on complications of influenza are difficult to draw based on the trial evidence. In part, this was due to the lack of standardised definitions. In addition, meta-analyses of these outcomes that lacked definitions were based on few events and therefore not robust. Caution is therefore urged in interpreting the meta-analysis result, which suggests that 100 patients (95% CI 67 to 451 patients) need to be treated with oseltamivir for one less self-reported, investigator-mediated, unverified pneumonia. The same applies to the zanamivir treatment result, which suggests a reduced risk of self-reported, investigator-mediated, unverified bronchitis in adults (NNTB 56, 95% CI 36 to 155). The evidence suggested that oseltamivir had a similar effect, although the result was non-significant.

As stated above, there is no evidence that definitions of complications in paediatric, elderly or adult trials were ever prepared and incorporated in the trials’ design. Therefore, the reporting of cases of ‘otitis media’, ‘pneumonia’, ‘sinusitis’ or ‘bronchitis’ are of unclear significance and importance, making it impossible to attribute a cause and draw conclusions. 128 This is probably why the FDA-approved oseltamivir package insert, since 17 November 2000, has consistently stated: ‘serious bacterial infections may begin with influenza-like symptoms or may coexist with or occur as complications during the course of influenza. TAMIFLU has not been shown to prevent such complications’. The original product label did not contain such a statement but, on 14 April 2000, after oseltamivir was approved for sale in the USA, the FDA sent Roche an untitled letter about ‘Misleading Efficacy Claims’, which the FDA had noted in Roche’s promotional materials (p. 3129). One of the statements that Roche made was: ‘Tamiflu reduces incidence of secondary complications (i.e. bacterial infections) by 45%’. The FDA commented: ‘Further, you have claimed reductions in severity and incidence of secondary infections with Tamiflu that are misleading because they are not supported by substantial evidence’ (p. 3129). We do not know how Roche responded to the FDA but in subsequently available Roche promotional material information, Roche’s statements were consistent with the FDA’s demands. 8

There is uncertainty in the ‘complications’ and ‘secondary illnesses’ outcome definition, therefore we carried out an analysis on the data from adult treatment trials on those complications classified as serious or those which led to study withdrawal. For oseltamivir, there was no evidence that treatment affected such complications (RD 0.07%, 95% CI –0.78% to 0.44%). This outcome could not be assessed in oseltamivir prophylaxis due to an insufficient number of events. For zanamivir, there was no significant evidence of a treatment effect on such complications (RD –0.04%, 95% CI –0.64% to 0.24%). This outcome could not be assessed in children because of an insufficient number of events.

Contrary to the FDA, the EMA’s oseltamivir Summary of Product Characteristics states that oseltamivir significantly reduces the incidence of ‘specified lower respiratory tract complications (mainly bronchitis) treated with antibiotics’ in individuals aged ≥ 13 years. This claim is based on ‘a pooled analysis of all influenza-positive adults and adolescents (N = 2413) enrolled into treatment studies’, of which 1063 were in the placebo group and 1350 were in the oseltamivir-treated population. 130 This statement appears in the EMA files as early as 2001. 131 These exact denominators appear in the Kaiser et al. 4 2003 meta-analysis.

The design of the trials – as defined in the protocol with amendments, SAP and CRFs – does not allow any further inferences. The effect on outcomes that were originally considered of secondary or tertiary importance (such as bronchitis and pneumonia) would have been clarified with better clinical definitions and investigations, as were some of the serious adverse events. These benefited from a paragraph-length narrative, which reported most of the salient features of the event.

Our previous decision to analyse the effects of oseltamivir and zanamivir on the ITT population has been confirmed for oseltamivir, with the demonstration of the effect on antibody responses in participants in treatment trials, although no such effect is discernible for zanamivir. This effect leads to the introduction of selection bias, with a significantly reduced probability of being diagnosed with influenza and an imbalance in the two arms if the ITTI population is analysed. The effect of oseltamivir on antibodies appears to be carried over to children with ILI. Its finding contradicts statements made by the manufacturer.

The apparent incomparability between arms of the influenza-infected subpopulations in the oseltamivir trials raises the question of how an appropriate analysis should be conducted. If influenza-infected groups are comparable (as appears to be the case in zanamivir treatment trials) then an appropriate analysis strategy (based on Senn132) would be to first determine the effect of treatment in the ITT population. If there is evidence of a treatment effect then treatment by infected status interaction could be tested. If there was evidence of an interaction then estimates of treatment effect could be derived separately for the influenza-infected and non-influenza-infected subpopulations. However, this analysis should be conducted on the ITT population using a single appropriate statistical model, obviating the need to conduct separate analysis on the influenza-infected subpopulation. Roche used geometric mean titres indicating antibody responses in the ITTI population to support their statement that oseltamivir does not affect antibody responses (e.g. in Table 15 and linked text of module 1 of trial WV1579965). However, the use of such measures can be misleading. What are required for such an analysis are data on how many ITT population participants responded by arm, at what level of antibody response and how many were tested. Such data could not be identified with certainty. A further effect of choosing a subpopulation analysis (ITTI in treatment trials and ITTIINAB (ITT influenza-infected index cases who had negative virology at baseline) in prophylaxis trials) as the primary analysis is the restriction of the generalisability of results. This is especially so in the case of design flaws (e.g. in the case of the PEP trial WV15799,65 in which all index cases were not treated and around 55% of participants were dropped from the ITTIINAB analysis). In this cluster trial design, households should be included as random effects in the analysis to take account of within-household correlations.

A significant but slight reduction of the proportion with serum antibody (mostly haemagglutination inhibition antibody) titre rise by fourfold or more among those who were tested was shown in this review. This was consistent with the evidence from animal tests using a subclinical dose of oseltamivir in influenza A/H1N1-infected mice. 133 Takahashi et al. 133 reported a non-significant slight reduction of haemagglutinin-specific secretory immunoglobulin G antibody in the serum and spleen, whereas they reported about an 80% significant reduction of haemagglutinin-specific secretory immunoglobulin A antibody in the nasal wash and bronchoalveolar fluids on day 12. From this evidence, they warned that the risk of reinfection may increase in patients showing a low mucosal immunoglobulin A antibody response following oseltamivir administration. These experiments were done because they had the unexpected finding that patients with paediatric influenza who were treated orally with oseltamivir for 5 days had significantly low levels (about 60% reduction on day five) of anti-influenza secretory-immunoglobulin A nasopharyngeal fluids compared with levels in patients who were not treated with oseltamivir. 134 Their findings are consistent with our findings that serum haemagglutinin inhibition antibody response was decreased by oseltamivir administration, although secretory immunoglobulin G antibody could not be analysed in our study because the data were not reported in the CSRs. 133,134 These findings are also consistent with the evidence on the mode of action of oseltamivir from animal models8,58,135 and from viral challenge, randomised, placebo-controlled studies in humans. 27

Pro-inflammatory cytokines, including interleukin 6, tumour necrosis factor alpha and interferon gamma, were completely suppressed by oseltamivir administered 28 hours after the experimental inoculation of influenza virus, whereas the reduction of viral titre in nasal lavages was partial. 27

There is decisive evidence that administration of oseltamivir in animals challenged by respiratory syncytial virus, which lacks a neuraminidase gene, showed a symptom-relieving effect (decreased weight loss) and inhibition of viral clearance. 136 These effects were accompanied by a decreased cluster of differentiation +8 T-cell surface sialoglycosphingolipid GM1 level, which is regulated by the endogenous sialidase/neuraminidase in response to viral challenge along with suppression of cytokine expression. 136 They are consistent with those findings from the pharmaceutical company and their investigators. The findings of the study by Moore et al. 136 suggest a risk of infection and exacerbation of infection by pathogens other than influenza virus despite the apparent reduction of symptoms from infection.

Sufficient plasma concentration of oseltamivir carboxylate from orally administered oseltamivir phosphate may act directly on the host endogenous neuraminidase to reduce (or suppress) the immune response even at the dose of 20 mg twice a day for 5 days. However, the bioavailability of inhaled zanamivir seems to be very broad: about 10–70%, as estimated by the area-under-the-curve data from the inhalation and intravenous study from the Japanese Summary Basis for Approval. The difference in peak concentration (C_max) was much larger (sixfold to 37-fold). This means that inhaled zanamivir could reach a high enough concentration to reduce the immune response, if it is administered at a high dose or for a long period, or if the patient is very susceptible. In fact, a double-blind, placebo-controlled trial137 using healthy volunteers to investigate the effect of zanamivir treatment (20 mg/day for 14 days) on the humoral immune response to influenza vaccine showed that the zanamivir group responded with significantly lower antibody titres to the H1N1. Pro-inflammatory cytokines, including interleukin 6, tumour necrosis factor alpha, interferon gamma and other chemokines, were almost completely suppressed in the viral challenge RCT using a very high dose (600 mg) of intravenous zanamivir before inoculation of the influenza virus in human adults. 138

These findings all suggest that the low immune response, with a low level of pro-inflammatory cytokines, induced by the action of oseltamivir carboxylate, may reduce the symptoms of influenza irrespective of an inhibition of influenza virus replication, which is widely believed to be the main mode of action of NIs.

In addition, the potential hypothermic or antipyretic effect of oseltamivir (but not zanamivir) as a central nervous system depressant may also contribute to the apparent reduction of host symptoms. 139,140

Zanamivir had no effect on pneumonia symptoms in treatment trials, even when the diagnosis was supported by a chest radiograph, nor did it affect antibody responses, but it did affect bronchitis. We think that this shows a symptom-relieving effect of both drugs, which also applies to more severe, if undefined, syndromes. Both drugs relieve ILI symptoms by around 0.6–0.7% day’s reduction although this is first relief and not necessarily complete relief. In the case of oseltamivir, the mix-up with the follow-up cards does not allow us to draw any conclusions on a possible length of the duration of symptom relief. Also of note is the fact that this important information came to light from the FDA reports and not from the CSRs of the relevant trials. 8,58 This points to the incomplete nature of reporting in the CSRs and the important role of Summary Basis of Approval (SBA) regulatory information.

In a subgroup analysis we found no evidence of a difference in treatment effect for zanamivir on time to first alleviation of symptoms in adults in the influenza- and non-influenza-infected subgroups. Both subgroups showed strong evidence of treatment effect of 0.5–0.7% day’s reduction in time to first alleviation of symptoms. This strongly supports our hypothesis that these drugs do not have an influenza-specific effect.

Oseltamivir relieves symptoms in otherwise healthy children, but no effect was noted with zanamivir, which may be because of the limited power of the two eligible trials, with just over 700 children in total. However, oseltamivir does not have any effect on asthmatic children with ILI, a population that should benefit most from its use. One explanation for this finding is in the nature of the young asthmatic population, which is well cared for and used to regular powerful medications and close follow-up. The incremental benefit of oseltamivir is thus likely to be undetectable in such a population. An alternative explanation could be the higher susceptibility of the immune system to suppression by oseltamivir carboxylate in asthmatic children compared with those in the placebo group. The finding that oseltamivir administered to asthmatic children reduces symptoms faster than in placebo recipients at the beginning of the study, but during the off-treatment period more recovered later than those administered placebo, gives some support to this explanation.

There is no evidence of an effect of oseltamivir on hospitalisations. Hospitalisations are an important but poorly defined outcome in the oseltamivir protocols, inconsistently reported in the CSRs and overlooked in the zanamivir protocols and reports.

The oseltamivir trials did not detect any influenza-related deaths, reflecting the relatively benign nature of influenza in the study populations. The zanamivir trials detected eight deaths, of which only two were likely to be caused by influenza and both occurred in the intervention arms. All of the trials were likely to be underpowered to detect differential effects on mortality, but the absence of deaths in placebo recipients again underlines the benign nature of influenza. In fact, mortality in Japan during the 2009A/H1N1 influenza outbreak was 198 among about 20 million patients with influenza (1 in 100,000 infected). Early deterioration leading to death was observed more frequently in oseltamivir recipients than in zanamivir recipients or no antiviral recipients. 141

Overall, the two drugs have similar benefits but markedly different toxicity profiles. On average, for every 28 (95% CI 14 to 112) adults treated with oseltamivir there will be one more report of nausea and for every 22 (95% CI 14 to 42) adults and 19 (95% CI 10 to 57) children there will be one more report of vomiting. Oseltamivir seems to have an apparent protective effect on diarrhoea, contrary to the other evidence of gastrointestinal disturbance. This finding might be as an effect of a placebo containing dehydrocholic acid or it might be one of the results of the ILI symptom-relieving effects (similar to relief of tachycardia and palpitation). The other apparent gastrointestinal events, such as nausea and vomiting, may be the result of central nervous disorders indicated by ‘only day 1 increase of vomiting’ in treatment trials in children.

For every 62 (95% CI 41 to 411) adults exposed to zanamivir there will be one fewer case of nausea and vomiting, but no such effect was visible in children, probably because of a lack of power. Zanamivir does not appear to affect the frequency of bowel movements.

In the prophylaxis data set, ‘influenza without laboratory confirmation’ (i.e. ILI) was only partially reported in the oseltamivir CSRs and not reported in the zanamivir CSRs, except for one report,86 in which no significant reduction was observed (zanamivir 9% vs. placebo 10%). As a consequence we are unable to report on that outcome. The size of the reduction in influenza symptoms in oseltamivir prophylactic trials is inferior in magnitude to that seen in hand washing to prevent severe acute respiratory syndrome, based on seven case–control studies [odds ratio (OR) 0.77, 95% CI 0.70 to 0.84; I² statistic = 68%; RD –0.12, –0.16 to 0.08; I² statistic = 26%], the NNTB being approximately ‘50’ for prophylaxis with oseltamivir and ‘8’ with handwashing. 142

There is a significant reduction in the proportion of patients with symptomatic influenza with both NIs. However, these findings do not reflect the true efficacy for prevention of influenza because they conceal the positivity of laboratory testing (measured through tests of viral shedding and fourfold antibody titre rise).

We found an apparent prophylactic effect of zanamivir on pneumonia (which was not defined in CRFs) when it was used for 14–28 days. However, we found no evidence of significant effects on other complications and no evidence of an effect of oseltamivir on complications or hospitalisations.

Oseltamivir induced nausea in people who were undergoing prophylaxis but there was insufficient evidence to show an association with vomiting.

On-treatment renal adverse events were three times more common in the oseltamivir arms than in the placebo arms, with 150 treated patients leading to one additional event. The two participants who died in the oseltamivir arms both experienced acute renal failure while on-treatment, although only one of those events was listed as an adverse event. The unlisted event was in a 91-year-old female who was ‘withdrawn from the study on Study Day 15 because her estimated creatinine clearance was less than 30 ml/minute (WV15708, p. 44). 61 The screening laboratory examinations, that were carried out 10 days before the start of study treatment, were normal’. Hyperglycaemic adverse events (aggravated diabetes mellitus or hyperglycaemia) were also more common in the oseltamivir arms, with eight events in total (one in WV15673/WV15697,59 two in WV1570861 and five in WV1582568) compared with none in the corresponding placebo arms. These data are only presented descriptively, as they are too few (< 10) to meta-analyse formally, as prespecified in our analysis plan.

Finally, oseltamivir caused headaches and psychiatric harms in adult prophylaxis trials. Headaches are one of the most prominent harms of oseltamivir. There is evidence of a dose–response effect in prophylaxis trials (p = 0.013),59 in which headaches were observed in 202 of 519, 225 of 520 and 242 of 520 participants in the placebo, oseltamivir 75 mg once daily and twice daily arms, respectively.

In the psychiatric category, several rare and severe single events (nervousness, aggression, hallucinations, psychosis, suicide ideation and paranoia) were reported significantly more frequently in the intervention arm. Added to other more frequently reported but not significantly different events (such as depression and confusion), this gave a large effect and a relatively small NNTH of 94 (95% CI 36 to 1538). The importance of such a finding lies in the distribution of oseltamivir to large numbers of asymptomatic individuals following pandemic plans. There were no prophylaxis trials in children that met our inclusion criteria, therefore we cannot report on prophylaxis harms in this important population.

The question of why oseltamivir treatment trials failed to identify a clear association between oseltamivir and psychiatric harms, although a weak dose-dependent association was observed, is a moot point. It is possible that ILI and influenza symptoms masked the harms in those who were already symptomatic and therefore recruited in the treatment trials (and influenza-type symptoms were excluded as adverse events to be reported). The reporting issue of compliharms may have helped to mask such events. Alternatively, it could be that these events are rare in the populations studied and that there was insufficient power to detect an association. The CI was wide (0.43 to 2.03) and does not rule out a doubling in risk as a result of treatment, as was found in the prophylaxis trials. It is also possible that the risk of psychiatric harm increases with increasing dose (as the data from two trials8,58 suggest) and increasing duration of treatment (as the prophylaxis trials suggest).

Toovey et al. 143 assessed the issue and failed to find an association between neurological and psychiatric adverse events and oseltamivir exposure. The outcomes studied were not based on the a priori definition of psychiatric adverse events as defined in the CSRs. The definition was constructed post hoc, based on a selected group of adverse events taken from the psychiatric, neurological and injury body systems in the reports. The issues are described fully in another report144 and Toovey’s response. 145 Toovey et al. 143 reviewed only retrospective observational studies and did not review three prospective cohort studies conducted in Japan.

A meta-analysis of three prospective cohort studies of neuropsychiatric adverse events (NPAEs) in Japanese children show a pooled OR for abnormal behaviours due to oseltamivir exposure of 1.55 (95% CI 1.21 to 1.98; p = 0.0005) without significant heterogeneity. 146 In one prospective study147 of several thousand children with influenza, carried out to test the hypothesis of a causal relation between oseltamivir and neuropsychiatric events, abnormal behaviour was observed more frequently in oseltamivir recipient children than in control subjects (RR 1.57, 95% CI 1.34 to 1.83). Abnormal behaviour was observed in 3.4 per 100 person-days (or 13.8%) in the oseltamivir group compared with 2.2 per 100 person-days (or 8.8%) in the control group. Reanalysis of this study population, focusing on delirium and unconsciousness, also showed a significant association between oseltamivir and neuropsychiatric events, especially in the very early phase of the illness within a day of commencement of fever. 148 These indicate that prospective and intentional collection with this scale of participants may be necessary in treatment RCTs.

Animal toxicity study results firmly support the effect of oseltamivir on the central nervous system. One of these is the hypothermic effect of oseltamivir (but not zanamivir) administered orally, intraperitoneally139,140 and intracerebroventricularly. 140 The other is that intraduodenal or intravenous administration of oseltamivir to mature rats induced respiratory arrest shortly followed by cardiac arrest. These studies clearly show central depressant effects of oseltamivir. 149 Moreover, in the post-marketing toxicology phase studies by Roche, many symptoms that the manufacturer considered ‘item-related’ were observed: alterations in respiration including decreased respiratory rate/gasping and altered mucous membrane/skin colour (pale) prior to death. Although the manufacturer denied the causality,150 symptoms at 2 hours after administration that showed dose-related increase were lack of olfactory orientation, lack of cliff aversion and low/very low arousal. Twenty-four of 52 pups that did not exhibit cliff aversion were later found dead. Fourteen of 17 animals with low or very low arousal died thereafter. These findings are consistent with the clinically observed psychiatric symptoms in the RCTs and post-marketing spontaneous reports.

Zanamivir was well tolerated. However, a potentially active placebo may have masked the occurrence of bronchospasm in zanamivir trials.

Treatment trials were mostly under-recruited and often their results pooled post hoc in two, or even three, trials, and yet they showed very high influenza positivity rates. One possible explanation for this lies in the intensive surveillance carried out in the predefined trial centre areas and the restricted time span of recruitment during high likelihood of positivity periods. This may be why many centres with low levels of recruitment are listed in the CSRs; this limits the generalisability of the results to everyday life.

In a primary or secondary prophylaxis indication the postulated central effect of oseltamivir is confined to suppressing symptoms, as infection was not prevented even when oseltamivir was administered prior to the inoculation of influenza virus both in animals135 and in humans,27 and the prophylaxis trials. However, the central problem remains the incompatibility of the two contrasting claims of its activity against antibody production. If, as reported in many documents, oseltamivir does not interfere with antibody production (e.g. see FDA151 and Roche Investigators’ Guide152), how is it possible that oseltamivir prevents cases of influenza when part of the definition of prevented cases in oseltamivir trials was based on the absence of antibody response?

The apparent ability of oseltamivir to interfere with antibody response calls into question the mode of action of the drug and puts in doubt the proposed effects of oseltamivir. One possibility in treatment trials is that oseltamivir administration, by interfering with antibody production, has the effect of selecting the strongest antibody responders in the ITTI subpopulation. These individuals are classified as influenza cases and are included in the oseltamivir arm of the ITTI population. This selected subpopulation probably represents the healthiest or those least likely to experience complications. An alternative consequence could be that interference with antibody production in the oseltamivir arm led to active arm participants being more likely to develop complications as a result of impaired immune function.

Evidence from prophylaxis and secondary prophylaxis trials suggests that in addition to the apparent similar mode of action as in the treatment studies, suppression of viral shedding in nasal swabs may be of importance. In the former, participants who become positive (i.e. who are subsequently classified as cases of influenza) in the oseltamivir arms are the few who mount a strong response despite oseltamivir interference. The remainder (who are significantly more than in the placebo arm) are classified as prevented or avoided cases. However, as prophylaxis CSRs do not report antibody responses and viral isolate results for the ITT populations either, it is impossible to tell whether or not this proposed mode of action fits all of the evidence. The effect of oseltamivir on nasal shedding is consistent with the proposed mode of action of NIs in preventing the virus from leaving the host respiratory epithelial cells, which are covered by a mucous layer. Compared with the rather small reduction of symptoms of ILI and reduction in antibody rise (up to 10%) by both oseltamivir and zanamivir, the extent of the reduction of symptomatic influenza is almost half. This may be as a result of reduction of influenza viruses in the nasal swab sample.

In prophylaxis there is no evidence that oseltamivir reduces symptomatic ILI. Oseltamivir reduces the number of prophylaxis participants testing positive (based on antibody rise and/or culture test). However, this finding is weakened by oseltamivir’s interference with the viral replication on the swab and effect on antibody production. In addition oseltamivir does not affect asymptomatic influenza and there is no evidence that it interferes with person-to-person spread.

Similar to the FDA,109,153 because of the problems with the design of study WV1579965 we could not draw any conclusions on the ability of oseltamivir to interrupt viral transmission.

This is important as the results of the trial65 formed part of the WHO3 rationale for use of the drug to interrupt transmission from person to person, and allow time before the arrival of vaccines in the event of a pandemic furnishing a seemingly powerful rationale for stockpiling oseltamivir.

This shows the importance of availability of full CSRs, something the WHO did not have.

Antibody suppression seems stronger for oseltamivir than zanamivir, probably because of the difference in bioavailability. It may be that evidence of other effects, such as hyperglycaemia and renal impairment (though significance was marginal) in the prophylaxis trials may be due to inhibition of the host’s endogenous neuraminidase, which impairs the cell function of various organs. 15 Overall, the significance of oseltamivir for nasal shedding is unclear but problems with sampling and culture undermine any claims as to its secondary prophylactic properties, as the FDA made clear in its response. 109

The dose–response increase in psychiatric events in the ‘pivotal’ oseltamivir treatment trials and the increase in vomiting only on day 1 in treatment trials in children may be caused by the sudden onset of the central action of unchanged oseltamivir. 15 Brain concentration of unchanged oseltamivir increases during the early phase of influenza in juvenile animals150 as a result of a reduced or low function of p-glycoprotein, a major transporter of oseltamivir at the blood–brain barrier. 15,149 The likely centrally mediated mode of action of oseltamivir is supported by the finding of adverse events in healthy people in prophylaxis trials. However, these effects may also be derived from a delayed action that is associated with host endogenous neuraminidase inhibition by oseltamivir,15 because this appeared after more than 1 week’s exposure to the drug and lasted for > 2 weeks. Other effects, such as pain in the limbs, hyperglycaemia or diabetic events, reduction of antibody rise and reduction of cytokine induction, may also result from the suppression of the host’s endogenous neuraminidase by oseltamivir. 136 Pain in the limbs and metabolic control events (mainly hyperglycaemia) were in excess in the oseltamivir arms, but we did not carry out a formal meta-analysis, as they were not prespecified in our analysis plan and the number of events was < 10 for metabolic events.

Statements made about the capacity of oseltamivir to interrupt viral transmission and reduce complications are not supported by any data that we have been able to access.

We have not reviewed other NIs, such as laninamivir and peramivir, or other antivirals, such as the adamantanes (amantadine and rimantadine) or antipyretic/anti-inflammatory agents. Laninamivir and peramivir may be more potent as NIs because their bioavailability is far higher than zanamivir and may affect the host’s endogenous neuraminidase. Adamantanes are well known centrally active agents and may be more harmful than oseltamivir and zanamivir. Anti-inflammatory antipyretics (except paracetamol) may be more toxic than NIs. 15 Hence, the other NIs, adamantanes and anti-inflammatory antipyretics may not be alternatives to oseltamivir and zanamivir.

Overall completeness and applicability of evidence

We used the Cochrane seven-domain risk-of-bias instrument to assess bias. The availability of partial or complete CSRs decreased the uncertainty and allowed definitive judgements to be made. Previous unclear risk of bias became certainty of bias or certainty of absence of bias. Certainty or low levels of uncertainty are due to our expectations regarding the complete CSRs. We were expecting to have all relevant and consistent information available for our reviews, but, when it was not, our judgements changed because we found gaps in the availability of information and inconsistent information. We are still uncertain as to whether or not the complete study reports represent an exhaustive and coherent source of trial narrative and data.

In the case of treatment trials, conclusions and generalisations are drawn from a subpopulation in which the two arms do not appear comparable as a result of the apparent ability of oseltamivir to interfere with influenza antibody production. The effect of oseltamivir on the gastrointestinal tract appears to be notable, although a definitive statement will be possible only once the mode of action and dosage of dibasic calcium phosphate dihydrate and dehydrocholic acid have been clarified. The high percentage of influenza infections appears to be in contrast with the need to pool or delay several trials and the small recruitment size of others because of a lack of influenza circulation. In the case of PEP trials, the selection of the infected population has the effect of excluding from the analysis large percentages (in some cases > 50%) of participants. This brings the generalisability of the results of these trials into question.

Much has been made in the trial programmes of viral nasal voidance as a marker of effect. However, its measurement was unreliable in treatment trials, as this verbatim quote from the FDA review shows:FDA, p.14109

Duration of viral shedding was measured from treatment initiation to the time of the first negative virus culture with no subsequent positive cultures. Upon reviewing a list of viral shedding patterns provided by the applicant on 8/16/99, two problems emerged: (1) the pattern of virus shedding was fluctuating in at least 33 subjects (i.e. pos-neg-pos-neg, with or without a subsequent negative result). (2) In at least 100 subjects, the last virus shedding sample was the first negative sample in sequence, meaning there was not a subsequent negative confirmation. Given the fluctuating pattern of virus shedding, to estimate the duration of viral shedding based on the occurrence of a single first negative data poses a high level of uncertainty.

In all of the programmes, the effect on complications was based on unclear and potentially unreliable definitions, often at the discretion of local clinicians, and confirmation (e.g. radiological confirmation of pneumonia) was not consistently reported when it did occur. In the ITT population, the correct population for analysis, there is no credible effect of oseltamivir against pneumonia as the significance of the term ‘pneumonia’ is not clarified.

In the case of PEP trial,65 nasal voidance was measured only in symptomatic subjects, as an adjunct to the protocol version. However, this does not prevent the manufacturers from making claims of effect for all of these outcomes.

Other general requirements, such as presentation within 36–48 hours, raise questions about the generalisability of the research evidence. However, underlying all our doubts is the conflicting evidence on the mode of action of the drug.

Most of the trials were substantially under-recruited and so had insufficient power individually to answer the research question.

Quality of the evidence

We assessed all full CSRs of relevant trials. An example of the kind of detail available in complete CSRs and the importance of the trial timeline in assessing the presence of bias is the observation that of the CSRs for the included trials, only one contained a protocol that predated the beginning of participant enrolment, only two had SAPs that clearly predated participant enrolment and three had clearly dated protocol amendments. No oseltamivir CSR included a clear date of unblinding.

All reports in our review were sponsored by the manufacturers. It is known that published studies sponsored by the pharmaceutical industry are more likely to have outcomes favouring the sponsor than studies that have other sponsors. 154,155 As the evidence relates to published studies, we do not know whether or not the findings are applicable to CSRs.

Potential biases in the review process

The main limitation of our study is our relative inexperience in dealing with large quantities of information and our lack of familiarity with certain trial documents, such as randomisation lists. Randomisation lists appeared to be of two types. The first was a pre-randomisation list of random codes with which participants’ IDs cannot be matched with the participant IDs used within other sections of the CSR. The second was a post hoc randomisation list to which individual participants can be matched but the original generated codes are not shown. In both cases the truly random generation of the sequence could not be properly assessed because either the original codes are not provided or the original codes cannot be matched to patients.

We have created methods and procedures to address the risk of reporting bias that we identified in published trials, but remain uncertain about the success of these new methods.

Agreements and disagreements with other studies or reviews

Several reviews of NIs are now available,156–160 including several separate versions of our previous reviews. 30–32,35 All are mainly based on published information and reach similar conclusions to our 2006 review, which sparked the reader’s comment and subsequent investigation and change of methods.

Following publication of our review update in December 2009, Roche asked the Harvard-based academics Hernan and Lipsitch161 to repeat the Kaiser analysis4 to confirm or reject Kaiser’s conclusions. They were not provided with any funding to carry out this analysis and Roche ultimately provided them with patient-level data sets and module 1 for the 10 Kaiser trials and one more treatment trial. 69 An important methodological difference between Hernan and Lipsitch’s analysis161 and that of Kaiser4 was Hernan and Lipsitch’s decision to privilege a true ITT analysis over the subpopulation analysis featured in the Kaiser analysis. Our Cochrane review also analyses the ITT population.

The Kaiser analysis4 concluded that oseltamivir provided two statistically significant reductions: in lower respiratory tract complications and in hospitalisations.

Hernan and Lipsitch161 evaluated lower respiratory tract complications and found a statistically significant, but smaller, reduction in the risk of these complications.

Hernan and Lipsitch161 omitted evaluating the Kaiser paper’s conclusion that oseltamivir reduced the risk of hospitalisation. They wrote, ‘it was not possible to assess the potential benefit for high-risk participants who are hospitalised, because the sample size of most studies was too small to consider hospitalisation as an outcome’.

Hernan and Lipsitch161 do not elaborate on or highlight their apparent methodological disagreement with the Kaiser 2003 analysis4 and it is not reflected in the news article published on the Harvard website entitled ‘Oseltamivir effect on complications confirmed by reanalysis’ (http://ccdd.hsph.harvard.edu/NewsEvents/Oseltamivir-reanalysis). In fact, Hernan and Lipsitch161 did not confirm one of the key conclusions of the Kaiser paper. 4

Unfortunately, the Hernan–Lipsitch analysis161 has been cited by influential bodies, such as the European Centre for Disease Prevention and Control, as ‘confirmation of the original Kaiser meta-analysis’ (http://ecdc.europa.eu/en/activities/sciadvice/_layouts/forms/Review_DispForm.aspx?ID=561&List=a3216f4c%2Df040%2D4f51%2D9f77%2Da96046dbfd72) despite the fact that Hernan and Lipsitch161 did not confirm one of the key conclusions of the Kaiser paper. 4

For complications, although Hernan and Lipsitch161 clearly produced similar results to Kaiser,4 we do not think that this means that the result is more credible. In view of our findings, we suggest that these results should be interpreted with caution. We have published our preliminary comments. 162 The approach Hernan and Lipsitch161 took in analysing data was insufficient to provide a credible, independent check on validity, and reinforces the importance of detailed, critical assessment of entire trial programmes, with access to full-length study reports. Our analysis questions the coherence between the evidence and the proposed mode of action of oseltamivir.

The Ebell et al. 163 review concluded that there was ‘no evidence that oseltamivir reduces the likelihood of hospitalisation, pneumonia or the combined outcome of pneumonia, otitis media and sinusitis in the ITT population’. This conclusion was based on module 1 of the 10 Kaiser trials plus WV16277. 69 These are the same 11 trials included by Hernan and Lipsitch. 161

Main conclusions

These data show that oseltamivir and zanamivir cause small reductions in the time to alleviation of influenza symptoms in adults, but not in asthmatic children. The use of oseltamivir increases the risk of nausea, vomiting and psychiatric events in adults and vomiting in children and may reduce risk of diarrhoea and cardiac events in adults. Observational studies fail to show that oseltamivir has a protective effect on mortality among patients with 2009A/H1N1 influenza. Prophylaxis with either oseltamivir or zanamivir may reduce symptomatic influenza in individuals and in households but there was no reduction in all other influenza outcomes, including overall ILI reported as an adverse event on-treatment. In previous Cochrane reviews it is likely that evidence from published data was insufficient to fully assess risk of bias within the trials. Our results do not discount a potential benefit of using zanamivir and oseltamivir in individuals under particular situations, for example in immunocompromised or in compassionate cases, for which few other therapeutic options may exist. However, NIs themselves may be immunosuppressants.

The balance between benefits and harms should be considered when making decisions about use of NIs for either prophylaxis or treatment of influenza.

Implications for practice

These results show that both oseltamivir and zanamivir reduce the time to symptomatic improvement in adults (but not asthmatic children) with ILI. The size of this effect is small – approximately half a day. We have no data comparing oseltamivir or zanamivir with paracetamol and other antipyretics. We did not find convincing evidence that either oseltamivir or zanamivir reduces the risk of complications of influenza, particularly pneumonia, or reduce risk of hospitalisation or death. Even in individuals at higher risk of complications, such as children with asthma or the elderly, there was no evidence of a beneficial effect for reducing risks of complications.

The findings demonstrate a minimal effect on prevention of influenza by oseltamivir or zanamivir among individuals or families. This suggests little support for their use as prophylactic agents, for example during influenza epidemics. 164

Implications for research

The considerable body of evidence from RCTs included in this review indicates either no effect or a relatively small absolute effect size against the complications of influenza. Such an effect, even if statistically significant, would be too small to warrant treatment with NIs in a primary care setting, especially as effective diagnosis and treatments for rare complications (such as pneumonia) are available. Lack of evidence of an effect on hospitalisations probably indicates lack of severity in the first place. Assuming an influenza incidence rate of 2% (similar to that in the control arms of oseltamivir treatment trials) to detect a 25% clinically significant reduction in pneumonia, 21,500 participants would have to be enrolled in a clinical trial.

Our findings have implications for research on the mechanism of action of NIs, with special regard to any direct central action of oseltamivir and the inhibitory effect of the host endogenous neuraminidase of various organs and systems. We could not reach a consensus on whether or not further trials are warranted and whether or not current trials should be discontinued.

Published trials are unlikely to provide the level of detail to allow the results of a drug trial to be properly evaluated and risk presenting a partial and potentially biased report of trial conduct and findings. This has implications not only for the reporting of trials but also the weight that can be applied to published studies alone.

Our calculation is likely to underestimate population size, as the 2% incidence rate was derived from trials that used enhanced ad hoc surveillance systems. Any trial design would have to ensure that the presence of complications is ascertained using objective diagnostic criteria (e.g. with confirmation using imaging or laboratory testing for pneumonia). Such trials would also have to consider the ethical implications of conducting studies for which the estimate of benefit (based on 11 RCTs) in otherwise healthy people is likely to be small, and would have to be balanced against the apparent risks of adverse effects from NIs. We think research should be aimed at more effective preventative measures and early identification of complications.

The methods used to conduct our evidence synthesis need to be repeated across further interventions and by other researchers, and may need to be refined further. Given the considerable resources involved in using these methods, a system is needed to prioritise reviews of important drugs so that such methods are reserved for drugs that meet certain conditions. Priority could perhaps be given to first drugs of a new family, drugs considered to be innovative or those that are likely to have a big market impact. Such reviews should be publicly funded and be independent of both regulators and manufacturers. Researchers who conduct these ‘high-scrutiny’ reviews need to be free of recent ties to either government or the pharmaceutical industry. Systematic review groups such as The Cochrane Collaboration should consider adopting these methods for other drugs and whether, perhaps, to scrutinise the published reviews of prioritised drugs.

Implications for policy

These findings suggest that the current recommendations for oseltamivir as an essential medicine for the treatment of seriously ill patients or those in higher-risk groups with influenza 2009A/H1N1 need revision. 33,34 The small benefits in symptomatic improvement and the lack of evidence for an effect on serious complications need to be balanced with the adverse effects found with these drugs in meta-analyses, especially diabetic/hyperglycaemic, renal and neuropsychiatric effects in all of those people for whom the WHO recommends its use.

Policy-makers should be cautious in interpreting and using the findings of systematic reviews including only published studies, particularly those that comprise only a portion of an entire drug trial programme or which contain only a portion of the results of trials. The current findings suggest that numerous national and international bodies may have been based on inadequate or poor-quality information. 165 This could be obviated by ensuring that documentary evidence relating to a trial on humans (including CSRs, regulatory documents, evidence syntheses) should be archived electronically with no statute of limitations.

Several steps are required to provide patients, clinicians and policy-makers with the most transparent assessment of the relative benefits and risks of new drugs.

Based on the length of time it has taken to provide a definitive answer on the efficacy of the NIs, the challenges in obtaining the full information and the methods that we needed to develop to conduct the evidence synthesis, we believe the main implication of our review is the need for reform of multiple components of the research and development, regulatory and assessment pathway of new drugs.

Pharmaceutical sponsors of drug trials should follow a data access and sharing procedure which is similar to that of the EMA, and sponsors should make all full CSRs available to be downloaded from their websites and shared freely once a regulatory decision has been made. Redactions should be kept to the minimum. Part of this process needs to include a full list of the entire drug development programme to avoid assessment of an incomplete set of trials. Researchers and industry employees who are listed in trial documents should be considered to have legal responsibility for the conduct and reporting of a trial.

Regulators should post an inventory of their documentary holdings on their websites with a brief description of the main content and size of each file. They should make all information available shortly after making a registration decision on a drug and within a reasonable time period. The information should be in electronic format and anonymised (i.e. participants’ details should be removed to prevent each person being identified but no further).

Trial registries have improved the reporting of new trials. However, on their own they will not be adequate to resolve the problems we encountered. The completeness of trial registries needs to be tested with a random sampling procedure. Clear instructions for the reporting and updating of their content should be promulgated and penalties imposed on breaches of these procedures. Trial registration should include the original and final versions of a trial protocol, with a full declaration of dated amendments. Procedures for trial unblinding and dates of unblinding should be routinely reported. Registration should be made compulsory for all studies in which human beings are randomly assigned to experimental arms. Ethical and consent procedures for all trials should include obligations of the trial sponsor to ensure that results are made public. Failure to report the existence of a trial on humans and to make results available should be considered as an ethical breach of conduct and be subject to appropriate penalties.

Authors’ note: In reviewing over 2 GB of data there is the possibility of mistakes. The authors would be grateful if readers could identify these. We promise that, if we concur, the record will be amended accordingly.

Objectives

To determine the effect of oseltamivir on mortality in 2009A/H1N1 influenza patients.

Design

Systematic review of observational studies.

Data sources

Summary and individual patient data (IPD) from published observational studies.

Eligibility criteria for selecting studies

Any study of 2009A/H1N1 influenza patients reporting mortality outcomes and exposure to oseltamivir with at least 5% of patients untreated with influenza antivirals and five or more deaths overall.

Main outcome measures

Mortality.

Results

A total of 1117 studies were identified and screened, with 154 full-text articles assessed for eligibility. Of these, 30 observational studies of hospitalised patients were eligible and a total of 11,013 patients were available for qualitative synthesis. Overall there were 1301 deaths (12%) with the percentage of deaths receiving oseltamivir similar to that of survivors (83% vs. 82%). We found evidence of time-dependent bias in the summary data and the IPD. The IPD came from four studies including 3071 patients and 242 (8%) deaths. After taking account of time-dependent bias, potential confounding variables, and the competing risk of hospital discharge, analysis of the IPD showed insufficient evidence that oseltamivir reduced the risk of mortality [hazard ratio (HR) 1.03, 95% CI 0.64 to 1.65].

Conclusions

We found insufficient evidence from 30 observational studies to support oseltamivir having a protective effect on 2009A/H1N1 influenza patients for mortality. However, the included studies were observational and IPD analysis was based on only four studies without adjustment for baseline severity of illness or other drug use hence the findings should be interpreted with caution. Observational studies with time-dependent treatment exposure appear to be at a high risk of time-dependent bias unless an appropriate analysis is conducted.

Quality of the included studies

We included observational studies that reported on series of cases with laboratory-confirmed 2009A/H1N1 influenza. There were no randomised or prospective cohort studies or case–control studies comparing death or severe cases with matched uncomplicated patients despite the call for this type of study. 211,212 Table 16 shows information on the design characteristics of the included studies and Table 17 provides details on outcomes and analysis methods used. The proportion of patients treated in the studies ranged from 35% to 94%, with a median of 82%.

Although all 30 studies classified patients by treatment exposure, only two defined it: one as ‘at least one dose of oseltamivir’181 and the other as ‘at least one day of treatment’182 [although it was unclear how they classified patients who received one dose only; there were 26 such patients excluded from the study [Iratxe Puebla, PLOS ONE journal editor, 7 August 2012, personal e-mail communication)]. It is unknown how many other studies failed to report excluded patients but we note that a number of studies had missing information on mortality status and/or treatment status (for 200 patients in one study,183 for which missing data were more common in patients with less-severe disease), whereas other studies reported complete information. None of the included studies reported over-the-counter medication use prior to hospital presentation. This may be important as, for example, NSAIDs have been shown to increase mortality in influenza-infected animals. 213

Although a number of the included studies conducted analyses adjusting for potential confounding variables, such as patient comorbidities, they used logistic regression or standard survival analysis, which may not be appropriate for observational studies in which treatment exposure is time dependent. This type of analysis misallocates the time from initiation of the study (e.g. hospital admission) to start of antiviral treatment and leads to time-dependent bias. Beyersmann et al. 214 provide mathematical proof that analyses which ignore the time-dependent nature of an exposure variable lead to biased estimates of treatment effect in favour of treatment. Time-dependent bias is also referred to as immortal time bias,215 survivor treatment selection bias216 and survival bias. 217 It causes selection bias because patients who die early do not get an opportunity to receive treatment. In addition, patients who are extremely sick may not be given the opportunity to receive treatment because other more critical procedures take priority. Only two of the included studies182,184 attempted to take account of time-dependent bias but they appeared to have used flawed approaches. One study excluded all deaths within 3 days of onset of symptoms and the other obtained results that appear incorrect because their analysis – taking into account time-dependent bias – increased the treatment effect in favour of oseltamivir. Only six studies reported data on time from hospital admission to treatment initiation (see Table 17). Beyersmann et al. 214 suggest that time-dependent bias can be avoided by proper hazard-based analyses. See Appendix 15 for a detailed description and example of time-dependent bias.

Analysis of summary data from included studies

Table 17 shows a summary of the numbers and percentages of patients with oseltamivir exposure by survival status for each of the 30 included studies. In total there are 11,013 patients included and 1301 deaths (12%). Overall, the percentage receiving oseltamivir among deceased patients (1085/1301 = 83%) was similar to the percentage receiving oseltamivir among survivors (7918/9712 = 82%). Meta-analysis of the included studies showed moderate heterogeneity [I² = 44%, χ² = 51.93, degrees of freedom (df) = 29, p = 0.006, τ² = 0.217] and no evidence of small study effects (p = 0.25). Figure 17 shows a funnel plot of the 30 included studies.

FIGURE 17.

Funnel plot of the 30 included studies. Please note that the pseudo 95% confidence limits are shown by the dotted lines. The funnel shape, bounded by the confidence limits, represents the expected distribution of studies where, assuming no heterogeneity, 95% of the treatment effect estimates should lie.

To investigate heterogeneity we conducted a random-effects metaregression using the metareg command in Stata/IC. Factors investigated included age group (adults vs. children), severity of illness (critically ill vs. hospitalised), log-odds of death (another measure of severity of illness) and log-odds of treatment. This last factor was included to investigate the potential effect of time-dependent bias on treatment effects. If time-dependent bias is apparent then we would expect it to have a greater influence in cohorts that had higher odds of treatment. This is because if few patients were untreated then patients dying before initiation of treatment would tend to have a large influence on the odds of patients dying in the untreated group. Results showed no effect of age group (p = 0.40) or severity of illness (p = 0.38), but evidence of an effect of log-odds of death (p = 0.024) when, as odds of death increases, the treatment effect in favour of oseltamivir increases, and log-odds of treatment (p = 0.003), when, as odds of treatment increases, the treatment effect in favour of oseltamivir increases. These effects appear to be independent as p-values in multivariable analysis are 0.031 and 0.006, respectively.

To illustrate these effects we conducted subgroup analysis by odds of treatment: ‘< 5’ compared with ‘≥ 5’ and percentage of death: ‘< 10%’ compared with ‘≥ 10%’. An odds of treatment of ‘≥ 5’ equates to a percentage of ≥ 83.3%. Results show treatment effect is in favour of oseltamivir and more heterogeneous in cohorts when odds of treatment were ‘≥ 5’. Conversely, in cohorts for which odds were ‘< 5’, heterogeneity is smaller and overall effect is in favour of no treatment (Figure 18). Similarly, treatment effect is in favour of oseltamivir and more heterogeneous in cohorts in which percentage of death was ≥ 10%; conversely, in cohorts in which the percentage is < 10%, heterogeneity is smaller and overall effect is in favour of no treatment (Figure 19).

FIGURE 18.

Meta-analysis of included studies by odds of treatment with oseltamivir.

FIGURE 19.

Meta-analysis of included studies by proportion of death.

Meta-analysis of individual patient data

We were able to obtain IPD from four included studies. 185–188 One was a German study of 93 critically ill children;185 another was a UK study of 1520 hospitalised patients of all ages;186 another was a Canadian study187 of 605 critically ill adults; and the last study188 was a US study of 926 hospitalised patients of all ages. After removing data from patients who received zanamivir and patients with missing death status (n = 73, 2%), we had data for 3071 patients, of whom 242 (8%) died, 1803 (59%) were discharged, 140 (4%) remained in hospital and 886 (29%) had missing date of discharge. Details of responses and non-responses from corresponding authors of included studies to our request for IPD are shown in Table 18.

In Table 19 we present results of analysis of our primary outcome using five different methods. The first is logistic regression; the second is standard Cox regression; the third is Cox regression including treatment as a time-dependent exposure (td-Cox); the fourth is td-Cox with inclusion of potential confounders as covariates; and the fifth is td-Cox with inclusion of covariates as well as imputing missing time from hospital admission to death/discharge (for which 33% of values are missing, mainly due to the US study not providing date of discharge for survivors), time from hospital admission to oseltamivir exposure (for which 10% of values of the patients treated with oseltamivir are missing) and time from onset of symptoms to hospital admission (16% missing). The percentages of missing data were smaller for the deaths than the survivors (0.4% vs. 36% for time to death/discharge for example), suggesting that data collection was more thorough for patients who died.

Results show that when the time-dependent nature of treatment is taken into account appropriately the treatment effects change direction, although none of the results is statistically significant. Adjusting for potential confounders and imputing missing data made little difference to the results. Table 20 shows results of the first three analysis methods by study, and a detailed analysis of the Canadian study,187 shown in Appendix 15, further illustrates why the time-dependent nature of treatment needs to be taken account of appropriately. This study was used for illustration because it had the most deaths and 93% of patients were treated, hence we would expect time-dependent bias to be large. However, bias is also apparent in the other studies if treatment is incorrectly assumed to be time independent (see Table 20). Table 21 shows distributions of potential confounding variables by treatment received stratified by study with p-values based on the chi-squared test. Detailed examination suggests a possible explanation for the reason oseltamivir is associated with greater risk of mortality in the Altmann et al. 185 and Nguyen-Van-Tam et al. 186 studies. In these two studies,185,186 comorbidities are generally more prevalent in the group that received treatment.

In Table 22 we show the variables that were associated with oseltamivir treatment. Respiratory and neurological comorbidities, as well as cancer, were all associated with increased odds of treatment, whereas the ages of < 20 years and > 64 years, as well as missing time from symptom onset to hospital presentation, were associated with lower odds of treatment. Early presentation was not associated with increased odds of treatment. In multivariable competing risks analysis, respiratory comorbidity was protective for mortality irrespective of exposure to treatment (HR 0.60, 95% CI 0.43 to 0.84) and each 10 years increase in age was associated with increased risk of death by 22% (95% CI 10% to 35%). Respiratory comorbidity was also associated with 15% (95% CI 3% to 28%) increased likelihood of discharge suggesting despite the comorbidity these patients had a good prognosis. Furthermore, a two-sample t-test showed respiratory comorbidity was associated with slightly lower Acute Physiology and Chronic Health Evaluation (APACHE) II scores in the Kumar et al. 187 study [mean (SD) 20 (10)10 vs. 22 (10)10; p = 0.014].

The results of mortality analyses by timing of treatment (Table 23) further illustrate the importance of appropriate analysis of a time-dependent treatment exposure. Logistic regression shows a statistically significant reduced odds of death associated with early treatment compared with no treatment, and estimates from standard Cox regression suggest treatment reduces risk of death compared with no treatment, although the result is not statistically significant. However, when time-dependent exposure to treatment is included appropriately in the Cox model, the results show no evidence treatment is beneficial. There is a suggestion that patients receiving late treatment do worse than those who received early treatment; however, this can be at least partially explained by the patients receiving late treatment having worse prognosis due to being older and having increased likelihood of a number of comorbidities, including infection, cardiovascular, diabetes and obesity (Table 24).

Competing risks survival analysis on the three cohorts that provided date of hospital discharge showed insufficient evidence that oseltamivir is associated with mortality (HR 1.03, 95% CI 0.64 to 1.65) but the results showed an association with increased likelihood of hospital discharge for survivors (HR 1.44, 95% CI 1.26 to 1.64).

Strengths

We systematically searched five widely used databases over a 4-year period, as well as hand-searched relevant WHO documents and previous reviews. There were no restrictions on the type of research articles or studies considered for inclusion in the review. Studies from the Americas, UK and Europe, Australasia, Asia and the Middle East were included. We were able to obtain IPD for four studies that included > 3000 patients and more than 240 deaths. Using the IPD we were able to take account of the time-dependent nature of treatment exposure appropriately, impute missing time to event data and adjust for important covariates in the analysis. We were also able to take account of the competing risk of hospital discharge in estimating the effect of oseltamivir on mortality.

Limitations

Limitations of this review include the observational nature and quality of the included studies. All included studies were of hospitalised patients, and severity of illness of patients at baseline was not always reported. Limited information was provided on drug use prior to admission and no study reported NSAID use. Only two studies reported a definition for treatment and the comparison groups were assumed to be the subsets of patients who were not exposed to antivirals. The percentage of unexposed patients was often small (median 18%). Reasons for non-treatment with antivirals were not provided and none of the studies described a policy or criteria used for selecting patients for treatment. Our IPD analysis showed that patients with certain comorbidities were more likely to be treated, whereas children, the elderly and patients with unknown duration of symptoms were less likely to be treated.

We were able to obtain IPD for only four of the included studies, thus limiting the amount of statistical power that we had for our IPD analysis and potentially resulting in a biased subset of the 30 included studies. However, some of the responses to our request for IPD suggest critical data to assess the impact of oseltamivir on mortality were not collected. Furthermore, the IPD we obtained appears to be a representative sample of the 30 included studies as the ORs for mortality were similar for the IPD (0.93 – see Table 19) and the summary data (0.90 – see Figure 19). These estimates are also consistent with the crude OR of 0.92 reported by Muthuri et al. 170 Furthermore, the studies were from four different countries with varying mortality rates (4–20%), varying odds of treatment (60–93%) and included patients with ages from 0 to 102 years.

There were missing data for time to death/discharge and time to treatment (33% and 10% missing, respectively) in the four studies for which we had IPD. The majority of missing data for time to death/discharge was due to one study that did not report date of discharge for survivors. Analysis excluding this study showed similar results to those based on all four studies, hence we do not believe the missing time to discharge data has introduced a significant bias.