The effectiveness, cost-effectiveness and policy processes of regulatory, voluntary and partnership policies to improve food environments: an evidence synthesis

Aims

To search systematically for, appraise the quality of and synthesise evidence on the effectiveness, cost-effectiveness and policy process of population interventions to improve diet, including regulatory interventions, voluntary approaches and PPPs, and to share the evidence synthesis, and formulate recommendations to improve interventions, with stakeholders with a view to informing more effective responses to unhealthy diet in England.

Research questions

1. How are regulatory interventions, voluntary approaches and PPPs to improve diet assumed to work in theory?

2. What regulatory interventions, voluntary approaches and PPPs to improve diet, and reduce inequalities in diet improvement, have been evaluated?

3. What factors relating to interventions, providers, populations and settings affect implementation of such population interventions to improve diet?

4. Have such population interventions improved process, impact (intermediate and distal) and cost outcomes?

5. Are there any reported unanticipated effects of such population interventions?

6. What is the cost-effectiveness of such population interventions?

7. How can the findings of the evidence review be translated into recommendations for improved interventions?

Issues with the Grading of Recommendations, Assessment, Development and Evaluation approach

In the three reviews assessing effectiveness (see Chapters 4–6), we had planned that the effect direction plots would have informed assessments with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach. 28 However, the application of GRADE itself was not possible. In the overview of reviews (see Chapter 4), there were insufficient details in most of the systematic reviews to proceed. In the systematic reviews on private commitments and PPPs (see Chapters 5 and 6), the framework was judged unfit for purpose given the nature of the studies used, and this despite the modifications suggested in the protocol.

The GRADE framework helps determining the level of certainty in the evidence (rated as high, moderate, low, very low) for each outcome. Randomised controlled trials (RCTs) start at the highest certainly level (high) while observational studies start at ‘low’. Certainty in the evidence is then rated down by considering the methodological limitations of the studies (risk of bias), indirectness (applicability and how it represents the research question elements), imprecision [number of events and confidence intervals (CIs)], inconsistency (heterogeneity in results) and likelihood of publication bias. Certainty can also be rated upwards, but this is an exception. As explained in the protocol, we believe that GRADE’s hierarchy of evidence is inappropriate for policy evaluations. It does not reflect the best possible type of evidence that can realistically be obtained for a research question by considering both practical and ethical implications. Thus, in addition to RCTs, we had planned to also attribute a ‘high’ certainty to the following study designs at the start: (1) pre–post time-series analyses and (2) (potentially) cohorts/follow-up studies involving both a comparison group and data collected before and after policy implementation. Eligible comparisons included (1) comparing a policy to either another policy, the absence of a policy, or the same policy but in another location (e.g. different states or countries); (2) comparing participants in a voluntary policy to NPs; (3) comparing products or audience (e.g. TV audience) targeted by the policy to some not targeted by the policy.

However, these modifications did not address the problem. The main issues were that (1) the GRADE framework was developed for specific or narrow research questions while the review included a wide range of heterogeneous interventions and outcomes. The latter is common for policy topics since decision-makers are interested in a range of outcomes, and the policies from which we can learn can be quite different across settings and time. The assessment of publication biases and inconsistency were particularly problematic for this reason; (2) it does little to differentiate levels of certainty in evidence from observational studies. They all are rated as ‘low’ or ‘very low’, and this despite being the best sources that we can sometimes realistically obtain for some policy topics; (3) the exceptions of study designs outlined above were mostly inapplicable to the research questions of the studies included in the systematic reviews in Chapters 5 and 6. Overall, given the nature of the research questions, we could only obtain ‘low’ or ‘very low’ statements no matter what the studies did. We felt that this led to the production of statements on the certainty of evidence that appeared absolute but were in fact uninformative at best, and misleading at worst. It could send the message to decision-makers that there is no good evidence, and that they might as well consider non-evidence-based sources, such as opinions, instead.

Regarding the exceptions outlined in the protocol (i.e. to also attribute a ‘high’ certainty at the start for pre–post time-series analyses and cohorts/follow-up studies involving both a comparison group and data collected before and after policy implementation) (see Appendix 2), they were proven useless because most studies, including all those in Chapter 6, sought to investigate whether a policy had led to improvements in the nutrient content, advertising content or labelling practices. These are outcomes of interest for policies aiming to improve food environments. However, these studies tend to be repeat cross-sectional, not cohort/follow-up studies. They compare the offer of products or adverts available at one point in time to that at another point in time. In this case, examining different products and adverts through time does not represent a methodological weakness. There were also a few document analyses investigating whether a policy was doing what it pretended or was on track. There is currently no guidance for the latter. To avoid producing statements of certainty of evidence that risk misleading decision-makers, a wider discussion about GRADE for policy evaluations in the systematic review community is needed.

What is a systematic evidence map?

Systematic evidence maps, also known as evidence gap maps, are a type of evidence synthesis and research translation tool that visually presents the breadth of research available on an area using a systematic approach. While they are generally used to produce high-level descriptions and identify gaps in evidence, we believe that they can also be employed from a critical perspective to question practices both in research and in the field. 29 In the present case, examining the characteristics of both the policies that have been evaluated, and of the evaluations themselves, can shed light on the body of evidence that countries have at their disposal to make decisions about policy design and effectiveness. Thus, this systematic map of global evidence reports on the breadth, purpose and extent of primary research evaluating the development, implementation, effectiveness and cost-effectiveness of regulatory, voluntary and PPP diet-related policies from both a policy and evaluation perspective.

Methods overview

This systematic evidence map used the literature search, eligibility criteria, screening and data extraction strategies detailed in Chapter 2. As a reminder, the data extracted consisted of (1) general policy characteristics, including countries and World Bank regions, policy names, policy levels, governance approaches and policy categories by adapting the ‘NOURIS’ part of the NOURISHING framework:18 N-Labelling, O-Specific settings including schools, child care, health care and leisure, U-Economic tools (narrowed down to taxes and price reductions on healthy items), R-Advertising and marketing control, I-Product reformulation by manufacturers, I-Retail and food services environment; (2) evaluation characteristics, including publication date; general study aim, study design; and for studies assessing policy effectiveness, the types of participants and outcomes assessed and health equity dimensions measured using a generous interpretation of the PROGRESS-Plus framework,20 which stands for Place, Race, Occupation, Gender, Religion and culture, Education, SES at the individual level, and Social capital as well as Age and Disability for the ‘Plus’. A study was considered as exploring an equity dimension when it compared a policy outcome between different groups by the said dimension.

Data were synthesised narratively using descriptive statistics based on the data extraction categories. Visual maps were produced with EPPI-Mapper (EPPI-Centre, UCL, UK) and Excel.

Included and excluded studies

In addition to the 483 publications reporting on primary studies, the 15 excluded full texts reporting on UK local policies and/or evaluations (see Report Supplementary Material 1, Table S1) (which were part of the 15 UK publications excluded because they were either about the local level or views or the public at the state or national level) were set aside for further analyses. The characteristics of the 483 included publications are described in Report Supplementary Material 2, while those on the local evaluations in the UK and on the excluded publications are listed in Report Supplementary Material 1 (see Table S2). In accordance with Cochrane’s guidance,30 only the characteristics of the full texts excluded for the least apparent reasons are listed, that is because of their policy level (EX 6), evaluation level (EX 7), policy mapping (EX 9), views of the general public (EX 10), evidence synthesis not considering governance (EX 11), UK local policy or evaluation, or public views not consultations, or full texts not obtained (n = 172).

Given the size of this evidence synthesis, we did not attempt at this stage nor for the systematic evidence map of primary research to link publications reporting on the same studies. Thus, the number of ‘includes’ are publications, not studies.

Regional differences across publications

Overall, 70 countries were documented. As shown in Figure 3, there were apparent inequalities in coverage between countries and world regions. The number of evaluations published each year on eligible real-world policies has nearly quadrupled over 11 years, ranging from 23 in 2010 to 85 in 2020 (Figure 4). Studies examining North American countries (38% of publications overall) dominated throughout the period except in 2017 and 2020. The increase in publications worldwide was mainly driven by the World Bank regions of Europe and Central Asia, and of Latin American and the Caribbean (25% and 19% of total publications, respectively), although no Central Asian country was documented and only three studies assessed the Caribbean. East Asia and the Pacific were covered in 18% of total publications. Regarding countries, 30% (n = 146) of publications included the USA, 11% (n = 54) the UK, 11% (n = 51) Australia, 8% (n = 41) Canada, 8% (n = 40) Mexico, 5% (n = 25) Brazil, 4% (n = 19) Chile, 3% (n = 14) France and Spain each, 3% (n = 13) Denmark, and 2% New Zealand and South Africa each (n = 12 and n = 11, respectively). Eighty-one per cent (n = 390) of publications considered these 12 countries alone (without any other country). By contrast, 32 countries were included in only 1 or 2 publications each. While some of these were high-income countries, disparities with and within the least-documented world regions was startling: 12 publications were found about Sub-Saharan Africa (all but 1 about South Africa), 6 about the Middle East and North Africa (4 of which about Saudi Arabia), and 5 about South Asia (all about India). Given that some of these evaluations included multiple countries, the level of details available for some of these countries was particularly limited. One publication had unclear countries and world regions because it assessed companies’ stock markets. 31 Nearly all publications were in English (n = 417), nine were in Spanish, two in Portuguese and one in French.

FIGURE 3.

Number of publications by World Bank world region (n = 482 because it is unclear in one publication). Note: the same publication can include more than one world region (non-mutually exclusive categories).

FIGURE 4.

Number of publications by world region and year (n = 483). Note: since the same publication can include more than one world region (non-mutually exclusive categories), this graph does not reflect the actual total number of publications per year.

Characteristics of policies assessed

The policies examined included 236 national policies (assessed in 73% of publications), 26 groups of state policies (assessed in 26% of publications) and 9 international policies (assessed in 6% of publications). One policy had both a national and a state component. All state policies were implemented in dominant countries (Australia, Brazil, Canada, Spain, the UK and the USA). The five most assessed policies consisted of American state school food standards (n = 37), the Children’s Food and Beverage Advertising Initiative (CFBAI; a national voluntary self-regulation advertising industry code in the USA, n = 21), the Mexican tax on sugar-sweetened beverages (SSBs) (national, n = 19), the UK SDIL (national, n = 14), and various American state SSB taxes (n = 14). Together, these three taxes represented 54% of publications about economic interventions (see below). The three most examined international policies were the Australasian Health Star Rating (HSR) (Australia and New Zealand, n = 11), the EU Pledge (n = 5) and the WHO Code of marketing breastmilk substitutes (n = 5).

Using the six ‘NOURIS’ categories, as shown in Figure 5, the most assessed policy categories were those specific to school, child care, health care and leisure settings (O, n = 122, 76% of which on schools), followed by labelling [N, n = 105, 39% of which about front-of-packs (FOPs)], advertising and marketing control (R, n = 104, 55% of which about television alone) and economic interventions (U, n = 94, 86% of which about SSB taxes). Evaluations of product reformulation by manufacturers were much less common (I, n = 66, 61% of which about salt) as well as those on the retail and catering sectors (S, n = 12, 83% of which at least aimed to increase the availability of healthy options). Eleven assessed policies that covered a wide range of NOURIS domains (these were grouped separately).

FIGURE 5.

Number of publications by NOURIS category (n = 483). Note: a publication can include more than one policy category (non-mutually exclusive).

Governance arrangements assessed

Sixty per cent (60%, n = 288) of publications reported on at least one regulatory initiative, 43% (n = 209) on at least one voluntary action, and 16 assessed mixed governance policies (e.g. the combined use of regulatory labelling and voluntary limits for trans-fats in Canada) (Figure 6). Evaluations of voluntary policies mainly consisted of those led by the private sector (self-regulation and pledges, n = 97) and actions by the public or not-for-profit sector (n = 90), while a minority has investigated PPPs (n = 31). Most publications on PPPs were about the RD in England, UK (n = 12) or the Australian Food and Health Dialogue (FHD) and/or its updated version, the Healthy Food Partnership (n = 10).

FIGURE 6.

Number of publications by governance mechanism (n = 483). Note: a publication can include more than one governance approach (non-mutually exclusive categories).

Figure 7 shows the distribution of publication by governance approach and world region. Regulatory approaches represent the majority of publications in all World Bank regions, ranging from 39% in East Asia and the Pacific to 87% in Latin America and Caribbean. East Asia and the Pacific is the region where governance approaches have been assessed the most evenly. Voluntary actions by the private sector were the second most assessed approach in North America and Europe, voluntary actions by the public and not-for-profit sectors were second in Latin America and Caribbean, the Middle-East and North Africa, and South Asia, and the two voluntary categories were equal in East Asia and Pacific and Sub-Saharan Africa, although the sample size in some regions is too small to draw conclusions. As highlighted previously, most PPP evaluations were conducted in Europe (especially the UK) and East Asia and Pacific (especially Australia). There were few evaluations of mixed approaches across all world regions.

FIGURE 7.

Number of publications by world region and governance approach (n = 483). Note: a publication can include more than world region and governance approach (non-mutually exclusive categories).

When combining information on governance approach, world region and policy category together, evaluations about labelling (N) were mainly about regulatory initiatives in North America (especially menu labelling) and Latin America [especially front-of-pack labelling (FOPL)], followed by the voluntary FOP HSR in Australia and New Zealand. Most publications on specific settings (O) evaluated regulatory interventions in schools in North America, then in Europe. Evaluations of economic interventions (U) were mostly concentrated in Latin America, Europe, and the USA, most of which were regulatory since they were taxes. Publications on advertising and marketing control (R) were mainly in North America, followed by East Asia and Pacific and Europe, and were predominantly voluntary actions by the private sector in all three regions. In fact, 53% of evaluations of voluntary approaches by the private sector in this policy category consisted of the CFBAI in the USA; the CFBAI in Canada; the Publicidad, Actividad, Obesidad, Salud code in Spain; and both the Australian Responsible Children’s Marketing and Australian Quick Service Restaurant Industry’s Initiatives (QSRI). Evaluations of product reformulation (I) were about equally distributed across governance approaches in the four dominant world regions, with slightly more PPP evaluations in East Asia and Pacific. The few evaluations of policies targeting the retail and catering sectors or using a wide range of policy categories were only in North America, East Asia and Pacific and Europe, with various distributions of governance approaches. We formulated the hypothesis that evaluations of the least assessed world regions and countries focused on back-of-pack labelling, assuming that it might be the most frequently implemented policy given its simplicity and neutrality. This was not the case: the 18 evaluations covering Africa, the Middle East and South Asia documented a variety of policy areas, which were mainly regulatory (including five on SSB taxes). Similar conclusions were made for the 24 publications that included the 32 least evaluated countries (countries assessed once or twice, in any world region).

We explored whether similar trends applied for local policies and/or evaluations using the 15 publications that were excluded because they were about the local level in the UK. One focused on Scotland, another on Wales, and the rest on England. Together, they examined nine different policies in only two categories: specific settings (O, n = 8; six regulatory about schools, two voluntary by the public sector in hospitals) and the retail and catering sector (S, n = 7; three regulatory, two voluntary by the public sector, two voluntary by the private sector). The number is substantial given that only 12 publications were identified for that category in the whole evidence map. This potentially reflects the greater capacity to implement such initiatives at the local level, whereas other policies such as taxes and TV advertising control are easier to establish at a higher level. The local regulatory initiatives in the retail and catering sectors included planning regulations within the healthy weight strategy and takeaway planning restrictions. The voluntary approaches by the private sector consisted of the Change 4 Life programme, the Healthy Catering Commitment in London, and self-regulated checkout policies by some supermarket chains.

Study aims, participants and outcomes assessed

The vast majority of evaluations (n = 389, 81%) assessed the effectiveness of a policy, followed by factors affecting their implementation (n = 67, 14%), factors influencing their development (n = 34, 7%), how a policy was portrayed in the news (n = 11, 2%), responses to public consultations (n = 10, 2%) and cost-effectiveness analyses (n = 4, 1%). The remaining study investigated whether the New Zealand Advertising Standards Authority self-regulation code protects child rights. Only the fifth (n = 25) of the 119 evaluations assessing other aspects than effectiveness covered other countries than the 12 dominant countries (i.e. USA, UK, Australia, Canada, Mexico, Brazil, Chile, France, Spain, Denmark, New Zealand and South Africa). Given that these types of studies aim generally to employ qualitative or mixed methods, unsurprisingly, the latter study designs were also mostly employed in these countries (as well as Fiji). Figure 8 shows the number of publications by governance approach and study aim category. PPPs were those that were the most assessed holistically with 32% of publications assessing factors influencing implementation and 16% assessing policy development. They were followed by regulatory policies (12% on implementation, 9% on development, and representing most studies on media coverage and public consultations). By contrast, policies led by private actors (mainly self-regulations and pledges) were particularly underevaluated on these aspects with only 7% of publications on implementation and one evaluation of policy development.

FIGURE 8.

Number of publications by study aim category and governance approach (n = 483). Note: a publication can include more than one study aim and governance approach (non-mutually exclusive categories). Dark green: ≥ 100 publications; pale green: 75–99; yellow: 50–74; red: 25–49; white: < 5.

Studies assessing effectiveness focused on a wide range of ‘participants’ or samples: out of 389 publications, 43% (n = 166) relied on data collected data via humans only, 57% did not involve humans at all (i.e. they collected data directly on the environment, in the news or documents, n = 182), and 11% (n = 41) involved both. Consequently, the type of outcomes evaluated also varied. Nine per cent (n = 35) have investigated health-related outcomes [e.g. mortality, diseases, disability-adjusted life-years (DALYs), nutritional status and anthropometric]; these were nearly all conducted in the USA, followed to a smaller extent by Denmark (n = 4) and Portugal (n = 3). The other types of outcomes considered included human behaviours (e.g. dietary intake, sales, purchases, advertising viewing and use of labels; 35%, n = 137), environment features (66%, n = 256), policy characteristics or implementation status (6%, n = 22) and other outcomes (2%, n = 9). All policy categories mainly assessed environment features except for U-Economic interventions, which mostly examined human behaviours.

Study designs employed

As explained in Chapter 2, study design was documented for all primary studies. The design of 67% (n = 254) of quantitative studies was further extracted using the classification of natural experiments by Leatherdale19 as well as presence of a comparison group enabling comparison of either policies, policies versus absence of policies, or participants/products within a policy versus NPs/products.

As described in Figure 9 and more detailed in Table 3, 78% (n = 376) of publications employed a quantitative design, 13% (n = 63) a qualitative method and a minority either used both (mixed methods; 6%, n = 27) or analysed policy documents alone (4%, n = 17). Among quantitative studies, most were single post cross-sectional studies (one data collection, 44%, n = 165), followed by repeat cross-sectional studies (32%, n = 121), follow-up and time series (22%, n = 81), and modelling studies and scenarios (n = 9). Note that the label ‘follow-up studies’ was only applied to human participants because non-human ‘participants’, such as products and adverts, are generally not the same through time. Pre–post studies (n = 160) were more common than post–post (n = 41). Three studies focused on the implementation phase alone (i.e. before the official policy implementation date).

FIGURE 9.

Publications by study design (n = 483). Note: a publication can only include one study design (mutually exclusive category).

Table 3 shows the number of publications by study design for the whole map (n = 483), for the publications only focusing on the 12 dominant countries (n = 390, 81% of the map), those focusing only on the USA (n = 141, 29% of the map) and those including other countries than the 12 (n = 93, 19% of the map). According to Leatherdale, the most robust natural experiments are those that include a control (or comparison) group, a pre–post design, as well as time series and follow-up studies. 19 The most used study designs in publications covering only the 12 dominating countries were single cross-sectional studies (34%), followed by repeat cross-sectional studies (26%) and follow-up studies and time series (17%). The same trend was observed for publications on the USA alone, although with a higher proportion of single cross-sectional studies in the American studies (41%). Publications on other countries also had a majority of cross-sectional studies (33% single, 22% repeat) but these were followed by qualitative studies (20%) and then follow-up studies and time series (14%). Quantitative studies in all three categories of publications all tended to use more pre–post designs than post–post. However, nearly all policy analyses, mixed-methods studies, modelling studies and scenarios were about the 12 dominating countries alone. Regarding the use of comparison groups, we documented the number of publications reporting on a quantitative study comparing two policies (or more), a policy versus none, or participants or products targeted by a policy versus some not targeted. Overall for the whole map, 57% of quantitative studies involved such a comparison compared with 60% for the 12 dominating countries alone, 66% for the USA alone and 44% for other countries. Thus, except for pre-post designs, publications focusing only on the 12 dominating countries tend to employ more frequently the most robust quantitative designs or features (i.e. follow-up studies and time series, and comparison groups) compared with publications covering other countries, as well as policy document analyses. However, a smaller proportion of their publications employ qualitative methods. The majority of the publications covering the least documented world regions (i.e. Africa, the Middle East and South Asia) were single cross-sectional or qualitative, although the latter were mainly about South Africa.

Regarding the use of quantitative study design by governance approach, 81% of follow-up studies and time-series analyses, and seven of the nine modelling studies assessed regulatory policies. Very few follow-up studies and time series investigated voluntary policies by the public/not-for-profit or private sectors (7% and 11%, respectively). PPPs were mostly assessed with pre–post cross-sectional studies (n = 6). By policy category, most follow-up studies and time-series analyses, and most modelling studies were employed to evaluate economic interventions. Since these were mainly taxes, it explains why these study designs primarily focused on regulatory interventions.

Health equity as a policy outcome

We documented the number of evaluations assessing policy effectiveness comparing a policy outcome between groups by PROGRESS-Plus equity domains. Overall, out of 389 studies used, 50% did not consider any equity domain, 50% assessed at least one, and 21% measured two or more. Age was the most assessed (30%), followed by education (16%, although these mainly related to school characteristics), SES at the individual level (15%), gender/sex (13%), race and culture (11%) and place (8%), whereas only 11 publications considered occupation, and 1 or 2 examined religion, social capital and disability each. This was using a generous interpretation of the PROGRESS-Plus framework, which also considered characteristics of products and places (rather than just humans) and those contributed to higher numbers for age and education. Equity was most frequently considered in the USA (n = 75 with at least one equity dimension), Australia (n = 20), Canada and Mexico (n = 18 each), the UK (n = 17), as well as Chile and France (n = 8 each). Figure 10 illustrates the distribution of publications by year and equity dimension assessed. While the number of quantitative and mixed-methods studies reporting on at least one dimension has increased from 2018, the proportion to the number of studies of effectiveness published per year has reduced, from 72% (n = 13 out of 18) in 2010 to 40% (n = 28 out of 70).

FIGURE 10.

Number of publications comparing policy outcomes by PROGRESS-Plus factors, by publication year. Note: a publication can include more than one health equity dimension (non-mutually exclusive category).

We explored whether evaluations conducted at a smaller scale could capture equity more frequently. In the 15 publications on local policy and/or evaluation in the UK, 8 assessed the effectiveness of a policy. Only four reported on at least one equity dimension: SES (n = 4), place (n = 1), and age (n = 1) representing a similar proportion than for the evidence map, albeit the very small sample size limiting such conclusions.

Eligibility criteria and quality appraisal

Fifty initial systematic reviews were identified in the overarching search strategy (see Chapter 2). Additional screening was comprised first, considering only those assessing the effectiveness of interventions. Within these, only the results sections or subgroup analyses containing a majority of studies meeting the project’s eligibility criteria (screened by one reviewer LB) were included. Second, the evidence syntheses without quality or risk-of-bias appraisal were excluded. Third, to limit primary study overlap between systematic reviews, as recommended by Cochrane,32 the ‘most recent, highest quality, “most relevant”, or “most comprehensive” systematic review for groups of overlapping reviews’ were selected.

The quality of each systematic review was assessed using the checklist for systematic reviews and meta-analyses developed by the SIGN with minor modifications, first, excluding systematic reviews without a quality or risk-of-bias appraisal, and second, merging the two lowest of four categories of overall quality proposed by SIGN. 33 The three resulting categories were as follows: ‘high’, ‘acceptable’ and ‘low’. We also added guidance for some of the questions to limit variations in interpretation between reviewers (see Appendix 3, Table 25). Three independent reviewers performed quality appraisal in pairs (LB, SR, CK) in the EPPI-Reviewer Web.

Primary study overlap is a frequent issue in an overview of reviews consisting of having the same primary studies included in multiple systematic reviews. This gives more weight to their findings than those of other primary studies, thus introducing biases, and making data extraction and data synthesis challenging to perform. 32 To overcome this, we employed Cochrane’s approach to ‘all non-overlapping reviews’, and selecting the most recent, highest quality, ‘most relevant’, or ‘most comprehensive’ systematic review for groups of overlapping reviews. The strategy for following this recommendation in the overview of reviews was as follows: First, we extracted general policy and study characteristics for all potentially eligible systematic reviews at this point (see Data extraction for more details). Second, following Cochrane guidance30 we documented primary study overlap in the systematic reviews by creating a matrix and calculating the ‘corrected covered area’. The latter considers the number of publications (including double counting) in evidence syntheses, the number of unique studies, and the number of reviews. 34 Overlaps ranging between 0% and 5% are considered as slight, 6% and 10% as moderate, 11% and 15% as high and more than 15% as very high. Additionally, we documented the number of reviews that overlap with others, the percentage of studies in each review that overlap with others, and the percentage of unique studies that overlap overall. Third, in each group of overlapping reviews, we selected the review that had the highest overall quality (for ‘most robust’), the most recent search date (for ‘most recent’), and that contributed most to the body of research evidence in terms of the number of studies included, topic (NOURISHING policy categories covered) and place (World Bank world regions covered) (for ‘most relevant or comprehensive’). Justifications for the choices were recorded for reasons of transparency. The selection was made by one reviewer (LB).

Data extraction

General review characteristics extracted in a standardised form consisted of which results sections or subgroup analyses were included and excluded, the number of studies included in these analyses, literature search dates in databases, funding, competing interests, and authors’ affiliations. Characteristics of the policies assessed within the selected results sections consisted of countries, policy categories using the NOURISHING framework,18 the types of outcomes assessed, and the equity domains (or effects for population groups that are prone to health inequalities) represented using the PROGRESS-Plus framework. 20 Data were extracted by one reviewer (LB) and a sample verified by another (CK).

The results of the interventions in the eligible results sections were extracted by policy governance approach, type of outcome and equity domain. Some results sections included non-eligible policies. To keep it simple, we extracted information for the whole section; however, results of policies that did not focus on the food environment were disregarded. Given that the results extracted by equity only covered three of the ten domains (gender, SES, age) of PROGRESS-Plus, we also extracted data on equity from the systematic reviews that had been excluded for reducing primary study overlap. This led to the documentation of two additional domains (race and education) as well as additional outcomes within the domains already covered. We also noted that some systematic reviews had assessed their data or drawn conclusions in relation to a hierarchy or ‘level’ of interventions and we documented these instances. Lastly, we briefly documented how the policies work (their mechanisms of action) as described by the review authors (usually as part of their results or discussions).

Data synthesis

As expected, a high degree of heterogeneity was observed among the primary studies and systematic reviews in terms of types of policies and outcomes evaluated. We adapted the effect direction plot developed by Thomson et al. 26 and Boon et al. 27 to visually display non-standardised effects by governance approach, type of outcome and quality rating, but at the review level rather than primary study level. We classified the direction of effect by type of governance approaches compared (regulatory vs. voluntary, regulatory alone, or voluntary alone). Direction of effect was documented independently from statistical significance, consulting primary studies when not reported in the systematic reviews where necessary. The same approach was applied to reporting the number of studies and total participants by outcome. We had planned to assess the level of certainty in the evidence for each outcome using the GRADE framework but information available was insufficient. Thus, as a compromise, the effect direction plot was also used to present jointly the direction of effect, review quality and indirectness.

Included studies

As outlined in Figure 11, of the 50 (49 unique) evidence syntheses identified in the overarching search, 5 were excluded as they did not evaluate the effectiveness of policies, 23 for not appraising study quality and 6 for not including a results section meeting the eligibility criteria. This left 15 systematic reviews, which were assessed for primary study overlap. Nine were rated high quality (including three Cochrane reports), four as acceptable and two as low. The assessment of primary study overlap and selection of reviews in each overlapping group of reviews is presented in Appendix 5, Tables 27 and 28. The latter process led to the full exclusion of 4 additional reviews35–38 (2 of high quality, 2 of acceptable quality) and of 1 results section in an additional review39 leaving 11 reviews in the overview of reviews (see Appendix 5, Table 30). However, as explained in the methods, three of the reviews35,36,38 excluded because of primary study overlap were used to document effects on specific population groups (equity). The excluded evidence syntheses along with the justifications are listed in the Report Supplementary Material 1. For the sake of transparency, the latter also lists the five publications representing four systematic reviews that were excluded as part of the screening for the overarching project (see Chapter 2) because of an insufficient focus on governance (i.e. governance was either not documented or documented but not considered in the analysis).

FIGURE 11.

PRISMA flow chart representing the selection process. Adapted from the PRISMA template by Page et al. (2021). 16

Primary study overlap

Calculations for the corrected covered area, which is the current recommended method to measure primary study overlap, suggests that before excluding reviews for reducing primary study overlap, overlap was already ‘slight’ (1.3% overlap). However, 9 of the 15 reviews overlapped with other reviews: 5 with 1 other review, 3 with 2 other reviews, and 1 with 3. Among these nine overlapping reviews, six had 80% or more of their studies included in other reviews; three of which overlapped at 100%. Overall, the 15 reviews included 98 unique primary studies, 17 of which overlapped once (i.e. it was included in 1 other review), and one twice (i.e. it was included in 2 other reviews). The detail of primary study overlap is presented in Appendix 5, Tables 27–29.

By excluding reviews to limit primary study overlap, overlap fell down to 0 with 92 unique primary studies and 11 reviews remaining. Among the primary studies excluded, 18 were included in other included reviews, whereas 6 (6%) did not. Yet, excluding the latter six had little impact since one did not assess policies, another one was not an evaluation, and four assessed SSB taxes (two in the USA, one in France and one in Mexico) – a topic that is well covered by other systematic reviews included in the overview.

Quality appraisal of systematic reviews

Table 4 presents the results for the quality appraisal. Seven were rated high quality (++), two acceptable (+) and two low (−). The main reason for attributing low ratings was inappropriate synthesis methods (in addition to inappropriate or lack of use of quality or risk-of-bias assessment). The main reason for attributing ‘acceptable’ ratings was inappropriate or lack of use of quality or risk-of-bias assessment. Other common issues related to not listing the excluded studies (n = 7) and not reporting competing interests in the included studies (n = 6). Most assessments of publication bias were rated as ‘not applicable’ due to the absence of meta-analysis in the selected results sections.

Characteristics of included systematic reviews

Table 4 describes the general characteristics of the systematic reviews included. The number of eligible studies per review ranged from 1 to 23. One conducted their search in databases in 2020, four in 2019, while the oldest were in 2015 (n = 3). None reported on PPPs. Twenty-eight countries were covered overall, all of which were high or upper-middle income countries according to the World Bank with the exception of Iran, Mongolia and Vietnam. The most assessed countries consisted of the USA (n = 6 systematic reviews), Australia (n = 4), as well as Canada, Denmark, Hungary, Mexico and the UK (n = 3 each). All focused on the general population. In terms of real-world policy topics assessed, seven evaluated labelling schemes (on front and back of packages, shelves and menus), five assessed economic interventions (taxes on salt, saturated fat, SSBs/non-essential foods), three included product reformulation initiatives, and two focused on the retail and catering sector. None considered initiatives in schools, nurseries, health care or leisure settings, nor advertising and marketing control. Three focused on SSBs or sugar, two on fat, and two on salt. Regarding the outcomes evaluated, nine investigated effects on consumer behaviour (e.g. dietary intake, purchases/sales), four (on the food environment (e.g. nutrition composition, price), three on health (e.g. heart diseases, obesity), and two on the economy (imports, governments’ and food industry’s revenue). Two conducted meta-analyses. None of the included systematic reviews reported industry funding though one author in the review by Dodd et al. 40 had been a paid consultant for the Novartis Foundation and Midway Corporation; and the systematic review by von Philipsborn et al. 46 was partly funded by the Danone Research Center to the National Institute of Public Health, Cuernavaca, Mexico, and two authors have received grants from the latter of the British Nutrition Society.

Quality and characteristics of systematic reviews and results sections excluded to reduce primary study overlap

Four systematic reviews35–37,50 were fully excluded to reduce primary study overlap, two of which were rated as high quality, and two as acceptable. Patterns in quality appraisal was similar to those included in the overview. Details of the quality appraisal is presented in Appendix 5, Table 30. The additional systematic review,39 which had a results section excluded to reduce overlap had an acceptable quality (Table 5). The five reviews and results section included between three and eight studies and covered the USA (n = 4), Denmark (n = 2), as well as in Canada, Costa Rica, France, Hungary and Mexico (n = 1 each). Four assessed different types of taxes (SSBs, sugar, saturated fat and restaurants), and another examined trans-fats relating to labelling, reformulation, and the retailing and catering sector. All assessed consumer behaviours, four assessed health outcomes, three the food environment and one economic impacts. Thus, apart from a few health and economic outcomes, excluding these to reduce primary study overlap appears not to have affected the scope of the studies included, but also reduced the overemphasis on the USA and on taxation that would have been present otherwise.

Direction of effects by governance approach for the general population

Table 6 describes the directions of effects of interventions categorised by types of governance approaches and their comparison, and types of outcomes. The first column describes 25 groups of outcomes grouped into consumer behaviours (n = 15), the food environment (n = 5), health (n = 3) and the economy (n = 2). The next three columns describe three categories of governance approaches that were assessed or compared: regulatory versus voluntary, regulatory alone and voluntary alone. As mentioned earlier, no review assessed PPPs. In each of these cells, the direction of the arrow indicates whether results were overall positive (or desired), negative (or undesired) or mixed. In the column comparing regulatory to voluntary actions, a larger triangle indicates a larger effect compared to the other governance approach. Twelve (48%) categories of outcomes only included regulatory interventions, five (20%) only voluntary actions and eight (32%) both.

Direction of effects for specific population groups (equity)

This section reports findings for specific population groups using the PROGRESS-Plus equity dimensions. Data come from seven systematic reviews: four of high quality, two acceptable and one low. Four of them44,45,49,50 were included in the whole overview of reviews, while three35,36,38 had previously been excluded to reduce primary study overlap. Table 7 illustrates the directions of effect stratified by PROGRESS-Plus dimension while the specific results extracted for each review are presented in the Report Supplementary Material 2. Only two policies, in two different reviews, were not regulatory. Both provided information on gender. The direction of the arrow indicates whether results were overall positive (or desired), negative (or undesired) or mixed while its size reflects the sample size. Most policies were American state SSB taxes, and most results relied on a single policy, although some evaluations had large sample sizes. Only 5 out of 10 dimensions were documented. SES was the most frequently documented. No information was found about place, religion, occupation, social capital and disability. Effects were generally positive for the population groups evaluated for race/ethnicity and gender, and mixed for SES, education and age.

Race/ethnicity was analysed in two systematic reviews. According to Hyseni et al. 38 (high quality), following the introduction of regulatory labelling and voluntary limits for trans-fats in the USA, trans-fat concentrations in blood samples of white (non-Hispanic) adults (n = 229) fell by about 56% over 9 years (reduced by 24.3 µmol/l; 95% CI 19.6 to 29.0). Alagiyawanna et al. 36 (acceptable quality) reported one study that found an association between American state taxes on SSBs and soft drink consumption among African American children and adolescents (direction of effect presumably positive), and in another study that effects on body mass index (BMI) varied by race and ethnicity (no details).

Gender was documented in two systematic reviews. In Alagiyawanna et al. 36 (acceptable quality), two studies found a negative association between state taxes on SSBs and fast-food restaurants and women’s BMI. One (n = 1,948,833) recorded a reduction of 0.55 kg/m², (p < 0.05) while effect size and precision estimates were not provided for the other. Hyseni et al. 50 (low quality) noted a reduction in salt intake in both men and women between the introduction of regulatory salt labelling and voluntary limits in Finland in 1987 and 2007 [reduced from 13 to 8.3 g/day in men, and from 11 to 7 g/day in women (n ~ 1000 individuals)]. A similar trend was observed for the voluntary salt labelling and limits in Denmark between 2006 and 2010: reduced from 10.7 to 9.9 g/day in men, and from 7.5 to 7.0 g/day in women (sample size not reported).

Education was only documented in the review by Backholer et al. 35 (high quality). The review found contradictory results from two studies on American state SSB taxes. One found no relationship between variations in taxes and BMI in adolescents by level of parental education; the other reported a significant reduction in BMI following a 1% increase in SSB tax rate among both adults who had a high-school diploma (0.0031 kg/m²) or a college diploma (0.0076 kg/m²).

Socioeconomic status was documented in three high-quality systematic reviews. Teng et al. 45 reported that effects of taxes on non-essential food in both Mexico (on purchases of taxed food) and Hungary (on consumption of taxed sugar-added food) were greatest in low-socioeconomic households. Impact of SSB taxes on SSB and calorie intake was mixed in Mexico and the USA, and were mixed as well for purchases and sales of SSBs in Mexico, the USA and Spain (n = 6 studies). Pfinder et al. 44 reported that the Hungarian Tax on non-essential food was most effective in the lowest income quartile (likely for the consumption of unprocessed sugar or sugar-added food). Backholer et al. 35 mentioned, on the one hand, no association between variations in American state SSB taxes and BMI in children from low-income families and on the other hand, a significant BMI reduction in both low-income (0.015 kg/m²) and high-income adults (0.008 kg/m²).

Age was documented in three systematic reviews. In a meta-analysis, Teng et al. 45 (high quality) compared the effect of American state SSB taxes on SSB intake between children and adults and found no statistical difference (it reduced by 6.4% in adults and by 7.7% in children, p = 0.91, five studies in the USA). In Alagiyawanna et al. 36 (acceptable quality), one study found that American state SSB taxes had a greater effect on BMI in middle-age and older people, while in another children from low-income families consumed 0.142 SSB less by week following a 1% increase in a SSB tax rate (statistically significant but arguably negligible). Lastly, in Rincón-Gallardo et al. ,49 effects from three studies (n = 291 restaurants) were inconsistent on whether regulatory menu labelling improve the nutrition composition of children’s menus. The first study, in the USA, found that mean calorie content of new items was lower than in the previous menus. In a second American study, mean energy content had reduced in quick-service chains by 40 calories and increased in upscale restaurants by 46 calories (no precision estimates). The third study, in Australia, found that mean energy, saturated fat, sodium and sugar had reduced by serving for some menus but had increased in others (no precision estimate).

Direction of effects by intervention level

Four systematic reviews of various quality ratings assessed different types of policy interventions and/or commented on their effectiveness by considering the level of restriction of choice or degree of structural change. Hillier-Brown et al. 41 (high quality) used the Nuffield intervention ladder53 to analyse policies aiming to improve takeaway meals. They found that policies restricting choices or manipulating price were more effective than those only providing information or enabling choice. They suggested that ‘the level of intrusiveness’ played a greater role than the type of policy itself. Downs et al. 48 (acceptable quality) came to similar conclusions for trans-fats policies. They found that limits, and particularly bans, were the most effective for reducing trans-fat levels in the food supply and diet while labelling had a lesser impact and was uneven across product categories. Hyseni et al. 38 (low quality) employed the upstream versus downstream classification, or population versus targeted interventions to assess salt-related policies. They concluded that ‘comprehensive multicomponent strategies involving upstream population-wide policies (regulation, regulatory reformulation and food labelling)’ had a much greater impact than downstream interventions targeting individuals (e.g. dietary counselling, media campaigns alone). They did not differentiate intervention levels within the ‘upstream’ category but highlighted the potential of combining interventions, such as labelling with salt targets. Lastly, Sisnowski et al. 39 (acceptable quality) concluded that regulations applied in isolation could improve the food environment and other distal outcomes but not dietary intake so far; yet they mentioned the promising results of taxation, which were published after their systematic review. Like Hyseni et al. ,50 they recommended combining interventions.

To explore the policies assessed in the overview of reviews further, we classified them by intervention levels and identified those with several components using the Nuffield intervention ladder. 53 However, since this project focused on interventions modifying the food environment, we did not document education programmes and public campaigns. As shown in Figure 12, most of the interventions assessed eliminated choice (by eliminating a nutrient or reducing its level), guided choices through disincentives (taxation) or provided information (labelling). One guided choice by changing the default (sides offered in children’s meals) while pledges by the retail sector encompassed several levels, from labelling to eliminating choice (reformulation).

FIGURE 12.

Types of policies assessed in the systematic reviews by governance approach and intervention level using the Nuffield intervention ladder. Black and grey: the Nuffield Intervention ladder. 53 Blue: regulatory. Green: voluntary. Orange: mixed (both regulatory and voluntary components). Line with ‘+’ sign: multicomponent intervention.

Furthermore, while some limited evidence suggests that it can help the public to make informed choices, has the support from the population, and can encourage food reformulation, Downs et al. and Hyseni et al. 38,48 highlighted that they are likely to increase inequalities for two reasons; (1) products can continue to contain a high amount of salt, sugar and fat, and low fibres, FV, especially cheaper options, so people from low SES are more likely to continue to consume them; (2) whether labels are regulatory or voluntary, customers need to understand and use them, which requires a certain health literacy level. By completely eliminating trans-fats, bans were identified as not only the most effective approach for reducing trans-fat consumption but also the most equitable. Therefore, the level of intervention, or extent of structural changes involved, potentially plays a role in both policy effectiveness and the reduction of inequalities. Yet, structural interventions also have their own challenges politically. Sisnowski et al. 39 mentioned that government policies can be softened to ‘appease industry and political opponents’, for instance, by bringing the focus on individual behaviours or reducing the extent of structural changes involved, thus misaligning them with public health recommendations.

Literature search and eligibility criteria

This systematic review used the primary studies from the systematic evidence map that assessed the effectiveness of a PPP (see Chapter 3, Figure 8). To be considered as such, policies had to either be clearly labelled as a PPP or involve collaboration between at least one public and one private actor. Studies of any design (including document analyses) were considered if they assessed effects on the food environment, human behaviour or health, or the economy; or if they assessed the content or progress of the policy itself. Results had to be provided for PPPs separately, that is those aggregated with findings for other governance approaches were excluded.

Data extraction

In addition to the policy and evaluation characteristics extracted for the systematic evidence map (see Chapter 2), in a standardised extraction form we documented the following information: policy and study objectives, the policy leader and other main actors involved, implementation and data collection dates, study design, samples’ characteristics, results about effects including policy content, project or adherence as well as effect size and precision estimates, and potential competing interests of study authors using information from the declaration of competing interests, funding sources and authors’ affiliations. Data were extracted by one reviewer (CK, GB) and checked by another (CK, GB, JB, LB).

Study quality appraisal

Study quality was assessed using a modified version of the Newcastle–Ottawa Scale for cross-sectional studies. Studies were rated on seven domains and rated overall as having a low, moderate, high or unclear quality. Each item was assessed by one reviewer and checked by two others (JB, GB, LB). Disagreements were discussed until reaching consensus. Details about the tool are provided in Chapter 2 and Appendix 4, Table 26. This includes specifications for the document analyses. For appraising the representativeness of the sample, we considered whether the literature search was comprehensive; for the sample size justification, we considered the variety of information sources used taking into account the study aim; we considered the ‘analytical methods’ rather than ‘statistical tests’ only; questions relating to ascertainment of exposure and confounders were deemed non-applicable (because the documents were about the policies themselves, and there is currently no guidance for assessing confounders for such studies including if this is relevant). For studies involving human participants, both the original question about non-response and the modified guidance about missing data were applied. For appraising ascertainment to exposure, when participation status in a policy was unclear in a study, we compared the number of participants in the policy in general to those mentioned in the study in order to get a sense of whether participation status was up to date and likely to have remained relatively stable during the study or not.

Data synthesis

Due to high heterogeneity in the outcomes assessed, data were synthesised narratively by considering the general direction of effect by type of participants (human, environment, documents) and policy as well as overall study quality.

Public–private partnership characteristics

Seven PPPs implemented in four countries between 2007 and 2016 were assessed (Table 8). Nine of the studies were about the FHD in Australia. Two of these also assessed the Australian Division of World Action on Salt and Health (AWASH) voluntary programme, including one together with the New Zealand Heart Foundation (NZHF) voluntary programme (the three policies overlapped in time). 54,55 Four studies evaluated the Public Health RD in England, UK,13,56–58 and another four investigated PPPs in the USA (the HWC twice,51,59 Choose Healthy Now (CHN)60 and the Strong4Life School Nutrition Program61). All studies in Australia and New Zealand (n = 9) focused on salt reformulation, two of which also assessed interventions to improve sugar, fat, fibres, fruit, vegetables content and portion size. In the other countries, the policy categories based on the NOURISHING framework ranged from menu and on-shelf labelling to food reformulation (calories or fat), the retail and catering sectors, and schools.

Study characteristics

Two studies were cross-sectional,57,60 10 were repeat cross-sectional (7 pre–post,51,54,58,59,61–63 3 post–post55,64,65), 1 a time-series analysis59 and 5 were policy document analyses. 13,56,66–68 Five had humans as study participants (directly or by using purchases or sales data),13,56,66–68 six assessed food items, one assessed display of calorie labelling and five assessed documents. The human studies were about the USA and the UK, the studies of products were about Australia and New Zealand, the study of labelling practices was about the UK, and the policy document analyses were about Australia and the UK. All policies were assessed between 2 and 8 years after their introduction, giving time for the participants to implement the changes. Sample size of participant groups ranged from 16 restaurant chains (sample size can be especially small when studies used the number of companies as a unit of analysis, which is limited) to more than 2500 products. All studies of food products focused on salt content and were from Australia (with one on New Zealand as well). Two studies involving humans assessed the purchases and sales of calories, one trans-fat intake, and two looked at awareness, belief and/or self-efficacy relating to the policy. Two of the five document investigated achievements while the remaining three focused on the policy process, for instance whether the PPP encouraged participants to do more than what they were already doing, whether the actions they commit on are likely to be effective based on evidence, and whether they document progress and make it a priority.

Quality appraisal

Nine of the 17 studies were rated as having a low quality overall, 3 as moderate, 3 as high and 2 as unclear (Tables 9–11). Six of the nine studies about Australia were given a ‘low’ rating, whereas the three studies rated ‘high’ were policy document analyses. The main reason for an overall low quality was the lack of control group (which was not applicable to document analyses), which highly increased the risk of bias. Two studies involving humans only included people exposed to the PPP, and five studies of humans or products combined data for both PPP participants and NPs together, making it difficult to determine whether the effect was due to the PPP or not. Nevertheless, of the latter five, three were about the FHD in Australia, where a high proportion of products on the market were part of the PPP. For instance, Trevena et al. (2014)62,64 noted that about 95% of the market share of processed meats, 85% of pasta sauces, 80% of bread products and 60% of breakfast cereals were from participants in the FHD. Changes (or lack of changes) in these products overall are likely to be at least partly due to changes (or lack of) in participating products in the PPP.

Furthermore, justifications for sample size in studies of humans and products, and of information on non-responders in human studies were mostly judged as non-appropriate or not reported (unclear). Only seven studies reported information on missing data and how they were dealt with. Eight did not report on the number of people collecting or verifying data independently, contributing to a low quality for assessment of outcomes since such information missing likely means that only one person was involved. Quality of the analytical methods was generally rated high. All policy document analyses were judged to have sufficiently diverse sources of information.

Effects of public–private partnerships

Table 12 presents a summary of the study characteristics and key findings. Six studies assessed changes in salt content in Australia, all relating to the FHD and AWASH programme, which took place concomitantly. Findings pointed towards no or mixed effects overall, with four studies (three of low quality, one moderate) showing mixed or no effects, and two studies on the FHD (one low and one unclear quality) showed improvements overall. On the one hand, Levi et al. (2018)63 (unclear study quality) found that sodium levels in soups had fallen by 6% between 2011 and 2014. Sparks et al. (2018)65 (low quality) assessed sodium levels in processed meats between 2010 and 2017. They reported an 11% reduction in median sodium levels in the processed meats that had a salt target versus no change in those without a target. However, by category of products they reported mixed results with significant reductions in bacon, ham/cured meat and wet savoury pasties (p < 0.001, p = 0.012 and p = 0.006, respectively) and significant no change in emulsified luncheon meats (p = 0.363 and dry savoury pasties (p = 0.111). On the other hand, Dunford et al. (2011)54 (low quality) found that while the proportion of breads meeting the national salt target increased from 29% in 2007 to 50% in 2010, the mean sodium content did not change (434 mg/100 g in 2007 vs. 435 mg/100 g in 2010). Trevena et al. (2014)62 (low quality) noted no significant difference in the mean sodium content of pasta sauces (451 mg/100 g in 2008 vs. 423 mg/100 g in 2011; p = 0.016), cautioning the need to reformulate sauce products well beyond the FHD commitment. Another study by the same authors64 (moderate quality) examined breads, ready-to-eat breakfast cereals and processed meats. They reported significant reductions of sodium in bread (unlike Dunford et al. ) and breakfast products, and some reduction in processed meats, though with no apparent difference between participating brands, calling into question whether the partnership drove the reduction. Christoforou et al. (2013)55 (low quality) set out to report on changes in sodium levels in the ready meal market between 2008 and 2011. They concluded that the voluntary efforts of industry to reduce sodium levels of these foods over 4 years yielded a reduction of less than 1%. They ascribed this to the lack of co-ordination across industry and a failure of the Australian government to take leadership in improving the quality of the food supply.

In Australia again, three studies assessed potential achievements or the policy process of the FHD using documents. Elliot et al. (2014)66 (low quality) assessed whether the FHD was reducing premature death and disability but found that the data available were limited, with several reports on achievements missing, concluding that the current voluntary approach to preventing diet-related ill health is insufficient. Jones et al. (2016)67 (low quality) evaluated the FHD over 6 years (from October 2019 to September 2013) but found limited data to assess achievements,67 much like Elliot et al. (2014). They found that no information about milestones had been published since October 2011. They concluded that strong government leadership, adequate funding, clear targets and timelines, management of conflict of interest, comprehensive monitoring and evaluation, and a plan for responsive regulation in the event of missed milestones will be required to ensure that the new partnership set out as a continuity of the FHD, the Healthy Food Partnership, achieves its public health goals. 66 This echoes the studies on the Public Health RD (England, UK) below. 56 Lindberg et al. (2017)68 (high quality) set out to understand whether salt reduction was a priority for processed food manufacturers in Australia. They reported that manufacturers did not appear to be making significant and comprehensive positive (nutrition) changes in relation to salt. This included 16 of the 33 manufacturers assessed not providing evidence of reducing salt in their products.

For New Zealand, Dunford et al. (2011) (low quality) reported an estimated 7% reduction of mean sodium content in bread products compared to none in Australia, highlighting important disparities between company efforts. The authors felt that this indicated lack of government leadership, non-governmental organisations (NGOs) lack authority and resources, and a renewed need for committed leadership from the government. 54

Four studies evaluated the Public Health RD in England, UK. As part of the RD evaluation covering alcohol, food, physical activity and health at work, Knai et al. (2015)56 (high quality) reported on the effectiveness of the food pledges using policy documents. Much like Elliot et al. (2014) and Jones et al. (2016) in Australia, they found that the lack of data reported by participating food industries made the evaluation challenging. They reported an overall lack of additionality of the RD, as most of the reported actions were already underway (meaning that the PPP had had little added value). They also found that the majority of the food pledges propose interventions that favour information provision, awareness raising and communication with consumers, which may have limited effect. Another study by Knai et al. (2017) (high quality) focused specifically on the RD’s trans-fat reduction pledges. Based on progress reports from 2015, the pledges appeared to have had negligible impact on reducing artificial trans-fats from England’s food supply beyond pre-2011 levels. 13 The authors concluded that the most effective strategies to improve diet are not reflected in the RD food pledges, and that there was paucity and heterogeneity of organisations’ progress reports. Hutchinson et al. (2018)58 (low quality) assessed whether the RD resulted in different intakes of trans-fatty acids (TFA) across different socioeconomic groups before (2000–11) and after the PPP (2010–2). They reported that voluntary reformulation had had some effect in reducing TFA of many UK products, when comparing data from the National Diet and Nutrition Surveys before and after the RD implementation. However, the surveys only include a small variety of popular foods and brands from large retailers, which may mask TFA differences between foods consumed by different population groups. High TFA consumption was associated with socio-economic disadvantage pre-RD, but post results were less clear regarding inequalities. Finally, Robinson et al. (2019)57 (unclear quality) studied calorie labelling practices in the UK eating out of home sector in 2018 and whether they were aligned with the RD pledges. They found that it was rare for eating out and takeaway chains to provide in store kcal labelling and when labelling is provided it does not meet recommended labelling practices.

Four studies from the United States. Two of these, both by Ng et al. 201451,59 (moderate quality), evaluated the impact of the HWC on total calories sales and calories purchased. They both reported successes, including over 10% fall in calories sold over the 5-year pledge period among participating companies, though absolute caloric sales from retailers’ private label products essentially did not change. As with the RD evaluation findings, the additionality of the HWC, that is whether there was added value from signing up to it to motivate change (e.g. reformulation), was questioned, and was reported as potentially relating to factors beyond the pledges such as changes in consumer demands, or economic factors. Moreover, it should be noted that 13 of the 16 signatories also belong to the CFBAI, which is another self-regulated voluntary pledge by the industry. Two other American partnerships were evaluated: Rajbhandari-Thapa (2017)61 (low quality) evaluated the Strong4Life School Nutrition Program. They found that the programme’s training increased school nutrition manager and staff member knowledge of the Strong4Life Smart Serving Strategies, as well as their beliefs, self-efficacy, and confidence in their ability to make positive behavioural and environmental changes in the school cafeteria. Beckelman (2020)60 (low quality) assessed the Hawaii State Department of Health’s work to scale up the CHN programme through partnerships with two convenience store chains at a statewide level. The programme was considered a mutually beneficial partnership, with leadership buy-in, support for implementation, and a need to meet retailer needs, for example, modifying signage size and type.

The findings remain the same when looking at the types of participants or study quality with a majority of studies pointing towards inconclusive or no effects overall except for human studies. Studies of food products and labelling practices showed overall mixed or negative results independently of study quality. Similar observations can be made for the document analyses, which all pointed towards a lack of information, lack of commitments, and/or unlikeliness of the policy to provide added value independently of their quality. As for the five human studies, they showed mixed results overall. The three human studies that showed overall positive effects were of low quality, while the two of moderate quality showed mixed results. This might be due to the types of outcomes assessed: two of the positive studies reported on participants’ awareness and self-efficacy rather than changes in the environment or human behaviours and another measured trans-fat intake, while studies with mixed results examined calorie sales and purchases. The type of products assessed could also explain the differences in directions of effects noted for the FHD, that is it is possible that progress was indeed made in soups and processed meats overall but not in other product categories.

Literature search and eligibility criteria

The systematic evidence map identified 183 primary studies that assessed the effectiveness of a voluntary policy (including the voluntary element of interventions, which include both a regulatory and voluntary component) (see Chapter 3, Figure 8). Studies of any design (including document analyses) were considered if they assessed effects on the food environment, human behaviour or health, or the economy; or if they assessed the content or progress of the policy itself. Results had to be provided for voluntary approaches separately, that is those aggregated with findings for other governance approaches were excluded. Three additional criteria specific to this review were also added and screened by a single reviewer (SR, LB): (1) Initiatives developed and implemented by a single company were excluded; (2) The study had to compare organisations participating in the policy to NPs in terms of adherence or compliance to policy guidelines or effects on the food environment, dietary intake or health; (3) Due to the high number of studies potentially eligible and the limited time available, we focused on research articles published in the past 4 years (2017–20). Annual reports on advertising control policies of the voluntary and private sectors were excluded. The references of the included studies were screened against the above eligibility criteria.

Data extraction

In addition to the policy and evaluation characteristics extracted for the systematic evidence map (see Chapter 3), we documented in a standardised extraction form the following information: policy and study objectives, the policy leader, implementation and data collection dates, study design, samples’ characteristics, results about effects or adherence between participants and NPs including effect size and precision estimates, and potential competing interests of study authors using information from the declaration of competing interests, funding sources and authors’ affiliations. For studies on advertising control policies, we only extracted outcomes relating to unhealthy foods because these are the ones that such policies normally aim to control for; and when on TV we captured such outcomes during both children’s peak time and children’s programmes (CP) since children are normally the target population of these policies. Data were extracted by one reviewer (SR, GB, JB, LB) and checked by another (GB, JB, LB).

Quality appraisal

Study quality was assessed using a modified version of the Newcastle–Ottawa Scale for cross-sectional studies (see Chapter 2). Studies were rated on seven domains and rated overall as having a low, moderate, high or unclear quality. Each item was assessed by one reviewer and checked by another two. Modifications to the tool are explained in Chapter 2 and detailed in Appendix 4, Table 26. For appraising ascertainment to exposure when participation status in a policy was unclear, we compared the number of participants described in the policy characteristics data extraction form to those mentioned in the study to get a sense of whether participation status was up to date in the study and was likely to have remained relatively stable.

Data synthesis

Due to the high heterogeneity in the outcomes assessed, data were synthesised narratively by policy and type of outcome by considering the direction of effect between participants and NPs. For a graphical representation of the summary findings, we used an effect direction plot26,27 (see Chapter 2), displaying non-standardised effects across multiple outcome domains. Studies were grouped by policy and ordered by overall study quality, publication date and study design. The effect direction plot was also used to present jointly the direction of effect, study quality and indirectness, which was a compromise for not using the GRADE framework (see Chapter 2).

Commitment characteristics

Nine policies, implemented across seven countries [Australia, Belgium, Canada, Germany, the Netherlands, UK (England), USA], were evaluated. Table 13 presents the characteristics of the policies across the time period that they were evaluated. Four policies were in Australia, namely the FHD (n = 2),63,65 the Healthier Australia Commitment (n = 1),70 the QSRI (n = 3),71–73 two of which also included the Responsible Children’s Marketing Initiative (RCMI) (n = 2). 71,72 The remaining five consisted of the CFBAI in Canada (n = 5),74–77 the European (EU) Pledge evaluated in Belgium, the Netherlands and Germany (n = 2),78,79 the Public Health RD in England, UK (n = 1),57 and both the CFBAI (n = 2)80,81 and Kids LiveWell (n = 1)82 in the USA. Seven of these policies [Canadian Children’s Food and Beverage Advertising Initiative (CAI), CFBAI, EU Pledge, Healthier Australia Commitment, Kids Livewell, QSRI and the RCMI] were officially led by a private organisation representing food and beverages corporations while the remaining two (RD and FHD) were led by a governmental institution. The Australian FHD, the Healthier Australia Commitment, and the RD in England were PPPs. The number of participating commercial actors across the policies ranged from 7 (QSRI) to 150 (Kids LiveWell82).

Regarding the policy categories assessed using the NOURISHING framework, unsurprisingly, the five advertising control policies (CAI, CFBAI, EU Pledge, RCMI, QSRI) were evaluated from an R-Advertising and marketing perspective; the QSRI was also assessed in the S-Retail and food service sector (children’s meals in fast-food outlets). The effect of both Australian partnerships was assessed on product reformulation (I) while studies of Kids LiveWell and the RD focused on labelling.

The five policies that include an advertising and marketing control component implied not to advertise unhealthy food to children under a certain age, which varied between 6 years old with the CFBAI in the USA to 12 years old with the CAI in Canada and the EU Pledge (although as part of the EU pledge, the Netherlands committed to prohibit it to < 7-year-olds only). The nutritional criteria used to define healthy or unhealthy food varied across the policies. The FHD was launched with specific sodium targets for healthier products. Healthier Australia Commitment provided a relative percentage decrease in sodium, saturated fat and energy. Four policies (CAI, CFBAI, EU Pledge and QSRI) were launched without unified criteria but implemented criteria later on (CFBAI in 2013, QSRI and EU Pledge in 2014 and CAI in 2018). Signatories to the RCMI developed their own nutritional criteria and the definitions were reported as being unclear (9). Nutritional criteria were also unclear or vague for Kids LiveWell and the RD.

Study characteristics

All studies were cross-sectional: nine measured data at a single point in time (single cross-sectional) and seven at two points in time after the policy was implemented (repeat pre–post). None used humans as participants but rather analysed food items, adverts and other environmental features. All policies were assessed at least 3 years after their introduction, which should have given some time for participants to implement changes, and 12 studies evaluated effects after at least 7 years after implementation (especially for advertising and marketing outcomes). Sample size of participant groups ranged from seven companies [sample size can be especially small when studies compare number of companies participating (P) and NP, which are limited] to about 100–200 participating products or adverts. One study assessed over 35 million food adverts on websites from participating companies.

Study quality appraisal

The overall judgement of each domain and final rating for each study is noted in Table 13. Studies about advertising and marketing control were generally of higher quality than those on the retail and catering sectors. Overall, six studies were assessed as having a low quality overall, five as moderate, and four were judged ‘unclear’. None were rated as high. Across all studies, there were concerns identified about selection bias with three studies using convenient or non-representative samples. There were generally low concerns about the ascertainment of the exposure except for one study mentioning that participation status in a policy had changed through time and was not accounted for in the analysis; otherwise, there were no obvious reason to believe that authors did not use an up-to-date list of participants or that participant status changed during the evaluation period and was not accounted for bias arising from comparability of participants and confounding factors controlled, although in some studies information was unclear. In outcome assessment, two studies had their data collected by a single researcher only. Only five studies included appropriate information on how they dealt with missing information and nine justified their sample size. When reported, the use of statistical tests was judged appropriate, but four studies failed to report information on their statistical analysis.

No important conflicts of interest were identified for authors of any of the 16 studies included (Table 14). Two studies’ lead author had previously worked on research projects funded by the food industry (American Beverage Association, Unilever, Nestlé, and the Dairy farmers of Canada). In 10 studies, authors declared having no conflict of interest; the remaining did not have a competing interest declaration. Funding was received from government or research bodies, universities, and private or not-for-profit organisations. Six studies did not report their source of funding. Authors were affiliated with universities or other public organisations (with affiliations partly unclear in three studies).

Effects of voluntary approaches

Table 15 presents the study characteristics and key findings.

The CFBAI in Canada was the most assessed policy, with four studies (three moderate and one low quality) concluding that there were mixed effects for the advertising of unhealthy foods to children on television,74 that participants used more child-directed marketing strategies to promote unhealthy foods on websites than NPs,77 that food advertised on websites to children were in majority unhealthy,76,77 and that participants did not conduct fewer corporate social activities to promote their brands to children. 75

The CFBAI in the USA was evaluated twice (moderate and unclear quality) and shown overall as having worse effects among participants than NPs. Participants advertised unhealthy foods during children’s television programmes more frequently,80 used more marketing practices on breakfast cereal packages, and the cereals said to meet CFBAI’s nutrition criteria had smaller serving sizes and lower density, suggesting that their nutritive values do not represent quantities likely to be eaten. 81

The EU Pledge had mixed effects in two studies (moderate and low quality) for a range of outcomes relating to advertising of unhealthy foods to children both on television and online including the number of adverts, nutrition profiles, use of marketing tactics and children’s protection strategies online. 78,79

The FHD in Australia was the only policy to show some better overall outcomes for participants, albeit with limitations including imprecise data. 65 The study by Levi et al. 63 (unclear quality) was a post–post cross-sectional study evaluating whether soup products, classified as dry and wet (canned and chilled) met the policy’s sodium targets. Sodium content was collected in 2011, 2012, 2013 and 2014 using food databases and labels. In 2014, 1153 soups were assessed, of which 68 were dry from participants (compared with 20 NPs); and 125 were wet from participants (compared with 87 NPs). Figures (no data provided) show a greater proportion of participating soups below the target for both categories compared to NPs. Mean sodium content in dry participating soups reduced significantly since 2011 but not for wet soups nor either NP soup categories, although this might be partly due to small sample sizes limiting statistical power. No values on sodium content and number of products meeting targets were provided. The study was rated as having unclear study quality due to a lack of reporting on missing data and ascertainment of participant status. Sparks et al. (2018)65 (low quality) assessed sodium levels in processed meats between 2010 and 2017. They reported an 11% reduction in median sodium levels in the processed meats that had salt targets versus no change in those without targets. However, by category of products they reported mixed results with significant reductions in bacon, ham/cured meat and wet savoury pasties (p < 0.001, p = 0.012 and p = 0.006, respectively) and no change in emulsified luncheon meats (p = 0.363) and dry savoury pasties (p = 0.111).

A study by Spiteri et al. 70 (unclear quality) on the Healthier Australia Commitment assessed ‘healthfulness’ of all new food and beverages launched in the country in 2015 using three nutrition profiling systems. Participants had launched greater proportion of foods classified as unhealthy according to the three systems (all p < 0.001).

Regarding the Kids LiveWell in the USA, Moran et al. 82 (low quality) compared trends in calories, sodium and saturated fat content of children’s menus (entrees, side dishes, desserts and beverages) in participating and NP large chain restaurants. Results were inconclusive for all three indicators.

The QSRI in Australia was investigated in three studies (one moderate, two low quality). Effects on television advertising practices for unhealthy foods to children were either mixed or worse among participants. 71,72 Wellard-Cole73 found that fewer children’s meals from fast-food chains exceeded recommendations for saturated fat but that the proportions of meals exceeding calorie recommendations were greater among participants and inconclusive for sugar and sodium (no inferential statistics performed).

Lastly, a study by Robinson et al. 57 (unclear quality) investigated whether the 16 restaurant chains that had signed the calorie labelling pledge in the Public Health RD in England tended to display calorie labels and meet the seven labelling criteria compared to 88 NP restaurant chains. While a greater proportion of participating chains displayed labels, only one participating and two NP chains provided calorie labelling for all items sold, and none fulfilled all criteria.

Tables 16 and 17 present a summary of the nine categories of outcomes assessed by direction of effect, sample size and study quality. The shape of the triangles illustrates the overall direction of result independently from statistical significance while their size reflects the final sample size of intervention group (participants). The subscript numbers represent the number of outcomes within each category synthesis. The colours refer to the study’s overall study quality.

Despite the effect direction tables not considering effect size nor precision estimates, they show that there is no clear evidence to suggest that policies designed as voluntary approaches led by commercial actors to reduce advertising or other promotion of unhealthy products to children, improve the nutritional composition of food and beverages, encourage calorie menu labelling in chain restaurants, or reduce marketing targeting children on food packages. When comparing participants to non-participants, the direction of effect or association for most outcomes was either inconclusive or worse for participants. No pattern could be noticed with overall study quality, country, policy category or type of outcome assessed.

An examination of the effect direction plots (see Tables 16 and 17) suggests that the results do not vary by study quality.

Literature search strategy and eligibility criteria

This systematic review builds on the primary economic studies (e.g. cost-effectiveness, cost-benefits) that were identified as part of the systematic evidence map (see Chapter 3). Economic simulations and projections were eligible if the data used for measuring effectiveness met the overarching eligibility criteria. Given that only four evaluations of cost-effectiveness were retrieved, two additional searches were conducted in MEDLINE (Ovid) to explore whether eligible studies had been missed (see Appendix 6, Tables 31 and 32): a very comprehensive and sensitive search on FOPL based on the search strategy by Croker et al. 43 and a less sensitive search on fat, salt and sugar reformulation. These two topics were chosen because of their higher odds of being less well captured by the original literature search given the focus of the latter on whether policies were regulatory, voluntary or PPPs. If more than one additional eligible study was to be identified in each search, additional searches would have been conducted for the other policy areas covered in the evidence map. However, since none was identified, this was deemed unnecessary. Lastly, the reference lists of the included studies were screened as well as those of evidence syntheses on cost-effectiveness excluded from the overview of reviews (see Chapter 4).

Data extraction

Information on the policies (name, topic, country, policy level, governance approach), study methods for assessing effectiveness (population, study design, mechanism, diseases considered), methods for assessing costs (types of costs considered, choice of model) and results (incremental health outcomes and costs, conclusion on whether a policy was considered cost saving) was extracted in a standardised extraction form by one reviewer (CL) and checked by another (LB). These fields were inspired by the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist. 84 Cost estimates were inflated and/or converted in 2020 international dollar using country-specific gross domestic product (GDP) deflator index and purchasing power parity conversion factor in 2020 from World Bank Databank. 85,86

Quality appraisal and data synthesis

The methodological quality of the studies was assessed by two reviewers independently (CL, LB) using Drummond’s 10-criteria checklist version 201587 for assessing economic evaluations. Each criterion was assigned ‘Yes’, ‘No’ or ‘Can’t Tell’. Disagreements were resolved by discussion.

Data were synthesised in a descriptive manner using the categories of information extracted. The initial plan was to summarise the measures of costs and cost-effectiveness using the policy areas described in the NOURISHING framework. 18 However, this proved impractical because of the small number of studies included. We had also planned to pool outcome estimates from comparable studies in a random-effect model meta-analysis to characterise average intervention impacts, but this was not possible due to the aggregation of the outcomes in broader indicators in some studies and the diversity of cost measures used.

Characteristics of included studies

As mentioned above, of the 483 primary studies included in the systematic evidence map, only four were economic evaluations of real-world policies. 88–91 No other evaluations were identified in the additional literature searches nor by screening references. Table 18 provides an overview of the policy interventions assessed in the four studies. These include two studies of the SSB tax in Mexico. 88,91 This SSB tax is a regulatory government intervention that was implemented as an excise tax of 1 peso per litre in 2014. This tax was added to non-alcoholic beverages with added sugar. 88 The third study assessed voluntary salt reformulation in the context of the Public Health RD in England. 89 The Public Health RD was a PPP in operation in England from 2011 to 2017. It aimed to foster collaboration between the government, business and voluntary sectors to improve population health. One of the pledges launched under the RD was to reduce population-level salt intake. The fourth study was on the HSR Labelling Scheme in Australia. 90 This is a voluntary FOP label initiative endorsed by the New Zealand and Australian governments. Packaged food and drink products are given a rating that ranges from half a star (least healthy) to five stars (most healthy).

Quality appraisal

The quality appraisal of the four included studies is reported in Table 18. All studies met at least 5 out of the 10 Drummond checklist criteria. Mantilla Herrera et al. 90 was assessed to be the highest quality because it was the only study to receive positive answers to all assessment questions. The other three studies88,89,91 were assessed negatively on regarding whether the research question was well-defined as the prospective of their analysis was not clearly stated. Without a clear specification of the prospective taken, it was also not possible to judge if all the important and relevant costs and consequences for each alternative were identified in these three studies. By contrast, using a limited societal perspective, Mantilla Herrera et al. 90 considered the costs to the government and the food industry but not other wider societal costs such as productivity loss. The earlier study on SSB tax in Mexico91 received the least number of positive answers as it did not clearly explain the competing alternative nor justify the effectiveness measure used. Additionally, no incremental analysis of costs and consequences were performed in this study, limiting the comparability of their results to other studies or any pre-determined willing-to-pay (WTP) thresholds.

Measures of effectiveness and cost

Table 19 highlights the heterogeneity in the included studies both in terms of the study populations and methods used for assessing effectiveness of the interventions. Two of the four studies examined the impact of the interventions on both children and adults,89,90 while the other two focused on adults aged 30 years or older. 88,91 In particular, the study on FOPL in Australia by Mantilla Herrera et al. 90 modelled the cost-effectiveness over the lifetime of the population of ages while the other studies used a shorter time horizon approach (10 years or less).

For measuring effectiveness, three studies used evidence obtained from quasi-experimental designs to identify policy changes outside the direct control of researchers: Mantilla Herrera et al. 90 and Basto-Abreu et al. 88 both used a difference-in-difference approach that included control groups while Laverty et al. 89 used an interrupted time-series design without a control group. By contrast, Sanchez-Romero et al. 91 assessed scenarios of effects using existing estimates of price elasticity of demand in a state transition model. The evidence around causality was stronger in the first two studies88,91 as the difference-in-difference approach they utilised the trend in the control group to forecast the trend in the intervention group that would have been expected if the intervention had not happened. The actual causal estimate was then obtained by comparing the outcomes before and after the change for the intervention group to the corresponding difference for the control group, allowing to infer causality. By comparison, the measure of effectiveness in Laverty et al. 89 was obtained from quasi-experimental designs without a control group, which makes it more prone to bias by confounding events that took place during the same time period as the RD. Furthermore, there is a lack of longitudinal data on salt intakes in the same people; thus, causality cannot be ascribed in the effectiveness measure of RD. As for Sanchez-Romero et al. ,91 although the price elasticity of demand used reflected real-world consumer responses to price changes, it was not an actual casual estimate of the effect of the Mexican SSB tax. Additional limitations in the measures of effectiveness in the studies include the lack of data on purchases outside of stores (i.e. Basto-Abreu 201988) and assumptions of consumer responses to the intervention. 90,91

In all four studies, changes in the distribution of weight and/or systolic blood pressure were used as intermediate outcomes to evaluate the impact of the policy interventions on CHD and stroke. All but Laverty et al. 89 also considered diabetes. Other diet-related health outcomes evaluated included various cancers, knee and hip osteoarthrosis, and non-cardiovascular disease mortality.

Methods for assessing policy effectiveness are summarised in Table 20. Two studies sought to apply quality-adjusted life-years (QALYs) or health-adjusted life-years (HALYs) as outcomes. 88,90 These are commonly used metrics that facilitate comparison between studies and different types of interventions and diseases. Even though all studies considered CHD and strokes, two89,90 did not explicitly report the number of cases reduced for these diseases. Both studies on the Mexican SSB tax showed reduction in the cases of CHD, strokes and diabetes but with quite different magnitudes of reduction. This could be due to differences in the underlying measurements of effectiveness as well as the mechanism used to model these health outcomes. The HSR labelling in Australia was predicted to result in incremental health benefits equating to 4207 HALYs (95% UI 2438 to 6081). While all health-related outcomes improved in studies assessing the Mexican SSB tax and HSR labelling, the study on the RD in England found the reverse relationship: their results suggest that the partnership has increased the CVD and cancer burden as the decline in population-level salt intake slowed down after its implementation in 2011.

There were variations in the types of other costs assessed across the four studies although they all estimated healthcare cost implications of the intervention. Basto-Abreu88 and Mantilla-Herrera90 assessed costs associated with implementing the interventions for governments, and the latter also considered potential costs to the food industry. Note that neither study on SSB tax considered potential societal costs and benefits from the increase in government revenue. This could be because the included studies may have taken different analytical perspectives (e.g. the healthcare sector, the government, the society as a whole and etc. 95). For instance, productivity changes are typically not considered under the healthcare sector perspective, which were also excluded under the limited societal perspective taken in the Mantilla-Herrera’s study on HSR rating in Australia. 90 Laverty et al. 89 was the only study to assess the broader economic implications by estimating costs included by changes in productivity. However, it did not consider implementation costs. While not clearly stated, Sanchez-Romero 201691 might have adopted a healthcare sector perspective and hence only healthcare costs were assessed.

Cost and cost-effectiveness of interventions

The incremental health and cost outcomes are summarised in Table 21. Overall, with their positive health outcomes, both the SSB tax in Mexico and FOPL in Australia identified reductions in healthcare costs. In addition to the differences in the measure of effectiveness and mechanism shown in Table 3, Basto-Abreu et al. ’s study88 on the Mexican SSB tax used an estimated cost per patient that was lower than the estimate used by Sanchez-Ramero et al. ,91 which may further explain the differences between these two studies in healthcare cost savings achieved by the SSB tax. Laverty et al. showed that ongoing reductions in salt intake preceding the RD slowed following its introduction, leading to increased rather than decreased healthcare costs. Based on the costs assessed, most studies showed a substantial societal economic benefit of population-level interventions aimed at promoting healthy diets. The FOPL intervention in Australia was found to be cost-effective relative to a WTP threshold of $40,861. Basto-Abreu 201988 found the Mexican SSB tax to be cost-saving as the healthcare costs saved would outweigh its implementation costs. For each dollar spent, the SSB tax was shown to save $7.28 in future healthcare costs. Sanchez-Romero 2019 only assessed healthcare cost savings and did not discuss the cost-effectiveness of the tax. Given their even larger estimates of healthcare cost savings, using their data it would be highly likely for the SSB tax to be even more cost-effective if the implementation costs were assumed to be the same as those estimated by Basto-Abreu 2019. 88 The costs of implementing the RD were not assessed, but the RD in England was not cost-effective at improving population health, as a result of the additional healthcare costs identified.

Literature search strategy and eligibility criteria

We included qualitative and mixed-methods studies (e.g. that conducted interviews, focus groups, qualitative document analyses) drawn from the systematic evidence map (see Chapter 3). To be included in this review, studies had to have assessed factors influencing a national policy process. We included papers published between 2010 and 2020, as well as studies focused on local (city, state – UK only) policies. Studies were screened by two reviewers (CK and MP).

Data extraction

We documented policy type, method and participant characteristics. We extracted information relating to reported barriers and facilitators of policy design and implementation, decision-making processes, the use of evidence or particular strategies, and any information that would help gain deeper understanding about some of the findings emerging. Data were extracted by one reviewer in EPPI-Reviewer and checked by another.

Quality appraisal

Each included study was independently appraised by two team members using the CASP Checklist for qualitative studies. Studies that received ‘no’ or ‘can’t tell’ for the two filter questions that begin CASP would be excluded. No studies were weighted or excluded based on the appraisal results. We also checked for conflict of interests in terms of industry funding.

Data synthesis

We synthesised the data using a narrative review approach to summarise findings and report on overarching themes.

Database search

A total of 91 relevant records were identified. We retrieved full text for all 91. Of these, 59 were excluded due to ‘not qualitative research’ (n = 6), ‘not about policy process’ (n = 18), ‘not national’ (n = 19), ‘media analysis’ (n = 8), ‘studies about public’s views’ (n = 1), ‘not a peer reviewed study’ (n = 1), ‘not food’ (n = 2), ‘not about a specific policy or agreement’ (n = 6) (Figure 14). Thus 30 papers met the inclusion criteria and were included in this synthesis (Table 22).

FIGURE 14.

PRISMA flow chart representing the selection process for the review of studies on policy process. Adapted from the PRISMA template by Page et al. (2021). 16

Study characteristics

Twenty-four papers studied the policy process of regulatory policies in 19 countries: Brazil, Canada, Colombia, Chile (n = 2), Denmark, Fiji (n = 3), France, French Polynesia, Hungary, Ireland, Mexico (n = 3), Nauru, Samoa (n = 2), Saudi Arabia, South Africa (n = 2), Thailand, The Philippines, UK (n = 2), USA. Three papers reported studies on the policy process of voluntary approaches on salt reduction in Argentina and Fiji, and on restrictions on marketing of unhealthy food to children in Spain.

Three papers reported studies of the policy process of PPPs on Healthy Food Partnership and FHD in Australia, on the Public Health RD in England, and on Healthy Lifestyles school-based PPPs in seven countries: China, India, South Africa, Mexico, Brazil, UK, Germany.

Nine studies reported the policy process of establishing taxes, and among these most (n = 7) focused on SSB tax [Saudi Arabia, South Africa, Mexico (n = 3), Chile, France, Fiji, Samoa, French Polynesia, Nauru]. Others reported on sugar tax (Ireland) and tax on fats (Denmark). Four countries reported on processes related to nutrition labelling (Chile, Colombia, UK) and specifically of trans-fatty acids (Brazil). Two studies focused on regulation to limit salt intake (Samoa, South Africa). Three studies reported on restrictions on marketing of unhealthy foods to children [Fiji (n = 2), The Philippines], two on restrictions on marketing of breastmilk substitutes (Samoa, Thailand), and one on regulation of television food advertising to children (USA). Finally, two studies reported on school food regulation (Hungary, Mexico) and one on mandatory standard for limiting food products and promotions in hospital retail outlets (UK). One study assessed the food and beverage industry influence on a national nutrition policy (Canada). These are summarised in Table 23.

Quality assessment of included studies

The quality assessment of included studies was done using the ten standardised criteria of the CASP tool (see Table 22). All of the included studies gave a clear statement of the aims of the research, used qualitative methodology appropriately, has an appropriate research design to address the aims of the research, used an appropriate recruitment strategy, collected data in a way that addressed the research issue, analysed the data sufficiently rigorously, and had a clear statement of findings. Nearly all had taken ethical issues into consideration. Though most studies were rated ‘unclear’ or ‘no’ against the question about whether the relationship between researcher and participants has been adequately considered, overall, the final criteria about whether the research is valuable was answered ‘yes’ for all included studies.

Conflict of interest statements and funding sources were reviewed for all studies and only one was funded by the food industry to review food industry partnership programmes.

Factors reported to shape the food and nutrition policy process

Legitimacy of commercial actors as policy actors

The dominant place of the food and beverage industry across nearly all reviewed policy process studies raises the question of what legitimacy they have in designing and implementing public health policies. These studies provide insight into these phenomena and mechanisms underpinning the policy process for improving diet, in particular how large food and beverage industries have established a relatively secure position in policy interventions to improve diets. As demonstrated in the reviewed studies, when commercial actors are able to dominate the narrative and set the agenda of public discourse, they acquire a disproportionately high ability to ‘define’ a public process and thus gain legitimacy,125 defined as a measure of the level of trust and confidence an institution should command. 125

Figure 15 summarises key mechanisms of interest as emerging from the literature. Brandon et al. (2020) discuss the food industry’s use of ‘leadership’ frames such as ‘choice giver’, ‘role model’, ‘master negotiators’. 100 For example, the authors provide the example of industry actors negotiating in a politically supportive environment (Australian government was not keen on regulation): industry actors not only negotiated but imposed an underlying assumption that pragmatism and compromise are fundamental to achieving food policy, at the expense of the best evidence. 100 Campbell et al. (2020)104 further explain that ‘corporate actors assume the role of the “choice giver”, while the dichotomized alternative is government, which is forced to occupy the opposing domain in the frame as the infantilizing remover of choice’104 This is relevant to understanding the legitimisation of industry as leaders in food/nutrition, as they ‘assume the role of the choice giver’ – this is a positive, confident act, one which demonstrates or even takes without asking the role of making a decision about what a consumer should do or not.

FIGURE 15.

Emerging higher understanding of the accepted legitimacy of food industry as global leaders.

Campbell et al. (2020)104 refer to the framing strategy of equating-equalising of logics, to level all audience; for example, emphasising ‘progress’ made on voluntary codes and positioning themselves as ‘industry leaders’; demanding a seat at the policy-making table, appealing to historical roots and reminding how they are firmly established in the fabric of society. 104 Campbell et al. (2020)104 also report how the food industry will present itself as fully committed to health problems, positioning themselves as concerned role models within the community, which contribute to supporting the establishment of their legitimacy as leaders. 104

Fuster et al. (2020) report that in Chile and Mexico the food and drinks industry carried out a failed legal challenge to the SSB tax constitutionality, while also appealing to the public, by positioning themselves as defenders of consumer freedom; this contributes to building an image of the industry as champions of consumer rights. Finally, Mialon (2020)114 report the use of key platforms such as Codex used by the industry to gain access to decision-making, in the case of FOPL in Colombia.

Structural changes through regulation appears the most effective approach to improve the food environment

A first implication relates to the finding that from the available evidence reviewed, policies appear most effective when targeting the structural rather than informational level and made mandatory. The findings align in large part with key existing recommendations for effective and cost-effective approaches including, notably the WHO Best Buys (2017),126 which advise to reduce salt intake via the reformulation of food products, eliminate industrial trans-fats by banning their use in the food chain, replace trans and saturated fats with unsaturated fats, and reduce sugar consumption through effective taxation on SSB. It also suggests that countries work to reduce portion and package size and implement FOPL. As reported by Francesco Branca, the WHO direction of Nutrition for Health and Development, in his 2019 BMJ article,127 a large proportion of countries have not implemented evidence-based ‘structural actions’ to effectively improve food environments, including implementing a ban on industrially produced trans-fats, taxing SSBs and unhealthy foods, and mandating simplified, interpretive FOP nutrition labelling.

In England, much of the government’s response to unhealthy diets has focused on obesity prevention and reduction. 128 The 2016 Childhood Obesity Plan proposed a set of actions, including the introduction of a SDIL; the reduction by 20% of sugar in products that contribute most to intakes of children up to 18 years by 2020; the reduction of calories in a range of products contributing to children’s intakes; reducing salt in products by achieving the 2017 salt targets; providing clearer food labelling; making healthy options available in the public sector; making school meals healthier; and developing a new framework by updating the Nutrient Profile Model. In 2020, the plan ‘Tackling obesity: empowering adults and children to live healthier lives’129 mentioned the need for mandating calorie menu labelling in large out-of-home food caterers; for using legislation to control volume promotions of HFSS foods, for example buy one get one free, as well as their placement in locations that trigger unintended purchases, in retail settings that sell food and drink both in-store and online in England; and for both expanding the ban of advertising of foods HFSS on TV and exploring options online.

Many of these have been rolled out. For instance, the SDIL came into force in April 2018, imposing a two-tiered levy on manufacturers of soft drinks who did not reduce the amount of sugar in their beverages to below one of two stated levels,130 aiming to reduce NCDs associated with excess sugar consumption. 131 The SDIL has led some of the soft drinks industry to reformulate products to below one or both of the thresholds. 132,133 Early estimates suggest that the SDIL will contribute to improving health and decreasing health inequalities. 134,135 The regulations restricting promotions of HFSS products by location and price were due to come into force in October 2022. Those on advertising less healthy products on TV before 9 p.m. and online are expected to be introduced in 2023, pending Parliamentary procedure. 136

Voluntary approaches and public–private partnerships appear not to be the most effective approach to improve the food environment

A second implication for public health policy relates to the finding that from the available evidence reviewed, voluntary approaches and PPPs do not appear to be effective at improving food environments for healthier diets. The findings align with earlier studies14,137–140 and can be explained by the understanding of the functioning of voluntary approaches and PPPs (both from the review and past studies), namely that, commitments or pledges tend not to be based on most effective interventions to improve diets; there is usually a major focus on education and information/behaviour modification rather than on structural (and more effective) changes; self-designed commitments are narrow, vague or loosely binding – not ‘SMART’; voluntary participation of companies in self-regulation schemes is often itself voluntary, which results in an unequal playing field (and even so, some studies included in Chapter 6 found that their outcomes were worse than that of NPs); reporting and monitoring of implementation is often poor; there is limited added value of signing up/pledging apart from promoting a positive corporate image; partners often commit to actions already undertaken; and failure to comply with pledges is not well enforced or penalised.

In England, voluntary approaches are part of the strategy to improve diets. Current initiatives as described below would benefit from taking on board the lessons from the study if we are meaningfully and sustainably to improve food environments for healthier diets.

Public Health England – the government’s public health arms’ length body in existence at the time – was commissioned by the Department of Health to work on sugar reduction in light of a Scientific Advisory Committee on Nutrition report. 134 Their role of encouraging industry compliance to reduce sugar and salt content through regulation and with voluntary approaches reported modest progress across a range of food categories and sectors. 141 The Prevention Green Paper (2019) made further commitments to reduce the population’s salt intakes by publishing revised salt reduction targets in 2020 for industry, to be achieve by mid-2023, ‘keeping all options open if a voluntary approach does not demonstrate enough progress by 2024’. 142

Currently, the regulation of online advertising of HFSS products is self-regulatory through the Advertising Standards Agency,143 and as such industry possesses extensive rule-setting power within the regulatory process. A proposal for the total online advertising ban policy proposed as part of the Government’s new approach to address high rates of obesity in the UK included the UK Government pledge to hold a consultation on the total online advertising ban policy, primarily aimed at protecting children from such advertising, which closed on 22 December 2020. The results of the consultation are yet to be announced, and the proposed ban may not be implemented. 144

Expanding geographical representation of the evidence

A third implication for public health policy is the need to redress the imbalance in policy evaluation evidence across regions of the world, with this study’s stark finding that 81% of mapped publications (see Chapter 3) focused on only 12 countries (USA, UK, Australia, Canada, Mexico, Brazil, Chile, France, Spain, Denmark, New Zealand and South Africa), and 30% on the USA. To the best of our knowledge, there are no evaluations for several countries that have implemented such policies. For example, we are aware that in the Caribbean, Dominica has introduced an excise tax on food and drinks with a high sugar content in 2015. In the Middle East, Iran banned soft drinks broadcast advertising in 2004 and has a salt reduction strategy. In South Asia, Sri Lanka introduced regulatory traffic lights labelling in 2016. In North Africa, Morocco implemented a SSB tax in 2019, revoked it, and reintroduced it in 2020. In Sub-Saharan Africa, soft drinks and unhealthy snacks have been banned in schools in Mauritius since 2009. 145,146 Addressing these gaps will require examining the reasons for the lack of all types of evaluations (or published evaluations) in countries which clearly have implemented policies promoting healthy food environments, and ensuring that future policy analyses capture an equitable geographical spread. Solutions are likely to include increasing both research and publication capacity in the least documented world regions as well as in many other countries. This might also involve setting up cohorts and monitoring systems and should aim at including cost-effectiveness studies within policy evaluations more consistently. Evidence syntheses should also analyse findings from real-world evaluations separately from those of modelling studies, and where appropriate retrieve evidence from all regions of the world.

Understanding the barriers to policy evaluation is essential, and in particular clarifying whether it is a lack of resources, or a lack of contextual evidence to ensure appropriate measures for the local geographical context. Isaranuwatchai et al. 2020 assessed the WHO’s list of best buy interventions and concluded that best buys evaluated in one setting may not be cost-effective in others, further highlighting the importance of the finding that the available evidence is not globally representative. 147 Isaranuwatchai et al. 2020 emphasise that low- and middle-income countries (LMICs) have greater pressures to identify and prioritise more cost-effective and equitable NCD interventions, in the face of even more limited resources. 147

Capturing dimensions of health equity as policy outcomes

A fourth implication for public health policy relates to the low priority accorded to health equity as a policy outcome. This is despite the measurement and evaluation of equity having been prioritised as a key principle of action by the Commission on Social Determinants of Health of the WHO in 2008,148 and employing a quite generous interpretation of the PROGRESS-Plus framework. While age, SES at individual level, and education are relatively commonly considered, the remaining dimensions are much more neglected, especially disability, religion and culture, and occupation at individual level, leaving some population groups largely ignored. Social capital was also seldom assessed but we recognise that it is more challenging to operationalise and capture. Some of the dimensions mentioned above can also be difficult to apply in studies of environmental features. However, the evidence map shows that it is possible to consider at least some of them. Guidance with specific examples for different types of ‘participants’ or measurement units might help promote this practice further. Funders and journal editors could require researchers more systematically to integrate an equity focus on projects beyond the description of study participants’ characteristics, and with a particular emphasis on the neglected dimensions where possible. In the overview of reviews, none of the findings on equity considered the epidemiology of diseases to assess impact on health outcomes or whether the gap between groups had reduced or increased through time. Where possible, considering these would help to fill the gap.

Need for a systems approach across policies to improve food environments

A systems approach is a way of conceptualising and thinking through a problem. It can lead to a practical illustration of the complexity of pathways of impact, how different factors can interact, leverage points for change, which may not appear in linear logic models or traditional theories of change, and potential unintended consequences of an intervention. 149 Ideally a systemic theory of change helps to identify the wider drivers of unhealthy diet and prompts a strategy that effectively integrates the identified leverage points. 149

Unhealthy diets, and diet-related outcomes, are complex, challenging to tackle and seemingly intractable problems requiring an approach that fully engages with and responds to their complexity. However, part of the challenge in finding effective actions is that solutions are conceptualised and planned using traditional theories of change150 rather than using a systems approach. Most theories of change, as illustrated in Figure 16, suffer from pitfalls, including linear thinking, simple chains of cause and effect, and relying on a fixed plan.

As demonstrated across the research, the approach outlined in the theory of change (see Figure 16) only partially works. This is because it rests on a series of important assumptions, including that food industry actors are invested in population health; food industry actors are supportive of voluntary agreements and compliant; PPPs are evidence-driven and designed to be equitably managed, and independently monitored.

Conclusions

The study contributes to the growing field of research on the commercial determinants of health, defined as adverse health impacts attributable to commercial activities and corporate political strategies employed by unhealthy commodity industries to promote products, which can damage health. 151–153 The food and drinks industry employ a range of non-marketing strategies to promote the sale and consumption of their products, including shaping scientific evidence, building constituencies beyond their core business, using litigation, lobbying of decision-makers, and publicly criticising policy process, in the interest of protecting markets. These strategies have been known to increase the chance of ‘regulatory chill’ – delays, weakening, or abandonment of planned policies154 – and are manifested in various ways including a dampening effect on specific measures once government is aware of an existing threat to regulation;155 the abandonment or weakening of proposed regulation to avoid future disputes based around past challenges to similar regulation; and the deliberate instilling of apprehension in other countries trying to implement similar regulation (‘cross-border chill’). 155

This study has addressed another corporate political strategy, policy substitution, referring to the industry promotion of voluntary and partnership approaches. Chapters 5 and 6 indicate quite clearly that these are not the most effective approaches to improve the food environment, and the policy process study (see Chapter 8) finds that the concept of partnerships as a preferred policy construct is often presented as the most effective to meet public health objectives; however this argument is reported to be seldom informed by a review of international evidence of best practice in supporting healthy diets, but rather compromise and pragmatism. In fact, the evidence map (see Chapter 3) suggests that few PPPs for promoting healthy food environment have been assessed, and that most of the evidence available is about two PPPs, one in England and one in Australia.

The main proposition of this work, based on the extensive evidence synthesis, is that an improvement of diets in England requires a complex system approach to maximise the potential of a strong portfolio of food and nutrition policies by understanding the more complex drivers and competing interests.

Figure 17 exemplifies this by illustrating some of the factors highlighted by the study that could both dampen and enable the effectiveness of a portfolio of policy interventions, and taking a systems approach can provide an initial illustration of how these important factors interconnect and influence each other.

FIGURE 17.

An illustration of how this evidence synthesis challenges the current theory of change.

The evidence synthesis in Chapter 8 highlighted factors working against the above theory of change, including practical considerations often around policy implementation and the generation of context-specific evidence; differing ideological positions; capacity constraints; the role of evidence; lack of key stakeholder engagement, and the deployment by the food and beverage industry of frames and framing strategies to ultimately oppose a policy process. It also highlighted mechanisms underpinning the policy process for improving diet, in particular how large food and beverage industries have become legitimate actors in policy interventions to improve diets, such as the industry’s deployment of leadership frames (‘master negotiators’), of ‘history’ frames to emphasise historical roots in community, the promotion of partnership as a preferred policy construct, and the use of policy entrepreneurial skills.

Drawing on qualitative system mapping methods and on past work expressing mixed data as system illustrations (e.g. systems evaluations of PPPs;156 and the formulation of concrete methodological implications of a complexity perspective when conducting systematic reviews and guideline development157) we revisit the theory of change, visually adding findings of the evidence synthesis to illustrate the wider picture. Thus Figure 17 displays certain factors of the wider system, as emerging from the findings, and how they work together to either improve (in blue), or reinforce (in red), unhealthy diets in England.

As noted above, the theory of change outlined in Figure 17 rests on the assumptions that food industry actors can participate in, and are supportive of, voluntary agreements to reformulate their products to make them healthier, and that they are able and willing to respond to regulations to make their products healthier or behave in the interest of public health.

Need for guidance on appraising risk of bias and quality of non-clinical studies

Regarding study designs and quality or risk-of-bias appraisals, the systematic review on PPPs (see Chapter 5) has highlighted the lack of employment of comparison groups. The aggregation of data for participants and NPs in PPPs together suggests that identifying a comparison group can be possible in several cases. Furthermore, in both the latter systematic review and that on voluntary commitments by private actors (see Chapter 6), we noted a lack of reporting of information that is essential for appraising study quality or risk of bias, including information on non-responders, the presence of missing data and how it was dealt with, and the number of people collecting or verifying data independently. Similar observations were noted in most systematic reviews, with data on before and after the intervention, effect sizes and precision estimates often missing. Such information should be reported systematically and highlighted when missing in the primary studies. There are already checklists guiding study authors to do so in both primary studies and evidence syntheses. Wider systemic changes in academia with meaningful incentives might be needed to encourage reviewers to examine these attributes more systematically. Moreover, this project highlights the need to revisit the notion of lack of implementation as a source of bias from a policy analysis perspective. Lack of implementation for a policy is not necessarily or simply a bias; it may also be an outcome. It could be a mistake or might have been deliberately planned not to happen. For example, the systematic reviews on the effectiveness of PPPs and voluntary approaches by private actors (see Chapters 5 and 6) found that companies participating in PPPs or pledges continued to advertise, promote or offer unhealthy foods, including to children. The policy document analyses in the review of PPPs also suggest that some of the policies had very low ambitions, included loopholes, or were not monitored by the companies themselves, suggesting that they were more likely to be intended to promote a ‘health halo’75 and a favourable corporate image than to improve population diet.

Furthermore, this project highlights a need for guidance for appraising the risk of bias or study quality of studies measuring the effects of interventions on environments (rather than effects on human) and for critical appraisal of document analyses (including policy and media sources). These types of studies represent 57% of effectiveness studies in the systematic evidence map (see Chapter 3).

Lastly, as explained in Chapter 2, a wider discussion is needed about the suitability of the GRADE framework for assessing certainty in the evidence of policy research. It should consider the feasibility of different types of studies, recognise the contribution of document and media analyses, and differentiate more effectively the level of certainty between good and weak observational studies. This is aligned with concerns raised by Boon et al. (2021)158 for some public health topics: they have noted the presence of ‘discordant recommendations’ in which methodologically the level of confidence is rated low or very low, yet there is a case for making strong recommendations given the context and evidence.

Need for reporting policy characteristics in evaluations

This project also highlights a need for both reporting and considering policy characteristics in evaluations more consistently. Documenting these was not as straightforward as anticipated in both primary studies and evidence syntheses due to lack of reporting of information. Yet, these characteristics influenced the interpretation of results. They could also inform study quality or risk of bias. Information that we felt was missing includes the following:

1. Policy names: several policies were not named or had their name changed through time without it being reported in the evaluation.

2. Policy content and context: some publications reported different aspects of the same policy, for example, in Canada some said evaluating trans-fat policies that included both regulatory labelling and voluntary limits, while others only mentioned the limits. We have extracted the latter at face value, but some studies might have missed the wider picture. Also, stating that a policy is a salt reduction programme is not enough: it can consist of awareness campaigns, targets, labelling, regulations in specific settings, or a combination of these.

3. Governance and policy process including the actors involved: for instance, the development of some regulatory and voluntary approaches by the public sector might have involved private actors and should thus rather be called PPPs.

4. Policy and evaluation timeline: the dates of data collection and of key policy changes inform reviewers’ understanding of the policy status at the time of data collection, for example decisions about when baseline data were collected: whether it was collected before a policy was debated, or announced, between the announcement and implementation deadline, just before that deadline, or after, and for how long.

5. Wider context including concurrent initiatives: it is now common to have several initiatives promoting healthy eating occurring at the same time. For instance, from the UK and English governments alone, there are the SDIL, salt and sugar reduction programmes, FOPL, an advertising control code, etc. While it is impossible to assess them all at once, and study authors are not expected to extensively describe them, mentioning the most relevant concurrent interventions could help interpreting results.

6. Policy and evaluation levels: sometimes these did not match. For example, some state policies were evaluated using data aggregated at the national level without considering or mentioning that the policy was implemented in a single state.

Need to recognise the value of studies of environment features and policy documents

This project showcases the contribution of non-human outcomes for evaluating policies promoting healthy environments. The primacy often accorded by evaluations to health or behavioural outcomes of interventions is frequently inappropriate and unwarranted. First, many evaluations take place over too short a time from for measurable changes in outcomes such as obesity, CVD or mortality to manifest themselves. 159 Second, these policies act on the environment rather than directly on humans, alongside other interventions taking place concurrently. This often makes the direct attribution of effects on humans for a specific policy infeasible. The impact of a FOPL scheme or an advertising control policy is highly unlikely to lead to measurable changes in weight status over the course of an evaluation, but it may nevertheless generate intermediate outcomes such as alterations to packaging or reformulation of products.

Moreover, lack of implementation of a policy is not necessarily or simply a bias; from a policy analysis perspective it may also be an outcome. It could be a mistake or might have been deliberately planned not to happen. For example, the systematic reviews on the effectiveness of PPPs and voluntary approaches by private actors (see Chapters 5 and 6) found that companies participating in PPPs or pledges continued to advertise, promote or offer unhealthy foods, including to children. The policy document analyses in the review of PPPs also suggest that some of the policies had very low ambitions, included loopholes, or were not monitored by the companies themselves, suggesting that they were more likely to be intended to promote a ‘health halo’75 and a favourable corporate image than to improve population diet. Other examples of studies that did not involve humans yet provide useful information on policy effectiveness include a governmental evaluation in Canada showing that only 14% of food categories met the national voluntary salt targets after 4 years. Not only was no progress observed for nearly half of the food categories, but sodium content had increased in some of them. 160 A comparison of marketing regulations for breast milk substitutes in Cambodia, Indonesia, Myanmar, Thailand and Vietnam concluded that none followed the WHO recommendation of applying them to children aged up to 36 months. 161 A study by Ejlerskov et al. 162 on voluntary checkout policies by 14 supermarket chains in the UK noted that the six chains with ‘clear and consistent’ policies offered a smaller median proportion of unhealthy foods at their checkouts and that all complied to their policy compared to the six chains with ‘vague or inconsistent’ policies and the two chains without a policy. In the UK again, an analysis of food industry pledges in the Public Health RD showed that none included the most effective strategies to improve diet. 56

We suggest attributing a greater value to the contribution of studies of environment features and analysis of policy documents for informing decisions about effectiveness, and to acknowledge this in funding programmes. Such studies also have the advantage of generally requiring less time, resources and advanced evaluation techniques, which makes them more accessible worldwide. This could also be reflected in logic models or systems frameworks more systematically, for instance with the analysis of the policy itself and implementation, to effects on the environment, human behaviours, and human health.

Studies on the role of corporate actors in food policy

The role and interests of policy stakeholders, and the accepted legitimacy of corporate actors in food policy, should be critically interrogated with healthy diets for the population as the first priority.

Evaluations of real-world policies equitably across geographic regions

First, future evaluations are needed across all geographic regions of real-world policies to improve food environments equitably. This will require greater global equity in access to research funding. This also includes the need for more cost-effectiveness analyses worldwide.

Capture equity dimensions in policy evaluations

Second, equity dimensions should be adequately built into policy evaluations. Moreover, research funding could be directed to study how policy interventions could appropriately and systematically capture PROGRESS-Plus equity dimensions.

Guideline development for quality/risk of bias and certainty in the evidence of policy evaluations

Third, guidelines should be developed to (1) help with systematically reporting policy characteristics in policy evaluations, (2) consider policy characteristics in the interpretation of results and risk-of-bias appraisal, (3) appraise the risk of bias or study quality of studies measuring environmental features (rather than human outcomes) and using document analyses (including policy and media sources), (4) discussing whether the GRADE framework for assessing certainty in the evidence should consider the feasibility of different types of studies, recognise the contribution of document and media analyses, and differentiate more effectively the level of certainty between good and weak observational studies for policy research.

Relevant to the reviewed literature

This study did not include participants as it was an evidence synthesis based on a large complex systematic review of the literature. However, most of the systematic reviews included the reporting of equity domains in the evidence, as per the PROGRESS-Plus equity dimensions. The systematic evidence map (see Chapter 3) reports that of the nearly 400 publications assessing policy effectiveness, only 50% assessed at least one equity domain: age was the most assessed (30%), followed by education (16%, although these mainly related to school characteristics), SES at the individual level (15%), gender/sex (13%), race and culture (11%) and place (8%), whereas only 11 publications considered occupation, and one or two examined religion, social capital and disability each. The Overview of Reviews (see Chapter 4) reported findings by PROGRESS-Plus categories and found a lack of reporting of outcomes in systematic reviews for specific population groups: overall, evidence on specific population groups is patchy, incomplete, mainly inconsistent, and largely relies on single studies (although some of these had large samples) rather than aggregated bodies of evidence. This limitation in the research is reflected in the recommendations.