Countries and donors increasingly recognise the benefits of evaluating public programmes that could increase access to safe and nutritious foods for the poor. However, low- and middle-income countries face three main challenges to evaluation. First is scale: many important programmes that are good candidates for evaluation are implemented nationally. Second is perception: governments want to emphasise the effectiveness of programmes they support, but evaluation can reveal shortcomings, with potential political repercussions. Third is capacity: evaluation planning and coordination can be a struggle in contexts characterised by limited specialised skills and expertise. Nonetheless, evaluation is feasible, as one recent study shows, and has potentially strong benefits for improving large-scale nutrition interventions.
In a landmark study, Bougma and colleagues (2018) took advantage of a gradual rollout of salt iodisation to examine the impact of iodised salt on young children’s mental development in Ethiopia. Their study – a community-based, cluster randomized effectiveness trial – showed that iodised salt had an effect on children’s mental development scores at a time when salt iodisation had not yet been mandated in Ethiopia. Iodised salt improved verbal (receptive language) and non-verbal (cognitive and fine motor) scores, as well as the composite mental development score. Effect sizes were somewhat small. This result provides a strong evidence that even brief exposure to iodised salt leads to a meaningful improvement in young children’s mental development.
The study fills an important knowledge gap. No study using a randomized design had previously been published on the effects of salt iodisation on the mental development of children under 2 years. Previous evidence suffered from methodological limitations: high attrition, uncontrolled confounders and small sample size. Other studies on the effects of salt iodisation on children’s mental development focused on children whose pregnant mothers received iodine supplements.
In addition to producing strong evidence, Bougma and colleagues’ study suggests three lessons about evaluating nutrition interventions in low- and middle-income countries.
Lesson 1: Important nutrition interventions can be evaluated
One of the main challenges of evaluating nationally implemented nutrition interventions is finding appropriate comparison groups. However, financial and administrative constraints in developing countries often limit the possibility of introducing a new intervention in all communities, so most new interventions are rolled out gradually. In 2011, when the Ethiopian government passed universal salt iodisation legislation, it was not feasible to serve all markets.
The need to phase in a programme geographically provides an excellent opportunity to evaluate the programme. Randomisation could be the fairest way of helping to choose the order of phase-in. Bougma and colleagues used this approach, randomly assigning early market introduction of iodised salt in Ethiopia. A similar approach was used to evaluate the impact of the PROGRESA programme on child health in Mexico.
Bags of salt on display in a market in Africa. © GAIN
Lesson 2: It is possible to evaluate market-based interventions supported by governments
Governments and donors are increasingly recognising the importance of evaluating public programmes. However, the foundation of the vast majority of policies is still ideology and experience. It takes courage for a government to allow such an evaluation, and serious coordination of efforts to make the evaluation work. A gap exists between the desire to assess the effectiveness of public programmes and the political economy of evaluation. Governments are sometimes reluctant to have their programmes (or the programmes they support) evaluated, because a lack of an effect can be perceived as a failure, with negative political repercussions. One reason for this perception is misunderstanding of what evaluation can deliver. Evaluation is not considered as a tool for improving public programmes. In addition, because evaluation results are not always available for short-term decisions, they can appear difficult to translate into immediate policy actions. Finally, a successful evaluation requires advance planning and strong coordination to select an external evaluator, find funding, and possibly adjust the rollout of a programme to accommodate the evaluation.
All of these conditions can be challenging in contexts characterised by limited funding and limited specialized skills and expertise. Nonetheless, this study shows it is possible to evaluate market-based interventions supported by a government – in this case, the Ethiopian government.
Lesson 3: Measurable impact for complex outcomes is achievable with a rigorous design
Bougma and colleagues’ study shows that measurable impact for complex outcomes, such as child development, is achievable with a rigorous design, even with a minimal period of differential exposure. In this study, children in the intervention group received about 4 to 6 months more exposure to iodised salt than controls. This period of differential exposure could be insufficient to observe measurable effect, because child development represents a dynamic combination of biological and environmental factors.
Two factors can explain the measurable effect observed in this study. First, the potential for impact was high at the launch of national iodisation efforts in 2011. At baseline in 2010, only 5.4% of households tested were using iodised salt. Thus, a significant increase of the availability of iodised salt during a short period produced measurable impact. Second, the strength of the evaluation design limited the influence of omitted variables that can bias the result of an evaluation. For example, mothers with a strong preference for early breastfeeding could be more likely to use iodised salt. A strong design, such as the one used in this study, limits the influence of this preference, which can render a measurable impact difficult to observe, especially with a short period of differential exposure.
Evaluating important nutrition interventions planned for scale at the national level is feasible
The case of salt iodisation in Ethiopia is an illustration. The evaluation of these programmes requires (1) taking advantage of gradual rollout of programmes; (2) an understanding of factors in the political economy that could hinder adoption of evaluation findings and strategies for overcome barriers; and (3) assessing the potential impact of these programmes. Evaluation of nutrition interventions can provide improved knowledge about the types of programmes that work in increasing access to safe and nutritious foods for the poor.