Residents of the camp of the Internally Displaced Persons (IDPs) receive food rations distributed by the World Food Programme (WFP) in collaboration with an international non-governmental organization, CARITAS. North Kivu, Democratic Republic of the Congo. Photo: Flickr/UN Photo/Marie Frechon.
According to the United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA) and the Financial Tracking Service (FTS), the number of people in need of humanitarian aid in 2017 rose to 141.1 million and they were located in 37 countries. The Global Humanitarian Appeal stood at nearly $13 billion as of November 2017, which represented 58 per cent of the total fund target set for humanitarian assistance.[1] The gap between the appeal target and funds raised, however, has been widening in recent years (Figure 1), as has the number of people affected by disasters—both natural and human-induced (Figure 2).
The least funded countries towards their targets in 2017 were Senegal, which had 18 per cent of its target funded, primarily for projects in the food security and nutrition sector; and Peru, which was 18.5 per cent funded, primarily for water and sanitation projects. The best covered country in terms of funding was Afghanistan, where 70 per cent of funding requirements were met by donors, which included 100 per cent of the funding needs identified for food security and agriculture. Gross domestic product in these countries lags behind the world’s average: in Afghanistan it was just US$ 561 GDP per capita in 2016, in Senegal the figure was US$ 958 and in Peru US$ 6045.[2]
When disaster strikes, most efforts are directed at coordination and the first essential relief work (providing water, food, medicine and shelter). In the first 72 hours following the disaster, UNOCHA sends skilled first response staff to start coordinating assistance efforts between the government, and local and international actors. Many disasters are characterized as ‘sudden onset’, meaning that they abruptly disrupt basic services and infrastructure and must be addressed quickly by a multiplicity of actors in order to save lives. However, even in such situations, when a complex and immediate response is required, evaluations of humanitarian assistance are important.
Figure 1: Trends in response plan requirements/appeal funding and total raised. Notes: The percentage labels shown in each bar represent the global appeal coverage for each year. Amounts shown for 2017 are figures for the year to November. Source: data from FTS.
Delivery of humanitarian response in complex environments creates unique challenges to evaluation, particularly impact evaluation which requires additional rigour for causal inference. The challenges are well-known to practitioners and evaluators (see, for example, the ALNAP's Evaluation Humanitarian Action Guide, in particular chapter 18). In addition to the monitoring and evaluation efforts already in place in many international organizations, which track whether aid reaches the intended beneficiaries, the methodology set out below—adapted from evaluations used to assess development programmes—might help to answer similar questions in the humanitarian aid sphere:
What is the magnitude of the impact of aid?
What are the impacts of aid provided by a particular agency?
How sustainable are these impacts?
What amount of assistance is needed and at what frequency should it be delivered?
Did certain groups within the affected population benefit more than others? If so, in what ways and to what extent were their livelihoods improved?
Were there any unintended or negative consequences?
Could aid be delivered in a more cost-effective way?
Since one of the international criticisms of humanitarian assistance is its lack ofsustainability, addressing these questions in the recovery and resilience phases, which follow the relief phase in the three to six months after the disaster strikes, is especially important. The earthquake in Haiti in 2010 is one example of aid that failed to prevent an outbreak of cholera, which affected 6 per cent of the population over two years.[3] In Nepal, two years after the 2015 disaster, 70 per cent of the affected population are still living in temporary shelters.[4] Recurring drought in Somalia continues to bring hunger to vulnerable populations despite international efforts to tackle the hunger crisis year after year.
Figure 2: Number of disasters and people affected, 1970–2016. Source: EM-DAT Emergency Events Database, Université Catholique de Louvain—CRED.
Consider, for example, the provision of supplementary nutrition to impoverished households with young children during a food crisis. There are several ways to provide this nutrition: food delivery, food vouchers or cash. There are also several channels of distribution, for example by distributing the assistance to the head of household or directly to the mother of the young children. What is the most effective way to provide the nutrition assistance given the particular country context? If providing assistance to the head of household leads to the optimal distribution of resources across all household members, this is the optimal solution. If, however, in contexts where the head of household is nearly always male, it leads to an unequal distribution of resources by limiting access by young children and women, then providing it to mothers may benefit the disadvantaged members of the household. Providing assistance directly to a woman rather than a man, however, could put a woman at risk in the most male-dominated societies. Famine is not always about there not being enough food. Providing food vouchers or cash in the absence of functioning food markets, however, will not be an effective solution. An impact evaluation helps to answer some of these questions.
What is the magnitude of the impact of aid? To be able to understand the process of how joint humanitarian efforts affect selected nutritional and health outcomes, nutritional, health and other data should be collected, ideally both before and after the assistance is distributed at the individual, household and community levels.
What are the impacts of aid provided by a particular agency? To answer this question, it is important to define an identification strategy that is able to attribute impact to a particular intervention. First, it is desirable to have a comparable ‘sample’ to which it was not possible to provide an intervention, or which received an intervention at a later stage, to compare with the intervention group that received assistance. Having a control group helps to eliminate parallel ‘shocks’ that may influence both groups at the same time and which are not due to the intervention. To guarantee fair allocation and comparability, both samples must be selected on a random basis before the intervention—in a randomized control trial. In a humanitarian context, however, it is often the case that large areas or a large number of the population are affected (a high co-variability problem), or there would be ethical considerations around the agency delaying assistance any longer than necessary. Therefore, withholding assistance from a certain group of the population could not be considered. However, when resources are limited, or the intervention has potential side-effects as in public health programmes,[5] random selection could still be considered. The impact is the difference between the intervention group and the comparison group before and after the programme (see Table 1).
Table 1: Estimating impact
Another solution would be to start evaluating the programme after the intervention. In such cases, the evaluation team must rely on existing data available on the community before the disaster.
How sustainable are these impacts? In this case, researchers return to collect additional waves of data at a later stage (medium to long term). The nature of the intervention defines the timeline of the evaluation. In the case of nutritional programmes, the research team may come back after six months, one year or even longer, depending on the scale of the programme and the availability of resources.
What amount of assistance is needed and at what frequency should it be delivered? An impact evaluation team should perform rigorous statistical analyses using the data collected. It is therefore important to ensure that the data contains full information on the outcome indicators that the implementation team was aiming to improve, such as health indicators, the dietary diversity of all household members, and so on. Questions specific to the intervention should be formulated—and if possible piloted in advance—in a such a way that it is possible to learn detailed lessons about the impact of the programme.
Did certain groups within the affected population become better off than others? If so, in what ways and to what extent were their livelihoods improved? The most vulnerable are usually hit the hardest by disasters. Together with the implementation team, the impact evaluation team should help to identify the intervention target group. During humanitarian emergencies, it is important that data on displacement and an existing census or similar data are accessible to the research team.
Were there any unintended or negative consequences? When providing household-targeted interventions, it is essential to understand how resources are redistributed among all members of the households and whether certain groups have restricted access. Therefore, even though this requires more financial resources, surveyingall household members is essential.
Could aid be delivered in a more cost-effective way? Impact evaluations using randomized control trials[6][7] can and have helped to identify the type of assistance—providing in-kind food, food vouchers or cash—that is most efficient or appropriate—in terms of both cost and effectiveness—for the target population and the conditions on the ground. The implementation and research team should identify the needs of the population and the context in which the intervention will take place before the intervention starts, using interviews during field visits and desk research.
To summarize, all the nuances of the intervention must be considered in order to be able to design an impact evaluation. Therefore, the implementation and impact evaluation teams should synergize to guarantee that all the available information is accessible to both.
The main advantage of a rigorous impact evaluation is that it contains a set of components that have been tested in many development programmes and that in combination can help to tease out unbiased impacts. Some methodologies—such as randomized control trials—are flexible enough to be adjusted to the humanitarian context and complement the existing monitoring and evaluation methodologies of humanitarian agencies. The key recommendations to humanitarian agencies are therefore:
Learn about where disasters may unfold and where assistance may be most required. Many natural disasters happen in the same geographic areas. Collect comprehensive data on the populations and local environments likely to be affected that will be useful in times of disaster.
Prepare and design impact evaluations in advance that are specific to the types of intervention the agency conducts.
As the climate changes, it is highly likely that the number and scale of disasters will grow in the coming years. The need for humanitarian assistance is therefore also likely to increase. This creates an urgency to ensure that humanitarian aid is spent in the most effective and efficient way.
[5] Lorenc, T. and Oliver, K. J., ‘Adverse effects of public health interventions: a conceptual framework’, Journal of Epidemiology and Community Health, vol. 68, no 3 (Mar. 2014), pp. 288–290.
[6] Aker, J.C., 2013. Cash or Coupons? Testing the Impacts of Cash versus Vouchers in the Democratic Republic of Congo. SSRN Electronic Journal, pp.1–51.
[7] Hidrobo, M. et al., Impact evaluation of cash, food vouchers, and food transfers among Colombian refugees and poor Ecuadorians in Carchi and Sucumbíos. documents.wfp.org.
According to the United Nations Office for the Coordination of Humanitarian Affairs (UNOCHA) and the Financial Tracking Service (FTS), the number of people in need of humanitarian aid in 2017 rose to 141.1 million and they were located in 37 countries. The Global Humanitarian Appeal stood at nearly $13 billion as of November 2017, which represented 58 per cent of the total fund target set for humanitarian assistance.[1] The gap between the appeal target and funds raised, however, has been widening in recent years (Figure 1), as has the number of people affected by disasters—both natural and human-induced (Figure 2).
The least funded countries towards their targets in 2017 were Senegal, which had 18 per cent of its target funded, primarily for projects in the food security and nutrition sector; and Peru, which was 18.5 per cent funded, primarily for water and sanitation projects. The best covered country in terms of funding was Afghanistan, where 70 per cent of funding requirements were met by donors, which included 100 per cent of the funding needs identified for food security and agriculture. Gross domestic product in these countries lags behind the world’s average: in Afghanistan it was just US$ 561 GDP per capita in 2016, in Senegal the figure was US$ 958 and in Peru US$ 6045.[2]
When disaster strikes, most efforts are directed at coordination and the first essential relief work (providing water, food, medicine and shelter). In the first 72 hours following the disaster, UNOCHA sends skilled first response staff to start coordinating assistance efforts between the government, and local and international actors. Many disasters are characterized as ‘sudden onset’, meaning that they abruptly disrupt basic services and infrastructure and must be addressed quickly by a multiplicity of actors in order to save lives. However, even in such situations, when a complex and immediate response is required, evaluations of humanitarian assistance are important.
Delivery of humanitarian response in complex environments creates unique challenges to evaluation, particularly impact evaluation which requires additional rigour for causal inference. The challenges are well-known to practitioners and evaluators (see, for example, the ALNAP's Evaluation Humanitarian Action Guide, in particular chapter 18). In addition to the monitoring and evaluation efforts already in place in many international organizations, which track whether aid reaches the intended beneficiaries, the methodology set out below—adapted from evaluations used to assess development programmes—might help to answer similar questions in the humanitarian aid sphere:
Since one of the international criticisms of humanitarian assistance is its lack of sustainability, addressing these questions in the recovery and resilience phases, which follow the relief phase in the three to six months after the disaster strikes, is especially important. The earthquake in Haiti in 2010 is one example of aid that failed to prevent an outbreak of cholera, which affected 6 per cent of the population over two years.[3] In Nepal, two years after the 2015 disaster, 70 per cent of the affected population are still living in temporary shelters.[4] Recurring drought in Somalia continues to bring hunger to vulnerable populations despite international efforts to tackle the hunger crisis year after year.
Consider, for example, the provision of supplementary nutrition to impoverished households with young children during a food crisis. There are several ways to provide this nutrition: food delivery, food vouchers or cash. There are also several channels of distribution, for example by distributing the assistance to the head of household or directly to the mother of the young children. What is the most effective way to provide the nutrition assistance given the particular country context? If providing assistance to the head of household leads to the optimal distribution of resources across all household members, this is the optimal solution. If, however, in contexts where the head of household is nearly always male, it leads to an unequal distribution of resources by limiting access by young children and women, then providing it to mothers may benefit the disadvantaged members of the household. Providing assistance directly to a woman rather than a man, however, could put a woman at risk in the most male-dominated societies. Famine is not always about there not being enough food. Providing food vouchers or cash in the absence of functioning food markets, however, will not be an effective solution. An impact evaluation helps to answer some of these questions.
What is the magnitude of the impact of aid? To be able to understand the process of how joint humanitarian efforts affect selected nutritional and health outcomes, nutritional, health and other data should be collected, ideally both before and after the assistance is distributed at the individual, household and community levels.
What are the impacts of aid provided by a particular agency? To answer this question, it is important to define an identification strategy that is able to attribute impact to a particular intervention. First, it is desirable to have a comparable ‘sample’ to which it was not possible to provide an intervention, or which received an intervention at a later stage, to compare with the intervention group that received assistance. Having a control group helps to eliminate parallel ‘shocks’ that may influence both groups at the same time and which are not due to the intervention. To guarantee fair allocation and comparability, both samples must be selected on a random basis before the intervention—in a randomized control trial. In a humanitarian context, however, it is often the case that large areas or a large number of the population are affected (a high co-variability problem), or there would be ethical considerations around the agency delaying assistance any longer than necessary. Therefore, withholding assistance from a certain group of the population could not be considered. However, when resources are limited, or the intervention has potential side-effects as in public health programmes,[5] random selection could still be considered. The impact is the difference between the intervention group and the comparison group before and after the programme (see Table 1).
Another solution would be to start evaluating the programme after the intervention. In such cases, the evaluation team must rely on existing data available on the community before the disaster.
How sustainable are these impacts? In this case, researchers return to collect additional waves of data at a later stage (medium to long term). The nature of the intervention defines the timeline of the evaluation. In the case of nutritional programmes, the research team may come back after six months, one year or even longer, depending on the scale of the programme and the availability of resources.
What amount of assistance is needed and at what frequency should it be delivered? An impact evaluation team should perform rigorous statistical analyses using the data collected. It is therefore important to ensure that the data contains full information on the outcome indicators that the implementation team was aiming to improve, such as health indicators, the dietary diversity of all household members, and so on. Questions specific to the intervention should be formulated—and if possible piloted in advance—in a such a way that it is possible to learn detailed lessons about the impact of the programme.
Did certain groups within the affected population become better off than others? If so, in what ways and to what extent were their livelihoods improved? The most vulnerable are usually hit the hardest by disasters. Together with the implementation team, the impact evaluation team should help to identify the intervention target group. During humanitarian emergencies, it is important that data on displacement and an existing census or similar data are accessible to the research team.
Were there any unintended or negative consequences? When providing household-targeted interventions, it is essential to understand how resources are redistributed among all members of the households and whether certain groups have restricted access. Therefore, even though this requires more financial resources, surveying all household members is essential.
Could aid be delivered in a more cost-effective way? Impact evaluations using randomized control trials[6][7] can and have helped to identify the type of assistance—providing in-kind food, food vouchers or cash—that is most efficient or appropriate—in terms of both cost and effectiveness—for the target population and the conditions on the ground. The implementation and research team should identify the needs of the population and the context in which the intervention will take place before the intervention starts, using interviews during field visits and desk research.
To summarize, all the nuances of the intervention must be considered in order to be able to design an impact evaluation. Therefore, the implementation and impact evaluation teams should synergize to guarantee that all the available information is accessible to both.
The main advantage of a rigorous impact evaluation is that it contains a set of components that have been tested in many development programmes and that in combination can help to tease out unbiased impacts. Some methodologies—such as randomized control trials—are flexible enough to be adjusted to the humanitarian context and complement the existing monitoring and evaluation methodologies of humanitarian agencies. The key recommendations to humanitarian agencies are therefore:
As the climate changes, it is highly likely that the number and scale of disasters will grow in the coming years. The need for humanitarian assistance is therefore also likely to increase. This creates an urgency to ensure that humanitarian aid is spent in the most effective and efficient way.
[1] UNOCHA. https://www.unocha.org/story/global-humanitarian-appeal-hits-record-225…
[2] Data from the World Bank, <https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?name_desc=false>.
[3] Cook, J., ‘7 years after Haiti’s earthquake, millions still need aid: The resilience of the Haitian people has endured through years of natural disasters’, Huffpost, 1 Dec. 2017, <https://www.huffingtonpost.com/entry/haiti-earthquake-anniversary_us_58…;.
[4] Ojha, H., Baldry, E. and Shrestha, K., ‘Two years after the earthquake, why has Nepal failed to recover?’, Reliefweb, 12 May 2017, <https://reliefweb.int/report/nepal/two-years-after-earthquake-why-has-n…;
[5] Lorenc, T. and Oliver, K. J., ‘Adverse effects of public health interventions: a conceptual framework’, Journal of Epidemiology and Community Health, vol. 68, no 3 (Mar. 2014), pp. 288–290.
[6] Aker, J.C., 2013. Cash or Coupons? Testing the Impacts of Cash versus Vouchers in the Democratic Republic of Congo. SSRN Electronic Journal, pp.1–51.
[7] Hidrobo, M. et al., Impact evaluation of cash, food vouchers, and food transfers among Colombian refugees and poor Ecuadorians in Carchi and Sucumbíos. documents.wfp.org.
ABOUT THE AUTHOR(S)