Moving a step beyond routine programme monitoring and evaluation activities, impact evaluations measure the extent to which an intervention or programme is responsible for change within a target population. In this regard, the main impact evaluation question is “To what extent is the programme or intervention responsible for the change?” To measure programme impact, an assessment or evaluation is typically conducted at the start of the programme as a baseline and again at the end of the programme as an end line. Attributing changes to the programme requires one to rule out all other possible explanations. Therefore, it is important that baseline and end line measurements are also taken in comparison groups of people who are not receiving the intervention or programme, so that difference between the two groups can be attributed to the intervention or programme. Comparison groups will typically have similar characteristics to the intervention group but without the intervention or programme.
Establishing causality between the programme and the change is important but can be difficult to measure because the change can also happen in the absence of the programme. In order to determine causality, all external or confounding factors that may account for the results must be accounted for, typically by randomly assigning individuals or communities to receive intervention or not. This requires extensive knowledge of sampling and statistical analysis. For example, if the study design does not involve a randomly assigned comparison group (known as a control group), the difference in outcome between the control group and the group receiving the intervention or programme becomes an analytical exercise rather than a direct measurement. Although impact evaluations represent the gold standard for evaluating change caused by an intervention or programme, real-world constraints such as time, resources and budget need to be carefully considered as these constraints are often further exacerbated in conflict and post-conflict settings. In such situations, an impact evaluation may not always be the most appropriate evaluation design.
When implementing an impact evaluation on VAWG in these settings, it is again important to consider the ethical implications of this work. The most rigorous form of evaluation (a randomized control trail or RCT) is only appropriate if we feel justified in withholding the intervention for some groups. As such, for many core VAWG response programmes these approaches are not appropriate. Evaluations typically focus on the quality of response services provided and track individuals who receive services without a corresponding control/comparison group. For VAWG prevention programmes, more rigorous evaluation design can be appropriate – particularly in protracted crises or post-conflict settings where the underlying population is more stable.
|
Box 3: Example of an Impact Evaluation for VAWG
While not conducted in a conflict-affected community, the cluster randomized control trial of the SASA! program in Kampala has a number of relevant lessons for evaluations in refugee and conflict-affected settings. The choice of a randomized trial, while the most rigorous evaluation design, had considerable impact on practitioners and how the program could be designed and implemented in this context. A particular concern for the implementing agency, CEDOVIP, in using this approach was how to balance the power of SASA!’s community mobilization approach with the potential for contamination.
In research terms, contamination occurs when members of the control group (who are not meant to receive the evaluation) are somehow exposed to the intervention and begin to change along with the intervention group. In an approach like SASA! this was a real concern as the programming approach relied on community members talking to each other and spreading messages to effect behavior change.
In the SASA! evaluation, considerable effort was made to minimize the potential for contamination. Buffer zones were established between intervention and control communities, community activists were closely supervised to ensure they worked only in the intervention communities etc. In the end, contamination was minimized and the researchers found a significant reduction in negative behaviors in the intervention communities as compared to the control.
See more on the study: https://jech.bmj.com/content/70/8/818 |