In general, an effective programme evaluation will do much more than simply fulfill grant requirements. Evaluation should shed light on the process of implementing the programme as well as the impact that the programme had on participants and beneficiaries. Evaluation can:
The United Nations Development Programme Handbook on Planning, Monitoring and Evaluating for Development Results includes a detailed description of general considerations when creating an evaluation methodology.
Evaluation draws on the data gathered during the monitoring process and will collect final data related to many of the same indicators. As described above, evaluation can compare “with and without” activity in different locations or “before and after” measures in the same location. It is critical to discuss evaluation planning with experts from other organizations, from universities, or with donor technical assistance groups in the planning stages of the programme so that evaluation can be seamlessly integrated into programme activities (OECD/World Bank, 2004).
Data collection methods in evaluations are varied and can include many of the models discussed in the programme planning and design section:
See an overview of the pros and cons of each of these methods for gender equitable evaluation on the UN Women website.
Kenya – Evaluation of Trainings on Sexual Offences Act
The Kenya Women Judges Association (KWJA) conducted an evaluation of its local trainings for stakeholders, known as Court Users, on the Sexual Offences Act and Children Act. The evaluation used a relatively simple, post-hoc model. Training participants from six districts were brought together for a workshop to discuss how they felt that the training had helped or not helped in their work on the ground. The evaluation workshop consisted of a survey administered to participants as well as group discussions about how the trainings had impacted practice. Nevertheless the evaluation provided valuable information to KWJA about its work. The evaluation survey administered to training participants included the following questions:
- I was very much aware of it
- I had heard about it
- I had never heard about it
6. Had you read the Act before attending the Court Users Committee meetings?
7. How has this knowledge enhanced your understanding of the Act?
8. How prevalent is Sexual Gender Based Violence in your area of operation
9. How many cases/incidences were reported to you and what has been the outcome?
10. What challenges, if any, did you encounter in your area of operation and how has the training assist in overcoming these challenges? Please explain.
11. How helpful has the training assisted you in overcoming the challenges faced?
- Extremely helpful
- Not quite helpful
- Not helpful at all
12. Please explain your reason(s) for the above answer.
13. To what extent has your knowledge in the subject of the Sexual Offences Act and Sexual Gender Based Violence improved and increased as a result of the training?
14. To what extent has the training helped to enhance your appreciation and understanding of your job as a whole on the Sexual Offences Act and Sexual Gender Based Violence?
15. How has the training enhanced your expertise and skills in handling Sexual Gender Based Violence cases?
16. Has the Court Users Committee enhanced the coordination of the stakeholders in dealing with Sexual Gender Based Violence cases?
17. Do you think the Court Users Committee is a good tool in handling Sexual Gender Based Violence cases?
18. Please explain your reason(s) for the above answer.
19. What were the positive outcomes, if any, in the application of the knowledge, expertise and skills acquired during the training on the Sexual Offences Act and Sexual Gender Based Violence? Please explain.
20. If you have handled a case either as a Hon. Magistrate, Lawyer, Medical Practitioner, Prosecutor, Investigator, Police officer, Chief, Gender Officer, Probation officer, Children officer, will you share it with KWJA.
- If yes, forward to this address:
- If no, please give reasons
21. Did you find the following factors adequate?
- Facilitators a) Yes b) No
- Venue a) Yes b) No
- Interactions a) Yes b) No
- The materials used a) Yes b) No
- The approach used during the training a) Yes b) No
22. If not, kindly give your reason(s) below.
23. Please give us your comments on how these programmes can be improved in future
24. Before these trainings, had you ever heard of Kenya Women Judges Association?
25. Any additional comments.
The evaluation team then compiled data from these questions into charts. You can review the findings from the KWJA evaluation in the group’s Assessment Report. Also, the feedback discussions provided important insight into what participants valued about the trainings and how they were using the information. The table below provides information from discussions in each of the areas where trainings were held:
Source: Interview with KWJA Staff, Nairobi, March 2011; Kenya Women Judges Asociation. 2010. Report on the Sexual Offences Act, Sexual Gender Based Violence and Children’s Act Training Assessment. December 2010.
When choosing evaluation methods, consider the following:
Gender Equality and Human Rights Responsive Evaluation (UN Women, 2010). Available in English. See also the UN Women online guide to gender equality and human rights responsive evaluation in English, French and Spanish.
USA – Evaluation Survey of Domestic Violence Program
A final report of an evaluation survey conducted in the US state of Colorado demonstrates how surveys, even of small numbers of people can help provide valuable evaluation data relative to programme components. The Survey Analysis of the Domestic Violence Case Monitor Position revealed the following:
Source: Schwartz. 2006. Domestic Violence Case Monitor Position Survey Results.
Sample Evaluation of Domestic Violence Case Coordination Project (Maine, USA): Final Post-Survey
Goals: To gather information and make recommendations about
A. The sharing of information regarding pending DV criminal and civil cases and orders and the sharing of information among community partners;
B. The coordinated management of related DV criminal and civil cases and orders;
C. Systematic review of offenders’ compliance with court orders and sentencing judgments; and
D. Whether these practices and protocols are improving victim safety and offender accountability.
1. What is your role in this pilot project, and how long have you been involved in the
work you are doing?
Coordinated Community Response
2. How do you interact and share information with other partners involved with domestic violence cases?
3. Are you getting the information you need to make informed decisions or provide services that ensure victim safety and/or offender accountability? If not, what additional information would you like to have?
Effectiveness of protocols
4A. Have the protocols in this pilot project (e.g. providing related DV case information,
relationships developed in the advisory committee meetings, judicial review hearings
presided over by the same judge, participation of probation and BIPs at JR hearings, etc.) made a difference in your ability to serve/respond to/make decisions regarding victims and offenders in DV cases?
4B. Have they made a difference in terms of victim safety and/or offender accountability?
4C. Can you provide specific examples of the positive impact of the protocols?
Impact of Training
5. Did you attend the January 20 training with the Vera Institute? If so, did you implement or did you observe any changes in practices or protocols after the training?
What do you believe or what have you observed to be the impact of those changes, if any?
Suggestions for Improvement
6. Is there room for (further) improvement in what your court is doing with its domestic violence docket? If so, what kind of changes would you recommend?
Key Practices and Protocols
7. What do you believe are the most important protocols or practices for other Maine courts to consider in developing their own DV docket? (Refer to “Draft Uniform
Protocols” document as time allows, focusing on sections appropriate to the stakeholders. An alternative is to provide/e-mail the uniform protocols and ask them to e-mail comments.)
8. Have there been any unintended consequences, positive or negative, of the domestic violence docket or of any of the protocols implemented as part of the pilot project?
See the full evaluation.
Additional tools on monitoring and evaluation:
A Compendium of Monitoring and Evaluation Indicators (MEASURE Evaluation, 2008). Available in English.
Researching Violence Against Women: A Practical Guide, (PATH/WHO 2005). Available in English.
Rule of Law Tools for Post-Conflict States - Monitoring Legal Systems (Office of the High Commissioner for Human Rights, 2006). Available in English.
The Comprehensive Assessment Protocol: A Systemwide Review of Adult and Juvenile Sex Offender Management Strategies. (Center for Sex Offender Management, 2011). Contains a series of questions to assess prosecutorial practices in sex offender cases. Available in English.
Women’s Initiatives for Gender Justice has released four Gender Report Cards evaluating the effectiveness of the ICC’s implementation of gender justice principles. Read the 2009 Gender Report Card in English.
Illustrative monitoring and evaluation reports in the justice sector:
Different Systems, Similar Outcomes? Tracking Attrition in Reported Rape Cases in 11 European Countries (Lovett and Kelly, 2009). Available in English.
Responding to sexual violence: Attrition in the New Zealand Criminal Justice System (Triggs and Mossman, Jordan and Kingi, 2009). Available in English.
Implementation of the Bulgarian Law on Protection against Domestic Violence (The Bulgarian Gender Research Foundation and the Advocates for Human Rights, 2008). Available in English.
Judicial System Monitoring Programme (Women’s Justice Unit, Timor-Leste). Reports are available in English, Bhasa and Portuguese.
Tracking Rape Case Attrition in Gauteng: The Police Investigation Stage (Sigsworth, Vetten, Jewkes and Christofides/The Centre for the Study of Violence and Reconciliation, 2009). Available in English.
Tracking Justice: The Attrition of Rape Cases through the Criminal Justice System in Guateng (Sigsworth, Vetten, Jewkes, Loots, Dunseith and Christofides/The Centre for the Study of Violence and Reconciliation, 2008). Available in English.
Previous Topic Monitoring