Independent evaluators, who may include organizations such as universities or research institutes or consultancy organizations with expertise in gender-based violence, may help to strengthen or coach an internal evaluation team and might be useful for evaluating a security initiative in the following contexts (adapted from CIDA, 2006):
-
The implementing organization or donor wants to ensure objectivity, to give greater credibility to positive results obtained and provide ideas and suggestions of how to improve the initiative. It might be particularly useful if issues are particularly sensitive or survivors are not happy with programme results (e.g. women report that police response has worsened, while police report improved response times and better treatment of survivors).
-
Data collected for monitoring is complex or weak and requires an experienced evaluator who can adapt the methodology and strategy to complete the evaluation in the absence of full data set. For example, when:
-
A wide variety of information-gathering methods with be used, requiring detailed comparison and analysis. This may involve quantitative and qualitative data collection methods with security institutions, oversight bodies, as well as other service providers, community groups and survivors.
-
Evaluation data are obtained at different points in time and they need to be analyzed to see what changes have occurred and why. This may be challenging where records have not been kept consistently overtime and there are gaps in the data and expert advice may be needed.
-
It is unclear what information is needed to answer evaluation questions (e.g. whether the focus should be on evaluating security sector institutional capacity, the legal and policy framework, service delivery and/ or oversight).
-
General Considerations for using an independent evaluator (adapted from Women against Violence Europe, 2006 and Popovic, N., 2008)
-
The terms of reference for an independent evaluator should clearly state the issues to be included in the evaluation, and the assessment team should explicitly require at least one member to have gender and ideally violence against women expertise. If this is not possible, specific gender-based violence briefings should be scheduled for the assessment team, including orientation on specific methodologies for gathering data on violence against women and ethical practices for engaging women and girls.
-
Where funds are not available for large-scale independent evaluations, a security institution can invite an expert from outside to conduct periodic evaluation seminars to develop internal evaluation capacity.
-
Independent organizations can apply for funding either from the government or international donors to conduct their own monitoring and evaluation of security institutions. For example, a consortium of research and women’s organizations conducted an independent evaluation of Women Police Stations in Latin America.
Example: Comparative study of initiatives to improve access to justice for women in situations of violence (Brazil, Ecuador, Nicaragua, Peru)
The independent comparative analysis conducted from 2007-2008 on the contributions of women’s police stations to access to justice for women in situations of violence and the exercise of their rights used a variety of participatory methods. The study aimed to make proposals for the improvement of public policy in the sector and evolved through coordination among researchers working on the Stations, starting in Nicaragua and Brazil, who had previously studied the Stations and had national and/or local levels contacts. In Ecuador, Peru and Nicaragua, where the Stations are national institutions, national permission was granted for the work. In Brazil, where the relevant police force is a state-level body, permission was granted by the relevant state police official. In all cases, the Stations were consulted on an ongoing basis throughout the study, along with other stakeholders. The researchers recognized the need for a comparative study due to the importance of the Stations across Latin America and the lack of existing regional studies. A preliminary literature review and consultation with specialists informed the project design.
The project involved International Development Research Centre (as the main donor); research centres – academic and non-governmental centres renowned for their research on violence against women, gender and women’s rights; and the Stations. The Centre for Planning and Social Studies (Ecuador) led the study, in partnership with: the University of Campinas Gender Studies Department (Brazil), InterCambios/PATH (Nicaragua), and the Flora Tristán Women’s Centre/ Manuela Ramos Movement (Peru). The York University Centre for Research on Latin America and the Caribbean (Canada) also collaborated in the project.
Participatory methods used by the regional research team in the study included:
-
Collaboration in the design of each phase, on the research components and during analysis of the results;
-
Consultations with national and/or local stakeholders, especially the Stations, other state institutions, feminist and women’s organizations, researchers, and local networks. Regular meetings were held to present the research design and preliminary findings at each stage, invaluable for ensuring the relevance of the research, improving access to information, and building ownership of final results.
An ethical protocol was designed focused on: safety and security of research participants/researchers, confidentiality, scientific rigour, and creation of benefits.
The study used an integral women, intersectionality, and power analysis. Instead of analyzing whether the Stations met their institutional mandates, the framework focused on examining women’s perceptions of violence against them and access to justice, as well as their paths to gain access to justice and eliminate violence. The Station models and services were studied in terms of their responses to women’s expectations and their contributions to defending women’s rights based on states’ regional and international human rights obligations. The research proceeded in three key stages:
-
A mapping study of the models with a historical and national scope (in English, Spanish and Portuguese).
-
A representative, population-based survey of adult women in each research site, covering their knowledge and perceptions regarding their rights, violence against women, and the Stations. (in Spanish - Ecuador and Peru; Portuguese for Brazil)
-
In-depth and exit interviews with survivors, and interviews with Station operators and authorities, other state institutions, women’s organizations and others that provide specialized services. Interview guide and protocol available (in Spanish).
See the full case study.
Source: Nadine Jubb, 2010, based on Jubb, et. al., 2010, Women’s Police Stations in Latin America: An Entry Point for Stopping Violence and Gaining Access to Justice. CEPALES.