Evaluation happens after a safe cities for women programme, activity, initiative, or project has been operational for a period of time. The purpose of an evaluation is to look back at what has taken place and decide how it was successful and how it was unsuccessful. There are three main types of evaluation: process evaluation, outcome and impact evaluation. The first type of evaluation determines whether or not the process of creating and implementing a programme or initiative is successful. Process evaluation usually occurs over a span of time – before, during, and after the process. By contrast, outcome and impact evaluations determine whether or not a programme or initiative met its desired outcomes or goal. Ideally, both process and and an endline evaluation (whether outcome or impact) should be used in safe cities for women programmes.
For more information on monitoring and evaluation, the terms and types of evaluation, see the monitoring and evaluation section of the Programming Essentials section of the site.
An evaluation is important for three main reasons. Firstly, it enables the public and safe cities for women programme partners to decide whether or not a strategy or programme was successful in meeting its objectives. Secondly, an evaluation helps programme partners learn from their experience and build on their successes in an ongoing process of improving their work. Thirdly, an evaluation allows programme partners to share what they have learned about successful practices, ideas and challenges with other people working in safe cities for women programmes elsewhere. This kind of sharing helps good approaches to be repeated and adapted, and bad ones to be avoided or improved. Information gathered during programme monitoring serves as the basis for evaluation. Safe cities for women programme partners should plan for evaluation from the beginning of any project because it requires them to take on special responsibilities and to allocate financial and human resources.
Gather baseline information before starting a safe cities for women programme.
Baseline information is information about conditions on the ground as they exist before programme interventions are undertaken. Baseline information is important because safe cities for women programme partners can use this information to compare what the community was like before and after their initiative. For example, if a safe cities for women programme aims to increase the number of women who feel safe in a particular public square, baseline information might be collected on how many women use the square, the activities women carry out in the square, and how women feel about using the square before the initiative begins. After actions have been taken to improve the safety of the square, this information is gathered again. Then, using the baseline information and the new information, programme partners can compare their data and determine whether or not their work was successful, based on whether women use the square more, use the square differently, or report feeling safer while in the square.
Resources:
Summary Information on Policies and Initiatives Promoting Women’s Safety (Gender Inclusive Cities Programme, 2009). This tool was created to help safe cities for women programme partners identify legislation, policies and initiatives that have positive or negative impacts on women’s safety. The tool provides programme partners with a policy review matrix worksheet that they can fill out with information such as the level of gender mainstreaming in local/regional/national budgets and the number of police initiatives on violence against women. Available in English.
Survey for Youth (2010). This survey, developed by METRAC and the Youth Alliance Project, is designed to find out more about young women's experiences with the police when they report sexual assault, physical assault and stalking cases. The survey is designed for young women in the Canadian urban context. Available in English.
Evaluating Crime Prevention through Social Development Projects: Handbook for Community Groups (Public Safety and emergency Preparedness Canada, 2006). This is a comprehensive handbook that community groups can use as a guide through any crime prevention project evaluation process. The handbook provides an overview of what evaluation is and the kinds of evaluations community groups should consider. It discusses preparing a logic model, creating an evaluation plan, collecting data, different options for designing an evaluation, and analyzing data. In addition, challenges and solutions, as well as case studies, are presented. Worksheets are given for different evaluation activities, as are resources for further research. Available in English and French.
Researching Violence against Women: A Practical Guide for Researchers and Activists (PATH and World Health Organization, 2005). The aim of this guide and toolkit is to help activists and other researchers determine the extent and type of violence women experience, so that policies and programmes can be put in place to address the particular kinds of violence women experience in local and regional contexts. The main focus of this guide is intimate partner violence. However, the methods used to collect data can be used for other types of violence against women, including violence against women in public spaces. Qualitative and quantitative research approaches are covered, including cross-sectional surveys, case control studies, in-depth qualitative studies and more. Tools are provided for several research steps, including formulating questions and collecting open-ended stories. Case studies are also given throughout the guide. Available in English.
How Do We Measure the Prevention of Violence against Women? (Prevention Connection: The Violence against Women Prevention Partnership, 2008). California Coalition Against Sexual Assault, USA. This is a short, annotated slide presentation about measuring levels of violence against women in a given community. This presentation discusses the challenges of measurement. It also discusses several methods for measurement, considering the unit of analysis, social climate, norms, community capital, and more. Evidence-based analysis is a main focus. In addition, the presentation provides three case studies of different organizational approaches to measuring violence against women. The information provided will be most useful to government agencies and academic researchers. Available in English: 12:11 minutes.
Ensure that evaluation methods are well-suited to the particular activities they are assessing.
There are different ways that safe cities for women programme partners can evaluate their work – conducting surveys, researching changes in crime statistics, monitoring use of public spaces or services, and reviewing meeting notes are some examples. Some methods, which require technical knowledge (such as analyzing statistical data), may only be appropriate for projects that are large-scale and involve a number of different actors. For smaller projects, handing out an evaluation form for participants to fill out and return may be more appropriate.
Resources:
Capturing Change in Women's Realities: A Critical Overview of Current Monitoring and Evaluation Frameworks and Approaches (2010). This document discusses the limitations within traditional approaches to monitoring and evaluation, as they apply to women's organizations. It details the reasons why some models of monitoring and evaluation fail to capture information about social change as it pertains to gender relations. This document also provides a feminist review of different M&E frameworks, including strengths and weaknesses. Available in English.
“Evaluation Form: CAC Workshop 1: Understanding Domestic Violence” in Mobilising Communities to Prevent Domestic Violence: A Resource Guide for Organizations in East and Southern Africa. Raising Voices, Kampala, Uganda: Appendix, page 296. This evaluation form has been created by Raising Voices for participants in a workshop on domestic violence. However, by changing the activities section to include activities used in a safe cities for women programme workshop, partners can use this form to find out whether or not their workshop was successful. Questions centre on how satisfactory the workshop venue, activities, and facilitators were. Available in English.
Research participatory evaluation.
Participatory evaluation is a methodology that is particularly suited to safe cities for women programmes because it allows local women and safe cities for women partners to be involved in the evaluation process. This type of evaluation can be used on its own, or combined with other methods. Participatory evaluation is important because it is based on the opinions of the people who are affected by the programme and, therefore, know firsthand what works and what does not. Usually, participatory evaluation involves the development of a evaluation work plan (which serves as a tool to decide what information will be collected and how, e.g. interviews, surveys), holding meetings to discuss results, generating a report, and collectively reviewing and finalizing the report. If safe cities for women programme partners choose to use this kind of methodology for evaluating their programmes, projects, initiatives or activities, participants should be selected according to the focus of the work. For instance, if a project is designed to raise awareness about sexual harassment in subways, then men, boys, women, girls, and transit employees should all be involved in the evaluation. The following tips, which focus specifically on girls and young women, can be used as an example of making an evaluation process participatory:
- Provide training for girls in engaging and interactive styles that expand and deepen their understanding of the basics and uses of evaluation.
- Involve girls in the development of the evaluation and in making decisions about its focus, methods to be utilized, how data will be used, confidentiality issues and reporting.
- Be mindful of literacy and language issues in choosing training methods and data collection tools.
- Set realistic goals and expectations for girls’ involvement in the process – agree ahead of time on support and resource needs and levels and means of accountability.
- Set realistic goals and objectives for the programme activities and for the evaluation, including a realistic timeframe that will be in line with the length of time girls are involved in the programme.
- Run interference with community or institutional partners who could potentially get in the way of the girls being able to complete their evaluation activities.
- Showcase the results of the evaluation in some way that involves family and community and provides validation for the girls and their involvement and ownership of the process.
From Fullwood, C. (2005). Working with Girls as Evaluators. Available in English.
Resource:
Tools of the Trade: A CWIT Guide to Participatory Evaluation (Center for Research on Women and Gender no date). University of Illinois at Chicago, USA., This guide and toolkit offers general guidance on methods of participatory evaluation, with a focus on involving women in the process. The guide breaks the evaluation process down into three stages: developing an evaluation plan, choosing data-evaluation methods, and writing an evaluation report. Step-by-step guidance is provided for each stage, as are worksheets and question-and-answer exercises. Available in English.
Consider hiring an external evaluator.
Hiring a professional to come in and evaluate a safe cities for women programme can be a beneficial option for those groups that are able to budget enough funds for this purpose. External evaluators are hired at the programme’s or project’s inception and are helpful because they provide a thorough and comprehensive look at all activities from an objective perspective (their judgement is not influenced by personal biases or attachments). Professional evaluators may point out aspects of the programme that others failed to notice. Moreover, using an external evaluator makes programme results more legitimate (e.g. for policy-makers, donors, experts, other stakeholders) because others will know that a professional and objective process was used to determine successes and good practices (Whitzman, 2008b, 96). It should be noted that any external evaluator hired for a safe cities programme should have relevant background issue in community safety and gender issues.
Ensure ongoing internal evaluation once the programme, activity, initiative or project has been operational for a period of time, or draws to a close.
Throughout the duration of a programme or initiative, and at its conclusion, safe cities for women programme partners should ask themselves what they have learned and how successful their process was. When this information is obtained while the project is operational, findings can be used to improve upon work happening on the ground. All partners, including community decision-makers, community organizations, and, of course, local women and girls should be involved in this process. Some questions programme partners can use to evaluate their work are:
- Why was this action (programme, activity, initiative or project) started? Are the reasons that were given at the beginning still relevant?
- What have been the results (both positive and negative) of these actions?
- Were the original goals and objectives achieved?
- Was the effort and expense of implementing these actions worth the outcomes that have occurred?
- In retrospect, are their other actions and strategies that could have been more successful?
- What does the evaluation suggest about the actions that should be taken next?
Resources:
“Program Planning and Evaluation” Guide in Leading Community Change: A Workshop Guide to Build Women’s Volunteer Leadership Skills (Status of Women Council of the Northwest Territories, Canada, and Women’s Voices in Leadership, no date). The guide provides workshop activities for facilitators to train women volunteers on how to plan and evaluate community programmes. Safe cities for women programme partners can use this guide to assess their strengths and needs, develop an evaluation framework, and hold practice sessions on planning and evaluation. Worksheets for activities (pages 79 – 100) and evaluation worksheets (pages 9 – 18) are included. Available in English.
“Developing an Evaluation Plan” Activity (2001) in Outcome Mapping: Building Learning and Reflection into Development Programs (Earl, S., F. Carden and T. Smuytlo, 2001). International Development Research Centre, Canada: Page 115. This activity in the toolkit is meant to be used by any programme or organization that wishes to evaluate its success based on changed attitudes and behaviors (as opposed to statistical impacts). This approach is particularly suited towards safe cities for women programmes because their aim is not only to prevent violence against women in cities and communities, but also to increase women's feelings of safety in public spaces. The “Developing an Evaluation Plan” Activity provides programme partners with an outline and worksheet to use to determine evaluation issues, audience questions, responsibilities and more. Available in English, French, Spanish, Portuguese and Thai.
Remember that some successes take a long time to manifest.
Safe cities for women programme partners should not be completely discouraged at first if they see few positive results from an initiative. Sometimes it takes years for positive changes to occur in a community. For example, programme partners may conduct a women’s safety audit and recommend that the municipal government strengthen the relationship between its police force and its women’s department as a result. This uptake and implementation of this recommendation, however, may take five years to materialize, or a change in administration may require programme partners to make the same recommendation during and/or after a mayoral electoral campaign to promote follow up on the recommendation.
Share your results.
The point of doing monitoring and evaluation is to learn from successes and mistakes. Sharing knowledge about successes and mistakes means that safe cities for women programme partners and other groups (community organizations, municipal governments, women’s groups, and others working on women’s safety) can find out what might work or what is ill advised for their own projects (Michau and Naker, 225). Moreover, sharing results can raise awareness about safe cities for women programmes and women’s and girls’ safety in general. Results can be shared in a report, in a media statement, on the Internet (on web sites, in emails or posted in discussion groups), in academic journals (especially in the case of rigorous external evaluations), in public community settings, in flyers, in plays – depending on the audiences, the objective/s of the dissemination effort (e.g. empowerment of community members, support for continuation at policy and budget levels, other) and what the programme team and partners determine.
Guidance Note on Developing an Evaluation Dissemination Strategy (UNIFEM, 2009). Available in English.
Do not be afraid to admit that some elements of your programme were less successful than others.
Sometimes, women, girls and other safe cities for women programme partners will be disappointed when they realize that their project is not a complete success, or they are unable to report on concrete results. However, the purpose of evaluation is to provide insight into what might have been done differently and what could be improved during the next steps. This knowledge is helpful for everyone involved. Just understanding what does and does not make cities safer for women and girls can be a success in itself.
Resource:
Women’s Safety Audits: What Works and Where? (2008). This report, produced by Women in Cities International, UN-HABITAT and Sida, is an example of a summary of evaluations to date on the women’s safety audit tool. The report begins with a literature review of evaluation of women’s safety audits. The literature review covers practices that work and positive outcomes that have been recorded about the women’s safety audit, as well as practices that do not work and negative outcomes. The literature review also discusses evaluation questions, and recommendations that have come up in material about the women’s safety audit. In addition to the literature review, this report also gives the results of in-depth surveys with groups who have undergone the women’s safety audit. Survey results provide further evaluation information on the successes and challenges of different aspects and adaptations of the tool. As a whole, the report is meant to provide information on how the women’s safety audit tool has been used thus far, and possibilities for its use in the future. It is directed at any group or government wishing to initiate a women’s safety audit in their community. The report is available in English.
Think about the key steps and challenges that are inherent in reviewing and evaluating public policy development and implementation.
When evaluating a public policy, it is important to always keep in mind and refer back to the original problem the policy was designed to address and the results that it aimed to achieve. In this sense, it is necessary to review and evaluate all the previous stages of the design and implementation process in order to foresee any problems that may arise during the evaluation process. It may be difficult to evaluate public policies because of the fact that the social and cultural environment in which the action is being implemented is constantly changing. This constant change can impact the policy’s original objectives. It may also be difficult to evaluate a policy if the people involved in the evaluation process attempt to influence evaluation results. For example, some evaluators can look for and report on certain kinds of information associated with policies, while ignoring other kinds of information, in an attempt to further their own interests (Burijovich; 2005).