Indicators

Once the conceptual framework is finalized, the next step in completing the monitoring and evaluation framework is selecting indicators.  Indicators are signs of progress – they are used to determine whether the programme/intervention is on its way to achieving its objectives and goal.

What are indicators?

• An indicator is a specific, observable and measurable characteristic that can be used to show changes or progress a programme is making toward achieving a specific outcome.
• There should be at least one indicator for each outcome. The indicator should be focused, clear and specific. The change measured by the indicator should represent progress that the programme hopes to make.
• An indicator should be defined in precise, unambiguous terms that describe clearly and exactly what is being measured. Where practical, the indicator should give a relatively good idea of the data required and the population among whom the indicator is measured.
• Indicators do not specify a particular level of achievement -- the words “improved”, “increased”, or “decreased” do not belong in an indicator.

Characteristics of good indicators

• Valid: accurate measure of a behaviour, practice, task that is the expected output or outcome of the intervention
• Reliable: consistently measurable over time, in the same way by different observers
• Precise: operationally defined in clear terms
• Measurable: quantifiable using available tools and methods
• Timely: provides a measurement at time intervals relevant and appropriate in terms of programme goals and activities
• Programmatically important: linked to the programme or to achieving the programme objectives (Gage and Dunn, 2009)

Challenges and considerations when selecting indicators

• In an ideal world, indicators judged to be the highest quality and most useful would be the ones selected and used to monitor and evaluate programme activities.
• However, in the real world many other factors may intervene. Links to programme activities, as outlined in monitoring and evaluation frameworks are important, as are the needs of the programme for decision-making.
• Many indicators in common use are not well-defined in clear terms, or at least include terminology that could be improved to add greater precision. For instance, “knowledge of dating violence”, “attitude towards violence against women”, “support-seeking behaviour” of victims of violence, or “quality of services” can all mean and imply different things in different circumstances.
• The more defined an indicator, the less room there will be for later confusion or complications. For example, “percentage of women accessing health services at X facility from TIME A to TIME B who state that they received appropriate care and assistance” or “percentage of men who state that it is not acceptable to hit, slap, punch their wives with hands or other objects under any circumstances.”
• Ideal indicators may not be practical; the feasibility of using certain indicators can be constrained by the availability of data and financial and human resources. The requirements and needs of donors, the government, organization headquarters and others may need to be given priority.

Some examples of these considerations are:

• Availability of data: Some data may be considered ‘privileged’ information by agencies, projects, or government officials.
• Data may be available only on aggregated levels or already calculated into indicators that may not be the ideal indicators for your programme or activities.
• Resources: Ideal indicators might require collecting data to calculate an unknown denominator, or national data to compare with project area data, or tracking lifetime statistics for an affected and/or control population, etc.
• The cost of collecting appropriate data for ideal indicators is prohibitive.
• Human resources and technical skills may be a constraint as well.
• Programmatic and external requirements: Indicators may be imposed from above by those not trained in monitoring and evaluation techniques.
• Reporting schedules may not be synchronized (e.g. fiscal vs. reporting year).
• Different stakeholders’ priorities may diverge.
• Standardized indicators should be used if available.
• In general, programmes should stay away from indicators that activities cannot affect, that are too vague, that do not currently exist and cannot realistically be collected, or that do not accurately represent the desired outcome. (Gage and Dunn, 2009)
• When quantitative indicators of success cannot be identified, qualitative methods offer a valuable alternative. When it is difficult or not possible to measure “benefits” or “risks” in simple, quantitative terms, it is almost always possible to gather qualitative data, such as information on the perspectives of health care providers and women who come for services. In many cases, qualitative indicators provide more relevant information with respect to the success and effectiveness of the intervention. (Bott, Guedes and Claramunt, 2004) See the qualitative approaches section.
• Information on the perspectives of women and girls (rights-holders) and service providers (governmental duty-bearers or NGOs) is essential. Evaluation efforts and indicators should include the perspectives of both clients and providers. Information from women accessing or attempting to access services is critical for assessing the effectiveness of an intervention.
• Select a set of indicators that pertain to the identified objectives for the programme. If the intervention focuses on training providers, select indicators related for example to providers’ knowledge, attitudes and practices, ability to provide care, ability to make referrals, or others depending on the specific objective.

How many indicators are enough?

• Some guidelines to follow when selecting indicators:
• At least one or two indicators per result (ideally, from different sources)
• At least one indicator for every core activity (e.g. training, airing of TV spot)
• No more than 8-10 indicators per area of significant programme focus

• Use a mix of data collection strategies and sources

Process versus result/impact indicators: It is important to remember the difference between process and results indicators.

Process Indicators are used to monitor the number and types of activities carried out.  Examples include:

• The number and types of services provided
• The number of people trained
• The number and type of materials produced and disseminated
• The number and percentage of female clients screened

Results Indicators are used to evaluate whether or not the activity achieved the intended objectives or results. Examples include:

• Selected indicators of knowledge, attitudes and practices as measured by a survey
• The perceptions of survivors about the quality and benefits of services provided by an organization or institution as measured by individual interviews (Bott, Guedes and Claramunt, 2004)

Results indicators can be developed at the output, outcome and impact levels. (Bott, Guedes and Claramunt, 2004)

Output indicators illustrate the change related directly to the activities undertaken within the programme (e.g. percentage of traditional leaders in community x who completed the training on international human rights standards related to violence against women and girls whose knowledge improved.)

Outcome indicators relate to change that is demonstrated as a result of the programme interventions in the medium-to-longer term (e.g. the number of decisions in the informal justice system of community x related to violence against women that reflect a human rights-based approach.)

Impact indicators measure the long-term affect of programme interventions (e.g. the prevalence of violence against women and girls in community x.)

An important issue that needs to be resolved in order to monitor project progress is how to define success.  Commonly, organizations are able to track how many events they have held, and how many people have participated (outputs), but not how people have changed their attitudes or behaviours as a result (outcomes), especially over time.

The main indicator of impact should be a reduction in the prevalence and incidence of violence, but that takes years to achieve and to measure. So more indicators are needed to gauge whether programmes are moving in the right direction.

Monitoring and evaluation frameworks and plans should incorporate both process and results indicators.

Illustrative Indicators at the National and Programmatic Level:

Violence Against Women and Girls: A Compendium of Monitoring and Evaluation Indicators (MEASURE Evaluation/USAID, 2008). Available in English.

Measures for the Assessment of Dimensions of Violence against Women: A Compendium (Flood, 2008). Available in English.

Women, Peace & Security – Report of the Secretary General on UN Security Council Resolution 1325 Indicators for Monitoring Resolution Implementation (United Nations Security Council, 2010).  Available in English.

Towards a common European Framework to monitor progress in combating Violence against Women (European Women’s Lobby, 2001).  Available in English and Spanish.

Proposal of New Indicators to Measure the Effects of Gender Violence (Gender Violence Effects Indicators, 2009).  Available in English and Spanish.

Handbook on Performance Indicators for Counter-Trafficking Projects (IOM, 2009). Available in English.

Guidelines for the Collection of Data on Trafficking in Human Beings Including Comparable Indicators (International Organization for Migration (IOM) and Federal Ministry of the Interior of Austria, 2009). Available in English.

Indicadores sobre violencia contra las mujeres: Sistematización y evaluación crítica (Instituto Interamericano de Derechos Humanos, 2008).  Available in Spanish.

CEDAW Indicators for South Asia: An Initiative (CENWOR/UNIFEM, 2004).  Available in English.

See also the illustrative indicators for specific areas of work:

Health

Justice

Community Mobilization

Conflict/post-Conflict/Emergency

Internationally comparable indicators on violence against women and girls

Indicators can be developed to track a specific programme or intervention; to track national progress across sectors; and to track progress across countries globally.  In this regard, policymakers and activists have called for a comprehensive set of international indicators on violence against women to monitor States’ progress in addressing such violence.

The United Nations General Assembly requested the Statistical Commission to develop and propose, in consultation with the Commission on the Status of Women, a set of possible indicators on violence against women in order to assist States in assessing the scope, prevalence and incidence of violence against women.

In response to the General Assembly, the Statistical Commission, formed the Friends of the Chair group to build on the previous work undertaken by the United Nations Statistical Division, Regional Statistical Commissions and the Division for the Advancement of Women (2007), and the Special Rapporteur on violence against women, its causes and consequences, to establish a proposed set of indicators.

An interim set of six indicators was initially proposed and adopted at the Statistical Commission’s 40^th session (2009) and expanded by the Friends of the Chair to nine indicators in December 2009. The current nine interim indicators, which were accepted by the Statistical Commission in February 2011, are as follows:

• Total and age specific rate of women subject to physical violence in the last 12 months by severity of violence, relationship to perpetrator and frequency.
• Total and age specific rate of women subject to physical violence during lifetime by severity of violence, relationship to perpetrator and frequency.
• Total and age specific rate of women subject to sexual violence in the last 12 months by severity of violence, relationship to perpetrator and frequency.
• Total and age specific rate of women subject to sexual violence during lifetime by severity of violence, relationship to perpetrator and frequency.
• Total and age specific rate of ever-partnered women subjected to sexual and/or physical violence by current or former intimate partner in the last 12 months by frequency.
• Total and age specific rate of ever-partnered women subjected to sexual and/or physical violence by current or former intimate partner during lifetime by frequency.
• Total and age specific rate of women subjected to psychological violence in the past 12 months by the intimate partner.
• Total and age specific rate of women subjected to economic violence in the past 12 months by the intimate partner.
• Total and age specific rate of women subjected to female genital mutilation.

For more information on the process of developing the indicators, see Indicators to measure Violence against Women: In the context of efforts by the Friends of the Chair group on Statistical Indicators on Violence against Women and by the United Nations Economic Commission for Europe (Jansen, H., 2010)

An Expert Group Meeting was convened by the United Nations Economic Commission for Europe Conference of European Statisticians in September 2009 to discuss the development and testing of a survey module and methodology to measure the interim set of violence against women indicators. The report is available in English and the updates, along with the survey  module and complimentary tools are available from the UNECE website.