Monitoring and Evaluation Frameworks (3 parts)

A clear framework is essential to guide monitoring and evaluation.  A framework should explain how the programme is supposed to work by laying out the components of the initiative and the order or the steps needed to achieve the desired results. A framework increases understanding of the programme’s goals and objectives, defines the relationships between factors key to implementation, and articulates the internal and external elements that could affect the programme’s success.

Why are monitoring and evaluation frameworks important?

A well thought out monitoring and evaluation framework can assist greatly with thinking through programmatic strategies, objectives and planned activities, and whether they are indeed the most appropriate ones to implement.

Monitoring and evaluation frameworks:

• Assist in understanding and analyzing a programme
• Help to develop sound monitoring and evaluation plans and implementation of monitoring and evaluation activities
• Articulate programme goals and measurable short, medium and long-term objectives
• Define relationships among inputs, activities, outputs, outcomes and impacts
• Clarify the relationship between programme activities and external factors.
• Demonstrate how activities will lead to desired outcomes and impacts, especially when resources are not available to conduct rigorous impact evaluations. They often display relationships graphically.

(PATH Monitoring and Evaluation Initiative)

Considerations when developing a monitoring and evaluation framework
Asking questions:

• What are the objectives of the monitoring activities?
• What are the specific questions that need to be asked to gauge the progress of the intervention?
• What information is needed to see if activities are being implemented in the way that was planned, and who can provide that information?
• What are the objectives of the evaluation?
• What are the specific questions that need to be answered to gauge the impact and success of the intervention?
• What information is needed to determine if the expected objectives and outcomes were accomplished and who can provide that information?
• Determining whether the questions being asked are appropriate ones for understanding how “successful” the intervention has been with respect to its expected objectives and outcomes?

Developing a corresponding monitoring and evaluation plan that acts as a monitoring tool by defining how information from the programme will be tracked.

Developing the framework and plan before activities are implemented.

Determining which framework is best to use. A number of different frameworks are used or requested by organizations and donors. Some donors combine aspects of frameworks in a customized approach. Others do not include explicit guidance for programmes around the selection of a framework. Programmes should select the type of framework that best suits their strategies and activities and responds to institutional requirements.

It is also important to keep in mind that:

• Different kinds of interventions (policy change, awareness raising campaigns, community mobilization, improving service delivery and response) will need different kinds of frameworks, tools and indicators.
• An appropriate framework for monitoring and evaluation of activities can be designed and implemented even when a) programmes do not have significant resources b) programme staff and implementers, service providers and policy makers feel they do not have additional time to devote to monitoring and evaluation.
• Many existing tools can be adapted to specific contexts and monitoring and evaluation needs. If monitoring and evaluation activities and tools are considered and built into programmatic work or service provision from the start, the resource and time burden is minimized.
• It is important to clarify objectives, what information will be most useful in reaching those objectives and what information is already available or easily collected.

 

All programmes should at a minimum conduct monitoring activities that allow them to ensure they are not putting women at greater risk.

• There are creative ways to deal with resource constraints, such as:
  • Including a generous line item for monitoring and evaluation when submitting proposals to donors;
  • Using resources wisely by choosing methods that are feasible, reliable and most likely to yield information to improve the programme.
  • Collecting only enough data than can be realistically analyzed or used.
  • Finding ways to pool resources and collaborate with other organizations. In some settings, university students can offer assistance in return for research experience.  

(Bott, Guedes and Claramunt, 2004)

Steps in Developing Monitoring and Evaluation Frameworks

Programme implementers (often from diverse sectors) should jointly take steps in developing the monitoring and evaluation framework (UNHCR Guidelines, 2003).

1.    They should determine the purposes of the monitoring and evaluation mechanisms and assess the information needs of each actor .
2.    Ensure prevention and response interventions have clearly defined objectives, outputs and indicators;
3.    Establish coordinated and common reporting tools;
4.    Determine methods for obtaining information on indicators;
5.    Assign responsibilities for information gathering, determine time frame and frequency of data collection, and allocate resources; and
6.    Establish mechanisms for sharing information and incorporating results into prevention and response planning.
(UNHCR Guidelines, 2003)

Though there is no ideal framework and different frameworks are used for different situations, three of the most common are conceptual frameworks, results frameworks and logical frameworks/logic models. (Frankel and Gage, 2007)

i.    Conceptual Frameworks
Conceptual frameworks are diagrams that identify and illustrate relationships among relevant organizational, individual and other factors that may influence a programme and the successful achievement of goals and objectives. They Help determine which factors will influence the programme and outline how each of these factors (underlying, cultural, economic socio-political etc.) might relate to and affect the outcomes. They do not form the basis for monitoring and evaluation activities, but can help explain programme results.

Key issues and questions addressed in conceptual frameworks include:

1)  What is the theory of change framing the intervention?

The theory of change reflects the underlying process and pathways through which the hoped for change (in knowledge, behaviour, attitudes or practices, at the individual, institutional, community or other level) is expected to occur.

 

An example theory of change, based on the ecological model, from a community mobilization project to prevent domestic violence is presented below. While so much detail may not be necessary when outlining theories of change, monitoring and evaluation frameworks should always involve this kind of analysis. Specific, appropriate activities and indicators can then be developed to monitor key moments of change and evaluate success in effecting it.

Mobilising Communities to Prevent Domestic Violence: A Resource Guide for Organisations in East and Southern Africa, (Raising Voices, 2003)

Theory of Change: Preventing domestic violence in homes and communities requires individuals to identify the problem of domestic violence, consider its importance, evaluate their own behaviour, and then begin making changes in their lives. Behaviour is a result of our experiences, attitudes, and beliefs, and thus it is deeply linked to the prevailing belief system in the community. The attitudes and actions of neighbours, friends, co-workers, religious leaders, police, health care providers, etc. greatly influence an individual’s behavioural choices and collectively create the climate in the community. Although each individual is unique and will come to the issue of domestic violence differently, the process of how individuals change often follows a similar pattern.

Individual Behaviour Change: The Stages of Change Theory provides a way of understanding the process of how individuals can change their behaviour.

Stage 1 Pre-contemplation: An individual is unaware of the issue and its consequences for her/his life.

Stage 2 Contemplation: An individual begins to wonder if the issue relates to her/his life.

Stage 3 Preparation for Action: An individual gets more information and develops intent to act.

Stage 4 Action: An individual begins to try new and different ways of thinking and behaving.

Stage 5 Maintenance: An individual recognizes the benefits of the behaviour change and maintains it.Facilitating Social Change: The Resource Guide adapts this theory of individual behaviour change and scales it up to the community level. It proposes that a community also goes through a process of change before any given value system is adopted, and projects that recognize this process and operate in harmony with it are more likely to facilitate an enduring change.

(Michau and Naker, 2003)

 

2)  What is the range of potential exposures people may have to the intervention?
•    First, who is going to be exposed directly to the intervention?  For example, who will be receiving services? Who will be receiving training? Who will be receiving materials? Who will be hearing/seeing/participating in the campaign?
•    Second, who might be exposed indirectly to the intervention?  For example, who will probably be hearing about the campaign even if they are not the intended primary audience? Who will probably learn that services are being provided in a neighboring community?  Who will probably be exposed to some of the ideas disseminated in the training even if they did not directly participate?  

3)  What is a realistic timeframe for behaviour change to occur?
•    Keep in mind that while changes in knowledge may be easy to effect, changes in attitudes and behaviours, especially with community level norms, demand a much longer process.
•    Justice sector personnel receiving training about domestic violence legislation or protocols for handling sexual violence cases may be reasonably expected to improve knowledge of the law or the protocol in one session. However, stigmatizing attitudes that serve as barriers to their meaningful implementation cannot be expected to change overnight.

4)  How will this change be measured?
•    The conceptual framework should identify appropriate measurements for the kind of change that is expected.
•    Following on the example above, it would be appropriate to assess a change in the knowledge and attitudes of the justice sector participants by comparing them before and after the training. However, to determine whether the changes in knowledge and attitudes translated into changes in practice over time, a different method would have to be employed, for example, by reviewing court records or by interviewing complainants on their experiences with the handling of their cases.

Illustrative Example from the the Rural AIDS Development Action Research (RADAR) Programme Intervention with Microfinance for AIDS and Gender Equity (IMAGE) in South Africa

IMAGE seeks to influence factors that predispose individuals to HIV infection and gender-based violence through targeting the environment in which they occur.  Individual agency, household well-being, communication and power relations, and the norms, networks, relationships and responses of communities constitute the environment in the IMAGE framework. The framework attempts to conceptualise the complexity of factors and relationships that constitute the environment in which sexual behaviour and gender-based violence occurs. The framework was developed to guide both the intervention and evaluation components of the IMAGE programme.

Results Frameworks
Results frameworks sometimes called strategic frameworks illustrate the direct relationships between the intermediate results of activities all the way to the overall objectives and goals. They show the causal relationship between programme objectives and outline how each of the intermediate results/ outputs and outcomes relates to and facilitate the achievement of each objective, and how objectives relate to each other and the ultimate goal. Results frameworks do form the basis for monitoring and evaluation activities at the objective level.

 

Logical Frameworks
Logical frameworks or logic models provide a linear, “logical” interpretation of the relationship between inputs, activities, outputs, outcomes and impacts with respect to objectives and goals.  They show the causal relationship between inputs, activities, outputs, outcomes and impact vis-à-vis the goals and objectives. Logical frameworks outline the specific inputs needed to carry out the activities/processes to produce specific outputs which will result in specific outcomes and impacts. Logical frameworks do form the basis for monitoring and evaluation activities for all stages of the programme.

Logic models are valuable tools for:

• Programme Planning and Development: The logic model structure helps think through your programme strategy—to help clarify where the programme is and where the programme should be.
• Programme Management: Because it "connects the dots" between resources, activities, and outcomes, a logic model can be the basis for developing a more detailed management plan. Using data collection and an evaluation plan, the logic model helps track and monitor operations to better manage results. It can serve as the foundation for creating budgets and work plans.
• Communication. A well-built logic model is a powerful communications tool. It can show stakeholders at a glance what a programme is doing (activities) and what it is achieving (outcomes), emphasizing the link between the two.

Logical frameworks are presented as diagrams connecting programme inputs to processes, outputs, outcome and impact as they relate to a specific problem or situation.  Logic models show what resources the programme will need to accomplish its goals; what the programme will do; and what it hopes to achieve, emphasizing links between these aspects.

A series of “if-then” relationships connect the components of the logic model: if resources are available to the programme, then programme activities can be implemented; if programme activities are implemented successfully, then certain outputs and outcomes can be expected.

The logical framework does not try to account for all of the factors that may influence a programme’s operation and results like a conceptual framework.  Instead, the logic framework focuses on the programme’s inputs, activities, and results. This narrow focus assists programme managers and monitoring and evaluation planners as they clarify the direct relationships among elements of particular interest within a specific programme.
(Adapted from Gage and Dunn, 2009 and PATH M&E Initiative)

Example Logical Framework for a Health Provider Training Programme
This logical framework presents a straightforward view of a project designed to improve health providers’ knowledge, attitudes and practices and to increase providers’ awareness of violence against women as a public health problem and a violation of human rights.



[Gage and Dunn, 2009]

Example logical framework framework for understanding the impact of communication, advocacy and training activities on violence against women and HIV (PATH, 2007)