Once the components of the framework have been established, staff can review the data that will need to be collected in order to measure the programme indicators. This should also involve listing where data to be collected is being recorded and the questions which they hope the project will respond to. As each question is reviewed, consider what data will address it. Questions may be listed in a table format alongside the specific variables being monitored to ensure there are no gaps in the data. Once all questions and indicators have been linked in this way, staff and/or advisory or steering committee members can determine what specific analysis processes will be appropriate for measuring each indicator.
Practices for monitoring shelter services generally involve documenting:
- The services the shelter is providing, by tracking shelter occupancy rates, numbers and types of shelter programmes accessed, etc.
- Staff and other resources required to provide the services, with details on budget, funding, facilities, materials, partnerships, etc.
- The groups of women using these services, with basic demographic information, history, exposure to violence, mental health and addiction issues, etc.
- The outcomes of services provided (women’s satisfaction with services and reported changes in their circumstances).
For some organizations, it is useful to review existing data collection processes and systems, and, if necessary, revise the tracking procedures and agency forms to ensure they adequately support the programme needs. Considerations in this review process include:
- Whether programme forms (intake/arrival forms or exit surveys) need to be revised or the new items added related to specific programme indicators.
- Many monitoring and evaluation resources can be saved if the organization adopts a uniform, well-organized file format for its written records as well as for its computer-managed database and ensures staff are trained/ able to correctly use these materials.
- Review the database used to aggregate programme information to identify whether it:
- allows the amount and type of data collected in the programme to be entered
- supports data analysis
- has a clearly identified person responsible for entering data into the system and whether the arrangement is practical for the new initiative
- requires any revisions to accommodate the new programme or to correct old problems
- will require staff to be trained in the use of the system
- If there are other organizations providing shelter or similar services for survivors in the area, it might be helpful for the shelter to engage the organizations through an inter-agency committee to select or develop a single data tracking system that can be applied across the community. This may facilitate information sharing while avoiding duplication (see guidance on maintaining survivor identity confidentiality), streamline staff training and support funders and others whose reporting requirements can be simplified and standardized under these conditions.
- If the organization is using specific assessment tools as part of its programming, it is important to ensure the tools are the most appropriate for the new initiative. Do they measure the outcomes of interest? Do they have adequate reliability and validity? Are they easy for staff to use with minimal training? Is the reading level appropriate? Caution should be used when employing tools that have not been tested.
As part of the planning process, shelters should identify the specific tools that will be used to appropriately monitor the activities and measure the anticipated outputs and outcomes according to the indicators selected. These tools may include a combination of:
- Paper tracking forms (e.g. arrival forms, referral forms, facility registration forms, and measurement tools such as rating scales). Individual paper files are usually kept for residents, with information entered in written form to meet important service needs such as goal setting, reporting on counseling sessions and maintaining essential contact information.
- Standardized or purpose-made assessment instruments (e.g. measure of trauma exposure, stress or mental health issues, a rating scale developed for observational activities) Assessment tools help staff better understand women and children’s situations, needs and service requirements. In most cases, with the exception of shelter-specific forms such as the intake/arrival form, monitoring and evaluation tools should be standardized; they should be scientifically tested, reliable and a valid means of effectively assessing the identified indicators. They should have a positive track record of use with the relevant population and include:
- standardized response scales
- a relatively low reading level requirement
- an established set of instructions
- clear information on administration, scoring and interpretation
- Computer-based systems such as Excel, Outcome Tracker or similar products that can be custom built for tracking data related to services used or results. Computer-managed formats facilitate the analysis of combined information across service users, allowing reports to be generated for funders or for other purposes. Computer-managed files should not be used for reporting identifiable information about specific individuals unless the files can be adequately password protected. See more on securing records.
- Both monitoring and evaluation projects may use these tools, although monitoring will tend to use data tracking systems and programme forms (e.g. intake form data, exit survey data) more than standardized instruments. Monitoring projects are generally undertaken to provide a detailed description of the population served by the shelter and the services provided to them. Where exposure to violence is an issue, monitoring may also require the use of standardized tools such as risk assessments for safety planning and/or various mental health measures (i.e. level of trauma, depression, anxiety) to provide a basis for the provision of specific services or to inform referrals.
- Standardized tools are often available in multiple languages and come with manuals and training requirements. Some such tools are proprietary (licensed to a particular company that sells them on behalf of the author), while others are available by mail or online from the author without charge. Instruments should be selected based on the specific context and resources available at the shelter (e.g. minimizing cost and training requirements while maximizing ease of use).
- The complexity of data analysis varies with the design of the project and the kinds of questions that are likely to be asked. Most monitoring projects can manage data analysis through procedures built into the database being used. For example, calculation of the overall number of clients seen or variables such as length of stay (range, mean, median, mode) or demographics of womn using services (age, number of children, income, referral sources) can usually be managed in programmes such as Excel or Outcome Tracker. More sophisticated databases like the Statistical Package for Social Sciences are often used for evaluation projects because they allow for additional testing to determine whether an observed pre- to post-test change is statistically significant or whether pre-existing conditions such as age or income level have an effect on outcomes. If such detailed analysis is required, it is important to use a database programme that allows data to be downloaded into an analysis programme such as the Statistical Package for Social Sciences and to include the cost of data analysis software and retaining specialized staff (e.g. consulting statistician) into the budget.