A range of tools are available to evaluate mass communication efforts. This list provides a few relatively simple tools.
Website monitoring
Software can track “hits” on the campaign website and their approximate origin, and detect website usage patterns. Websites can display simple surveys, e.g. asking users to express their opinion on a specific subject, and track variation in the answers over time. Other interactive tools and web 2.0 activities (e.g. the social web) can be used to solicit comments from target group members which can be analyzed as part of an evaluation.
Example: The Choose Respect campaign, by the US Centers for Disease Control and Prevention (CDC), is a national prevention effort to help parents, caregivers, older teens, educators, and other caring adults motivate teens to challenge harmful beliefs about dating violence and take steps to form healthy and respectful relationships. Its website offers information and resources for use at home, at school, and in the community, such as the Choose Respect playbook, an activity guide for parents, educators and community members to engage teens on teen dating violence topics, and Dating Matters, a free, hour-long online training designed to help educators, youth-serving organizations, and others working with teens understand the risk factors and warning signs associated with teen dating violence. The campaign team measured the number of hits to the website and usage patterns to monitor awareness about the campaign.
Watch the video and download the discussion guide. See the TV ads.
Internet surveys
Internet surveys via specialized websites are a common tool, e.g. Survey Monkey, which offers a free package for basic internet surveys. Preconditions for success are: (i) prospective survey participants must be computer-literate, (ii) their e-mail addresses must be known by the evaluators, and (iii) a reasonable response rate can be expected.
Examples:
The APC Women's Networking Support Programme, a global network of women for social change and women’s empowerment, has developed a Gender Evaluation Methodology (GEM) designed for ICT-based (information and communication technology) initiatives. The toolkit provides step-by-step guidance to evaluate whether ICTs are improving women's lives and gender relations.
The ‘Bringing in the Bystander’ campaign targeted students, emphasizing the role of the bystander in sexual violence prevention. To motivate students to participate in the evaluation survey, they were given the opportunity to win an mp3 player. The online survey asked students questions about their willingness to intervene in a situation where there was the potential for sexual violence to occur. After answering questions regarding their demographics and bystander behaviour, they were shown the campaign posters and asked whether or not they had seen the posters on campus. This enabled the researchers to compare the differences in the responses between students who reported seeing the campaign posters and those who did not. All survey participants were asked demographic questions in order to analyze the campaign's impact across the different groups on campus.
(Source: Potter, S., 2008. Incorporating Evaluation into Media Campaign Design. Harrisburg, PA, on VAWnet, a project of the National Resource Center on Domestic Violence/Pennsylvania Coalition Against Domestic Violence.)
Newspaper, television and radio tracking
There are professional services that can be hired to gather and analyze regular media reports and coverage of campaign issues, as well as advertisement (ad) placement in newspapers and other media. This allows for tracking the volume of coverage a campaign generates, message outreach and placement in the media, , and how often the coverage reflects the campaign’s messages or intended framing of issues.. Note however, that it is usually costly to engage such professional services so they should be appropriately factored into campaign budgeting. (See also Media monitoring).
Advertisement (ad) assessments can provide measures of advertisement recognition and recall. The process begins with face-to-face interviews whereby a researcher goes through a publication page-by-page and asks a reader whether he or she recalls seeing a specific ad, remembers the name of the advertiser or campaign, and how much of the ad was read.
Example: In 2010 UN Women, with the support of the Swedish International Development Agency (Sida) launched project ShiELd to enhance prevention and the response to domestic violence. As part of the prevention efforts under this initiative, the Georgian Rugby Union (GRU) and the International Rugby Board (IRB) joined the UN Secretary General’s “UNiTE to End Violence against Women” Campaign and devoted to promoting zero tolerance for gender-based violence through messaging and T-shirts, and by raising awareness on the domestic violence hotline. The campaign received wide national publicity.
See the Communications Monitoring Report.
Framing analysis examines how issues are presented or discussed (framed) in the media. It looks for key themes, expressed as arguments, metaphors, and descriptions to reveal which parts of the issue are emphasized, which are pushed to the margins and which are missing (McManus & Dorfman, 2002: Silent Revolution: How U.S. Newspapers Portray Child Care). Content analysis is combined with focus groups and interviews so as to garner different perspectives. Framing analysis is typically done in a campaign’s creative design phase, but analyses can also be done, for example, before and after a campaign to examine changes over time.
Direct response tracking
Some ads or public service announcements used in campaigns ask readers or viewers to give a direct response (e.g. hitting a voting button on a website), or perform a measurable action like calling a toll-free number or sending in a coupon. These responses can be counted and used as an indicator for campaign reach.
Rolling sample surveys
Adapted from political polling methods, this method has been applied in public communication campaign outcome and impact evaluations (Henry, Gary T. & Gordon, Craig S., 2003. Driving Less for Better Air). Daily surveys obtain measures of target outcomes (e.g., attitudes, behaviours) from an independent sample of individuals drawn each day. This method allows the evaluator to track the day-to-day shifts in public interest and behaviour, and enables evaluators to create natural experiments based on when known events or media coverage will take place. (Treatment measures focus on the days when campaign events, e.g. public service announcements, are planned; comparison measures focus on days when no campaign events take place).
TOOLS:
Public Communication Campaign Evaluation, Harvard Family Research Project, 2002 – an environmental scan of what has been happening in the field of public communication campaign evaluation in recent years and what choices evaluators have been making in terms of their evaluation designs and methods. Offers a discussion of how campaign evaluations have taken place in practice and the challenges that exist. It provides examples of campaign evaluations, including a number related to violence-prevention.
Lessons in Evaluating Communications Campaigns: Five Case Studies, Communications Consortium Media Center/ Harvard Family Research Project, 2003.