There’s more to evaluation than finding out if a project has been a success or failure. It can also say something about evaluation design. Here are seven take outs from our Public participation evaluation frameworks: USA.
Evaluation frameworks reveal the benefits of public participation
To ‘evaluate’ something, by the very meaning of the word, is to determine its worth. Tellingly, many of the resources in the primer go beyond rigid parameters of success or failure to understand the value of participation for both communities and decision-makers. They illustrate the ‘how’ and ‘why’ of evaluation – and further, the ‘why’ of engagement itself.
Evaluation is more than an audit – it is also learning
Participatory processes vary in their design and goals. As an emerging and contested area, evaluation reflects this complexity. Measuring participation is more than just seeing if participation has worked – it also examines what has worked and what this means for the people involved. Tina Nabatchi, in A manager’s guide to evaluating citizen participation, explains that evaluation can help managers understand ‘what type of participation, under what circumstances, creates what results’. The Environmental Protection Agency’s brochure ‘How to Evaluate Public Involvement’ makes practical suggestions for evaluation to help the agency improve its public involvement activities.
Evaluation frameworks are responsive to the objectives of participation
The Chittenden County Regional Planning Commission’s Evaluation Matrix suggests ways to measure the success of a strategy according to where it belongs on a spectrum of participation. Evaluation criteria can also be chosen based on standards for the quality, innovation, and inclusivity of participation, as Carolyn Lukensmeyer, J. P. Goldman, and David Stern do in ‘Assessing public participation in an open government era’.
Evaluation can be process-oriented and/or outcome-oriented
Nabatchi’s guide features two types of evaluation, with step-by-step illustrations of what to measure, and how to measure: process evaluation, which reviews program management, and impact evaluation, which reviews program outcomes. Gemma Carr, Günter Blöschl, Daniel P. Loucks sort evaluation methods into three broad groups in ‘Evaluating participation in water resource management: A review’. Process evaluation focuses on the quality of participation; intermediary outcome evaluation looks at side benefits of the process such as social capital and trust; resource management outcome evaluation measures change and improvement. In a public health context, Frances Dunn Butterfoss’ ‘Process evaluation for community participation’ explores process evaluation methods and criteria as a way to measure the ‘extent, quality, and fidelity’ of programs.
Evaluation can reveal social outcomes that transcend a project
Social learning, or the exchange of knowledge between citizens, experts, and decision-makers is the fourth dimension in Bing Ran’s adapted model in ‘Evaluating public participation in environmental policy-making’. Process social learning fosters solidarity and breaks down barriers between people, while outcome social learning brings about attitude changes, learning, and capacity building. Thomas C. Beierle ‘s model in ‘Public Participation in Environmental Decisions: An Evaluation Framework Using Social Goals’ expands outcome evaluation to look at the broader social concerns and benefits of a project: educating the public, shared decision-making, trust-building, conflict resolution.
Participant perceptions can inform and strengthen evaluations
Citizen satisfaction and understanding of information, risks, agency response, and opportunities are at the centre of an evaluation of a community involvement program in hazardous waste management for the Environmental Protection Agency, reported in Susan Charnley and Bruce Engelbert’s ‘Evaluating public participation in environmental decision-making’. Thomas C. Beierle’s case study of a two-week online deliberation, ‘Democracy On-line. An Evaluation of the National Dialogue on Public Involvement in EPA Decisions’, includes evaluation questions based on participant satisfaction, experiences, and benefits (learning, networking).
Surveying participants who appealed agency decisions, Rene et al investigate participant (dis)satisfaction in ‘Public perceptions of the USDA Forest Service public participation process’. Stakeholders helped researchers at the Oak Ridge National Laboratory select key attributes of successful participation in environmental management in a study for the US Department of Energy: ‘Measuring the accomplishments of public participation programs: Overview of a methodological study performed for DOEs Office of Environmental Management’ – ‘perceptual’ attributes listed relate to how stakeholders view the program’s performance.
Evaluation can borrow from and lend to fields of practice
John B. Stephens and Maureen Berner build on overlapping values and shared objectives in ‘Learning from your neighbor: The value of public participation evaluation for public policy dispute resolution’ and review evaluation models in public participation to pick out transferable criteria. The Center for Urban Transportation Research draws from performance measurement studies to develop an evaluation framework for public involvement in the Florida Department of Transportation, as reported in ‘Performance measures to evaluate the effectiveness of public involvement activities in Florida’.
Photo: Bethany Legg/Unsplash/cc