Public Participation Evaluation is essential to testing assumptions, strategies and outcomes of public participation processes. But it remains an overlooked – perhaps undervalued – element of the planning and engagement process. Not only valuable for clarification and accountability – how engagement processes unfold – evaluation considers the why of engagement. It provides insights and reflections that improve practices.
Sourced from online guides and manuals, critical reviews, hands-on case studies and current policy practice, this Primer provides an evolving resource to support and squarely face evaluation as a core methodology of the public participation process.
Building out our Engagement Primers – International public participation models 1969-2016, and International frameworks for citizen engagement and civil society – Public participation evaluation sets out 14 evaluation frameworks across government and non-government sectors, including public health, sustainability, and town and transport planning. It traces the emergence of evaluation frameworks and criteria and provides new knowledges in current public participation practice across the USA.
Public participation evaluation frameworks
1. ‘CCRPC Evaluation Matrix’ CCRPC Public Participation Plan, 2014, Chittenden County Regional Planning Commission
The CCRPC Evaluation Matrix offers evaluation criteria and performance goals for organisational and project-specific participation techniques, mapped to inform, consult, or involve the participants. For example, it identifies the CCRPC website as a strategy to inform the public, and measures it by the number of visitors. Surveys to consult the public are evaluated for the quantity of expected and received responses.
2. ‘Evaluating public participation in environmental policy-making’, Journal of US-China Public Administration, 2012, Bing Ran
Bing Ran builds on Rene H. Germain, Donald W. Floyd and Stephen V. Stehman’s 2001 model, which measures citizen satisfaction by the factors of equity, effectiveness, and efficiency in participatory processes and outcomes. Ran adds a fourth factor to these variables – social learning: the exchange of knowledge between decision-makers, experts, and citizens. Ran’s revised model measures process social learning by how it fosters convergence and solidarity between stakeholders and outcome social learning by how it encourages behaviour change, problem-solving, and capacity building.
3. A manager’s guide to evaluating citizen participation, IBM Center for the Business of Government, 2012, Tina Nabatchi
In this practical manual for public managers, Tina Nabatchi argues that the goals and design of participation need evaluative frameworks that can work across diverse settings while remaining specific enough to be useful to research and practice. The guide presents two types of evaluation and includes step-by-step illustrations of the different design objectives and data for assessment. Process evaluation focuses on program management to identify gaps between design and delivery and areas for improvement. Impact evaluation looks at program outcomes to assess how a program achieves its objectives. The report is targeted to managers to support thinking through elements crucial for evaluating citizen participation processes and in conducting evaluations, “helping managers determine whether, where, when, why, and how to engage in direct citizen participation efforts.”
4. ‘Evaluating participation in water resource management: A review’, Water Resources Research, 2012, Gemma Carr, Günter Blöschl, Daniel P. Loucks
In the context of water management, Carr, Blöschl and Loucks categorize evaluation methods into three groups: process evaluation focuses on the quality of participation, addressing criteria such as accountability, inclusion, and legitimacy; intermediary outcome evaluation identifies the ‘side benefits’ of participation such as the development of trust, shared knowledge, and social capital; and, resource management outcome evaluation identifies achievements such as ecological improvements, consensus, and conflict resolution.
5. ‘Learning from your neighbor: The value of public participation evaluation for public policy dispute resolution’, Journal of Public Deliberation, 2011, John B. Stephens, Maureen Berner
Stephens and Berner review evaluation frameworks in public policy dispute resolution (PPDR) and public participation (P2), identifying opportunities to transfer P2 frameworks into PPDR evaluations. Selecting P2 evaluation models, the authors organize their criteria into process, outcome, and cost indicators. Drawing on parallels between them, they highlight general and project-specific factors from P2 evaluation that could inform PPDR evaluation, and present examples of works that bridge the fields of practice.
6. ‘Assessing public participation in an open government era’, A Review of Federal Agency Plans, USA: IBM Center for The Business of Government, Fostering Transparency and Democracy Series, 2010, Carolyn Lukensmeyer, J. P. Goldman, David Stern
Lukensmeyer, Goldman, and Stern present evaluation criteria for public participation in agency open-government plans based on standards for ‘good’ participation such as information accessibility, neutral framing of issues and diversity. The criteria revolve around the levels, quality, innovation, and inclusivity of participation that agencies build into their programs, represented through a set of eight questions. The report identifies the types of participation in agency plans and details participation activities in selected high-performing agencies.
7. Performance measures to evaluate the effectiveness of public involvement activities in Florida, Center for Urban Transportation Research, 2007, Rusty Ennemoser, Jeff Kramer, Kristine Williams, Christina Hopes
Developed for the Florida Department of Transportation (FDOT), this technical memorandum offers an extensive literature review focused that includes public-involvement performance measures of state transportation agencies and metropolitan planning organizations. The findings highlight established definitions and components of performance measurements such as targets, outputs, and outcomes. Effective performance measurement systems are grounded in agency goals, supported by senior management and valid to public and decision-makers.
8. ‘Process evaluation for community participation’, Annual Review of Public Health, 2006, Frances Dunn Butterfoss
Butterfoss reflects on the use of process evaluation in public health community participation and interventions. It distinguishes process evaluation as a means of mapping the “extent, fidelity, and quality” of programs, how they are implemented and how they achieve their outcomes. The authors describe process evaluation methods – surveys, logs, interviews, focus groups, observation, document review – and select indicators of community participation used widely: participant diversity, member recruitment and retention, role in activities, quantity and type of events attended, time contributed to activities, benefits and challenges, satisfaction levels with the process, and perceptions of leadership and balance of power.
9. ‘Evaluating public participation in environmental decision-making: EPA’s superfund community involvement program’, Journal of Environmental Management, 2005, Susan Charnley, Bruce Engelbert
Charnley and Engelbert chronicle their experiences of an evaluation project in the US Environmental Protection Agency’s Superfund community involvement program, conducted over a period of eight years. The authors detail four criteria that revolve around public perception of resources, opportunities and responses: 1) citizen satisfaction with information provided; 2) citizen understanding of health and environmental risks; 3) citizen satisfaction with opportunities for community input; and 4) citizen satisfaction with the EPA’s response to their contribution. Their effectiveness evaluation approach focuses on how the public participation program aligns with the program goals set by the Agency.
10. How to Evaluate Public Involvement, 2003, United States Environmental Protection Agency (EPA)
Supplementing the US Environmental Protection Agency’s (EPA) Public Involvement Policy, this staff guide offers a starting point for developing evaluation processes and goals. The guide outlines evaluation criteria as: measuring information and access provided; outreach – including marginalised communities, motivations for participating or abstaining, participant perceptions, equity, cost-effectiveness of techniques, opportunities provided; and, capacity-building. This is in addition to achieving consensus, building trust and nurturing stakeholder relationships.
11. Democracy On-line. An Evaluation of the National Dialogue on Public Involvement in EPA Decisions, Resources For the Future, 2002, Thomas C. Beierle
Beierle evaluates a two-week online deliberation, the National Dialogue on Public Involvement in EPA Decisions, exploring various facets of participation processes and participant perceptions, and identifying value delivered to the agency and the public. The evaluation uses five primary questions, starting with participant satisfaction with the process and the prospect of future opportunities. Addressing the digital aspect, it asks if processes allow for new and diverse voices and examines the quality of communication online. It examines benefits for participants around influence, learning, and networking, and benefits for the agency in terms of learning, knowledge sharing, relationship-building, feedback and goodwill.
12. ‘Public perceptions of the USDA Forest Service public participation process’, Forest Policy and Economics, 2001, Rene H. Germain, Donald W. Floyd, Stephen V. Stehman
Germain et al investigate participant satisfaction in the processes and outcomes of USDA Forest Service consultations in the context of forest management conflicts. The assessment measures equity, effectiveness, and efficiency of process and outcome. In terms of process, equity identifies neutrality, impartiality, opportunity for input; effectiveness measures sufficient opportunities, consultation design; and efficiency, cost and time investments. By way of outcome, equity measures participant perceptions of fairness of the final appeal decision; effectiveness ascertains participant input, environmental and public interests represented by the final decision; while efficiency assesses time, cost and technical feasibilities.
13. Public participation in environmental decisions: an evaluation framework using social goals, Resources for the Future, 1998, Thomas C. Beierle
Beierle presents a social goals approach to evaluation with criteria to meet three distinct objectives: relevance to multiple participatory mechanisms, objectivity, and outcome-orientation. Describing social goals as those which transcend stakeholder interests to serve participants and the wider community, Beierle offers six social goals to form an evaluative framework: educating and informing the public; incorporating public values into decision-making; improving the substantive quality of decisions; increasing trust in institutions; reducing conflict; and, achieving cost-effectiveness. The outcome-oriented social goals approach addresses the broader question of how public participation benefits society.
14. Measuring the accomplishments of public participation programs: Overview of a methodological study performed for DOEs Office of Environmental Management, Oak Ridge National Laboratory, 1997, Martin Schweitzer, Sam A. Carnes, Elizabeth B. Peelle, Amy K. Wolfe
Schweitzer et al compile attributes and measures of successful public participation identified by stakeholders for the US Department of Energy’s (DOE) Office of Environmental Management. Respondents rate sixteen attributes of success gathered from open-ended interviews, grouped around five themes: the decision-making process; effects on stakeholder understanding and perception; effects on environmental management decisions; effects of decisions on site conditions; and, effects of decisions on stakeholder objectives. Building on these findings, the study presents seven attributes of success which address stakeholder representation, process legitimacy, mutual understanding of concerns, public trust, decision-making, decision legitimacy, and the achievement of mission-specific goals.