Do I need a representative sample for my consultation?
If you’ve ever had a member of your community challenge the validity of your engagement results, you will have first-hand experience in understanding the importance of being able to support your findings with a solid engagement methodology.
This often includes being able to pinpoint and explain how your findings are representative of your community.
Understanding when a representative sample of your community might be required in community engagement is an important skill to learn to ensure better outcomes from your engagement activities.
In this article, we look at when to use sampling in your engagement practice, which types of sampling you should think about, as well as describing the key differences between community development and research led approaches to engagement.
We will also look at the advantages and disadvantages of outsourcing sampling practices to market research agencies and contractors.
So when do I need a representative sample?
The answer to this question lies in the aims of your research project or engagement objectives.
It’s not until you have spent considerable time unpacking the aims and objectives of your project as well as the potential impacts of your decision, that you will be able to determine if you require a representative sample of your population and what type of sample that needs to be.
Generally speaking, a representative sample is useful if you are conducting broad population research on an issue or set of issues and you want to ensure any decision made based on that research can be reinforced with an appropriate research methodology.
In practice, these are often engagements that deal with strategic planning or business planning, services and asset management, annual community satisfaction surveys and even major transport studies.
These projects are all similar in that they each impact on the community as a whole and so a broad cross-section of the community must be represented in order for appropriate decisions to be made.
This is in contrast to much smaller impact consultations that are either aimed at community development (relationship building), behaviour change or more general and smaller scale issues where interested parties can self-select to take part.
An important consideration when making any judgement about whether a representative sample is required for you project is to ask, does the community need any prior knowledge of the issue or do they need to be informed to take part in the consultation?
With engagements that require a representative population sample, participants don’t usually require tacit knowledge of the project because population sampling is generally a research focussed methodology that can be completed without project specific knowledge.
This is different to localised issues which don’t require representative sampling because participants self-select to be involved based on their interest and prior knowledge of a project.
What’s the difference between community development and research led approaches to engagement?
Put simply, the main difference between community development focussed engagement and research led approaches is the inclusion of relationship building objectives versus a tightly focussed one-way intercept style approach as with research led approaches.
In 2009, our CEO Matt Crozier wrote a fantastic article title Market Research vs Community Engagement outlining some of the main differences between the two approaches to engagement.
These included some crucial observations as made by Phillip Sheldrake from his article Continuous engagement… the death of market research.
Here’s a list of the primary differences: Research is ad hoc or regular interval; engagement is continuous; Research is one-way (+ prize or payment!); engagement is two-way (mutually rewarding); Research is unemotional; engagement is emotional; Research is independent of loyalty; engagement inculcates brand loyalty; Research has a tight focus; engagement has a wide focus; Research deals with sequential parameters; engagement is multi-parametric; Research is designed to achieve statistical confidence; engagement is designed to detect weak signals.
Understanding what you are trying to achieve through your engagement is essential in developing your strategy and making the decision to commit resources to either practice.
I’m doing research led engagement. Which sampling technique should I use?
There are many different types of sampling techniques which are employed in research led engagement. Knowing which method to use is important when conducting research on your own and having a general knowledge of the different methods can help you work better with research agencies who might be undertaking the work for you. Below is a summary of the two most common sampling techniques;
Simple Random Sample
Simple Random Sampling (SRS) treats each unit of a population with equal opportunity for selection. Using this method will give you a basic indication of what a statistically significant sample looks like for your test population. This method is most useful for small sample populations and is rarely used for larger samples where more accurate and complex methods for establishing sample sizes are required.
The National Statistical Service in Australia provides an easy to use SRS calculator that can be used for your projects.
Stratified Random Sample
When the use of auxiliary information about your population is required, in order to inform your engagement research, such as age, sex or location, Stratified Sampling provides the best methodology to use.
Stratification essentially means to divide your population into homogenous strata or similar groups and then using SRS within each stratum to work out your test group. With stratified sampling, the definitions of the boundaries of each strata should be precise and unambiguous and the total sample size will be the sum of the samples for each strata.
As an example, if we were interested in our communities views towards council service delivery objectives, we could select a sample based on the community as a whole, or we could select samples independently from each of the suburbs that make up the community. Using this latter approach we can ensure that an appropriate amount of people in each suburb are represented, which would not necessarily occur with a simple random sample.
Of course, using this method also presents issues such as increased costs, danger of over-stratification and even challenges in finding appropriate information about each strata.
This method is the preferred method for most statistical bureaus and you can read more about how to do stratified samples here.
Alternatively, if you are feeling up-to it, you can watch a great explainer video from the Kahn Academy below. It provides a simple refresher on sampling and bias and is well worth the watch.
For more information on sampling and setting up research based engagement check out these resources;
- National Statistical Service – Basic Survey Design Manual
- Food and Agriculture Organisation of United Nations – Market Research and Information Systems. Chapter 4 on survey design and Chapter 7 on sampling are particularly relevant.
- Stats NZ – A guide to good survey design
- National Audit Office UK – A Practical Guide to Sampling
- Explorable – Sampling
Should I do the sampling myself or outsource it?
If you have the skills available to conduct sampling and research internally in your organisation, it is highly recommended you do so. Ensuring that skills are frequently put into practice and organisational learning and knowledge sharing is happening in your organisation is essential for any modern learning organisation.
It might be advisable to use external market research providers in instances where the skills internally don’t exist or if your research is about a particularly sensitive and important issue where you would like it to be seen as independent from the research.
Obviously, outsourcing work comes with costs and its own set of risks, so it will be important if you do go down this path to ensure that you work closely with the provider to properly plan out your engagement plan and set clear boundaries around who will deliver each component of the work.
The simply conclusion to this story is that you don’t always need a representative sample of your community for your community engagement projects.
Remember, engagement is as much about relationship building, community learning and deliberation as it is about surveying and research focussed engagement.
For those projects where you require some rigour, choose the best sampling method and make sure you’re able to explain your methodology to members of your community.
Learning sampling and survey design skills will go a long way in your community engagement practice and help you get the most out of your larger more strategic consultations.