BTT: What are ‘Feedback Frames’ and what problem do they help solve?
Jason Diceman: Feedback Frames are a large group decision making tool. They are secret rating ballots with instant visual results. Participants drop tokens along a custom rating scale to secretly and anonymous record their opinions on any number of ideas or other content for gathering feedback. Results are revealed at the end for everyone to see as visual graphs.
Feedback Frames are a major upgrade on traditional multi-voting with stickers or check marks for several reasons: votes are secret during the process, which avoids the bandwagon effect and the likelihood of “groupthink” and each option is rated independently. As a result, there is no concern of vote splitting between similar or related options and no problem with participants suffering from choice overload. Signatures validate the number of tokens and minimize the potential of cheating or ‘ballot stuffing’.
Because they are non-electronic hardware, they are very reliable and easy to set-up. There are no network connection, passwords, or batteries to worry about. Online tools make a lot of sense for distributed participants, but for face-to-face meetings, Feedback Frames are a natural upgrade from flip charts and sticky notes.
BTT: What sort of user testing was used to create the prototypes? In what contexts have the prototype Feedback Frames been used?
JD: Feedback Frames are based on my earlier invention, ‘Idea Rating Sheets’ (previously known as Dotmocracy Sheets). Both Feedback Frames and Idea Rating Sheets use the same approach of collecting written statements, collectively rating them on a simple scale, using signatures to validate results and with options for comments.
Idea Rating Sheets have been adopted by hundreds of meeting facilitators in over 33 different countries, in more than 15 different languages, used by government, education, business and community groups. Feedback Frames are an upgrade on the sheets because they keep votes secret during the process, which avoids the bandwagon effect and the likelihood of ‘groupthink’, making collective results much more reliable.
I tested the use of Feedback Frames prototypes in wide range of real scenarios including: government directors’ strategy workshop; a high-school English class; a scientific conference session; family discussions (children to seniors); and a facilitators’ best practices workshop. I also have plans to use them for public consultation, professional sector conferences, church meetings, agile programmer meetings, business innovation and more.
BTT: At what point would Feedback Frames be introduced into a decision process? Where do I use this in an overall engagement (particularly workshop) methodology?
JD: Feedback Frames can be used in almost any stage of a process:
- Short listing topics to investigate
- Observing pre-discussion existing opinions and perspectives
- Recognizing preferences, levels of agreements and outstanding areas of conflict midway through a deliberation
- Prioritizing options to inform decision making
- Voting to formalize a final group decision
For a simple question, using Feedback Frames could be the whole process. For more complicated topics, Feedback Frames could be inserted at strategic stages to support a more involved deliberative process.
BTT: How would you recommend using the data that is produced?
JD: I recommend taking photos of the results for each option and entering the data in to a spreadsheet. You can then easily sort the results by an agreement score and share the results (images and spreadsheet) with decision makers, participants and observers. This makes the results transparent to everyone, which promotes responsive and accountable decision making.
BTT: Did you notice any differences in the way different people or groups used the frames?
JD: There is a tendency of some meeting planners to want to pre-determine the options to be rated using Feedback Frames. While this can work, using an online or paper survey could achieve similar results. Where Feedback Frames uniquely excel is in allowing participant to rate content generated within the same meeting, that is, start with blank sheets for participants to suggest statements for others to rate. This way participants can own all the outputs.
BTT: Did you notice any behaviors that would warrant special instructions to help facilitators to get better results?
JD: The key factors that can influence meeting results using Feedback Frames are first and foremost the factors that affect the quality of every deliberative meeting: having the right people in the room, with right information, a skilled facilitator, and a smart meeting plan. Specific to Feedback Frames, you need to include in your agenda time dedicated to dropping tokens, and table space to set-up the Frames. More time can also allow participants to note comments along with their token drop.
There is a tendency to underestimate the number of Frames you’ll need. Imagine if you were brainstorming with sticky notes and how many sticky notes you would see posted – that’s about how many Frames you should have ready to use in the meeting. Before and during the token dropping process you will need to remind participants to drop one token and then sign. Some may overlook collecting signatures, but this is necessary to confirm there has not been any ballot stuffing. The signatures are also handy to see how many tokens have been dropped in each Frame, without peaking at the results.
Depending on the number participants, options for rating, and how much time is given for dropping tokens, you may only get a sample (a fraction) of all the participants opinions on each option, and that’s okay. As long as that sample of participants is random, and not a voting block, and that the sample is consequential, e.g. more than 15 or 20. The maximum number of tokens per column is about 30.
BTT: Are there people who can’t use it? What sort of instruction is required?
JD: Because the rating scale and content can all be customized, it is easy to modify the options and interface to be appropriate for your participants. You could even use exclusively symbols (with no text) for non-literate participants. For participants who are visually or physically impaired, they may require an assistant to read, write and drop tokens on their behalf.
BTT: How is ‘sense-making’ of results achieved?
JD: The results of the token dropping is determined not by the absolute number of tokens in any column, but the relative pattern of tokens in each Frame. For example, are the opinions spread out across the spectrum, focused on one polarity, in the middle, or polarized?
Interpreting and comparing results between the various options that were rated really depends on the topic. Some solutions may be able to apply and combine many of the top agreements, while other topics may require one option to be chosen, to the exclusion of others. The philosophy of the organization’s decision making will also effect interpretation, for instance, what’s more important, consensus (lack of objection), popular support (majority rule), or enthusiastic support (average level of strong agreement). The good thing is that the results are visible to everyone, it’s just a matter of interpretation, which can represented mathematically and/or with descriptive comments.
The rating scales are 100% customizable and flexible. If you want to simplify your scale to just two options like yes/no, you can. If you want to have a longer scale of 12 options (like for choosing a month to host a special event) you could combine multiple frames. You could have multiple Frames with different scales for rating different characteristics on each option, such as one Frame for ‘important’ and one for ‘urgent’.
I’m currently building a mailing list of people who are interested in using Feedback Frames and can be contacted at firstname.lastname@example.org.
Jason Diceman is a professional meeting facilitator and full-time Senior Public Consultation Coordinator for the City of Toronto. More information can be found here: feedbackframes.com