How to Analyze Consultation Feedback: Thematic Analysis and AI

To analyze consultation feedback, read everything once, code each comment against a coding frame built from the decision you are consulting on, group the codes into themes, check those themes against the raw text, and keep every theme traceable to the comments behind it. That traceability is what lets you report honestly, explain how feedback shaped the decision, and answer a challenge months later.

This guide is the hands-on version. It walks through thematic analysis of consultation feedback and open-ended survey responses step by step, shows a small worked example, and covers how to use AI tools such as ChatGPT or Copilot without losing the thread from comment to conclusion. For the wider picture of delivery, coding frameworks and governance, see our analysis guide.

Before you start: what do you need in place?

Most analysis problems are set up before the first comment is read. Fifteen minutes of preparation saves days of rework and makes the final report far easier to defend.

Be clear on the decision

Write down, in a sentence or two, the decision this engagement is meant to inform and which parts of it were genuinely open to influence. Your analysis exists to serve that decision. If the consultation asked about the layout of a park but the budget was fixed, comments about budget still matter, but they belong in a different part of the report from comments that can change the design.

Go back to the questions you asked

List the questions people were actually answering, word for word. An open-ended response only makes sense alongside its question. "Too many" means something different under "What do you think of the proposed parking?" than under "What concerns do you have about events in the park?" If the same comment field appeared on several pages of a survey, record which page each response came from. (If you are still designing the survey, our guide to community survey questions covers how to ask questions that are easier to analyze.)

Bring every channel into one dataset

Online surveys, emails, written submissions, workshop notes, phone calls and pop-up conversations all count. Put them in one place with a stable ID for each response, plus the channel, date and any participant details your privacy commitments allow. If the survey export sits in one spreadsheet and the letters sit in an inbox, the analysis will quietly favor the spreadsheet.

Clean and de-duplicate, carefully

Remove blank and test responses, and flag (rather than delete) exact duplicates from the same person. Keep the original text untouched in its own column, and do any cleaning, translation or transcription in a separate column. Remove or mask personal information that is not needed for the analysis, such as phone numbers or a neighbor's name mentioned in a complaint.

Identify organized campaigns and form letters

Campaigns, petitions and form letters are a legitimate way for people to take part, so do not discard them. Group identical or near-identical submissions, record how many there are, and analyze the shared text once. Then read each one for anything personal added to it, because the sentence someone typed above the template is often the most useful part. In your report, state the campaign separately: how many submissions used the shared wording, what it said, and any distinct points individuals added.

How do you do thematic analysis of consultation feedback, step by step?

Thematic analysis is a widely used qualitative method for finding, organizing and reporting patterns of meaning across text. Academic versions vary in their philosophy, but applied to consultation the practical steps are similar. Here is a sequence that works for a few dozen comments or a few thousand.

1. Familiarize yourself with the material. Read a broad sample before coding anything: responses from every channel, long and short, early and late. Note first impressions, surprises and anything you do not understand. For smaller datasets, read everything.

2. Develop a coding frame. Start deductively, from the decision and the questions asked: if people were asked about access, safety and facilities, those are likely starting codes. Then add codes inductively as the material shows you things you did not expect. Define each code in a sentence, with an example of what belongs and what does not.

3. Code the responses. Apply codes to each comment. A single comment can carry several codes, so code the parts of a comment rather than forcing the whole thing into one box. Keep a column for the response ID next to every code so nothing comes loose from its source.

4. Review and merge codes. After a first pass, look for codes that overlap, codes used only once or twice, and catch-all codes such as "other" or "general concern" that have grown too large. Merge, split or redefine, then log each change with the reason. If a definition changes, recheck the comments coded under the old one.

5. Name the themes. Group related codes into themes. A good theme name says something, such as "Residents want lighting on the path for evening use", rather than just naming a topic such as "Lighting". Topics are where comments cluster; themes are what people are telling you.

6. Check against the raw text. For each theme, go back and reread a spread of the original comments behind it. Does the theme name fairly describe what people said? Are there comments coded to it that actually argue the opposite? Ideally, have a second person code a sample independently and compare.

7. Quantify carefully. Counts help readers see the shape of the feedback, but counts are not votes. Report how many responses raised a theme, not "how many people support" something, unless the question actually asked about support. Say what the denominator is (all responses, or only those who answered that question), and remember that a consultation sample is self-selected, not representative.

8. Capture minority and outlier views. Look specifically for points raised by only a few people, especially those directly affected, from groups you struggled to reach, or offering information the project team did not have. A single comment from the owner of the house next to a proposed path can matter more than fifty general comments about the path. Give these their own place in the analysis rather than letting them disappear into "other".

9. Record how themes informed the decision. For each theme, note what the project will do in response: a change, something already addressed, something out of scope (and why), or something passed to another team. This is the link participants will look for in the report.

A spreadsheet is enough for many projects: one row per response or per coded segment, with columns for the ID, channel, question, original text, code or codes, theme and notes. What matters is not the tool but that every theme can be traced back to specific rows.

What does coding consultation feedback look like in practice?

Here is an example for a hypothetical upgrade to a neighborhood park, where the survey asked: "What would make the park work better for you?" The comments are invented for illustration.

Notice three things. R033 and R072 pull against the lighting theme, so the analysis has to report a tension, not a simple request. R072 comes from someone directly affected, which changes how much weight it carries even though it is one comment. And R089 is a single comment, but it describes a barrier that may affect people the consultation struggled to reach, so it is flagged rather than buried.

Can you use AI to analyze open-ended survey responses?

Yes, and for large volumes it can help a great deal, provided a person stays responsible for the coding frame, the interpretation and the conclusions. The distinction matters. As we argue in reading is not coding, the risk is not that a model assigns labels. It is that a model asked to "find the themes" designs the framework itself, invisibly, with nobody on the record for its assumptions.

What AI does well

• Reading a large volume of text quickly and consistently, without the fatigue that sets in after a few hundred responses.

• Noticing that many people are making the same point in different words.

• Proposing a first draft of codes or topics for a person to review.

• Applying an agreed coding frame across thousands of responses once you have defined it.

• Pulling out example quotes for each theme, if you ask it to.

What goes wrong

• Invented or overstated themes. A general chatbot can produce a theme that sounds plausible but is supported by only one or two comments, or by none.

• Lost minority views. Summaries naturally favor what is common. The single comment about wheelchair access, or the adjoining resident's concern, can vanish entirely.

• Confident summaries that cannot be traced. A neat list of five themes with percentages attached looks like evidence. If you cannot point to the comments behind each one, it is not.

• Truncated input. If the dataset is larger than the tool can handle in one go, some responses may never be read, and the output will not always tell you.

• Smoothed-over tension. Models tend to resolve disagreement into a tidy summary, when the honest finding is that people disagree.

How to use ChatGPT or Copilot for survey analysis responsibly

1. Check privacy first. Do not paste personal information into a tool your organization has not approved for that purpose. Use your organization's approved assistant, and remove names, addresses and contact details before analysis even then. Enterprise versions offer stronger protections than consumer accounts, but approval is your organization's call.

2. Give every response an ID before you start, and keep the IDs in the text you provide, so the output can be traced back.

3. Provide the context: the decision, the question asked, and your draft coding frame if you have one. A model working without context will invent its own.

4. Ask for themes with supporting quotes and IDs, and for a count of responses per theme. Ask it to list comments that do not fit any theme, and to flag comments that disagree with each theme.

5. Spot-check against the source. For each theme, look up a sample of the cited IDs and confirm the quotes are real and the theme fairly describes them. Search the raw data for words linked to likely minority views and check they were not dropped.

6. Keep a human sign-off. A named person should review each theme, confirm the interpretation and stand behind it before it goes into a report.

For a closer look at where general assistants fit and where engagement work needs more, see CE Canvas, Copilot and ChatGPT compared.

How do you make consultation analysis defensible?

Defensible analysis is analysis someone else can follow. If a councillor, a board member, an objector or an auditor asks why a theme was reported the way it was, you should be able to show them the comments, the codes, the reasoning and the person who signed it off. Our page on defensible community engagement covers this across the whole process; for analysis, it comes down to a few habits.

• Traceability: every theme links to the response IDs behind it, and every response can be found in its original form.

• An audit trail of the coding frame: what each code meant, when it changed and why, and who approved the change.

• Documented judgment calls: how you handled campaigns, duplicates, off-topic comments and ambiguous responses, written down rather than left in someone's head.

• A record of who did what: who coded, who checked, which tools were used (including any AI), and who signed off.

• A decision log: for each theme, the decision it bears on and what the project did in response.

None of this needs to be elaborate. A short methods note and a well-kept spreadsheet will answer most challenges. What will not hold up is a summary written from memory, or a set of AI-generated themes that nobody can trace back to source.

How do you report what you found?

Lead with the themes, not the method. For each theme, give a plain description, an indication of how often it came up (with the denominator), a few representative quotes, any significant differences between groups or channels, and what the project is doing in response. Report campaigns separately, give minority and high-impact views their own space, and be honest about who you did not hear from.

Keep the methods short and in one place: how feedback was collected, how many responses came through each channel, how they were coded, how AI was used if at all, and who reviewed the findings. Our community engagement report guide gives a section-by-section template for the what we heard report itself.

Where CE Canvas fits

If you are doing this regularly, the hardest part is usually keeping the thread from comment to theme to decision across channels and across a team. CE Canvas clusters responses into topics with every comment one click away, and your team decides what becomes a theme. Findings show their sources, and drafts wait for a person to sign off. You can see the flow on how it works. The method in this guide works just as well with a spreadsheet and a careful team.

Frequently asked questions

How do you analyze open-ended survey responses?

Read a broad sample first, build a coding frame from the questions and the decision (then add codes as the responses show you new things), code each response, group codes into themes, and check every theme against the original comments. Keep a response ID next to every code so each theme can be traced back to the comments behind it.

What is thematic analysis?

Thematic analysis is a qualitative method for identifying, organizing and reporting patterns of meaning in text. In consultation work, it means coding comments and grouping those codes into themes that describe what people are telling you, so the findings can be reported and traced back to what was said.

Can ChatGPT analyze survey responses?

It can help, especially with large volumes, but it should not be left to do the analysis on its own. Use only a tool your organization has approved, remove personal information, give each response an ID, ask for themes with supporting quotes and IDs, spot-check the output against the source, and have a named person sign off the findings.

How do you report the number of people who raised a theme?

Report the number of responses (or submitters, if you can tell) that raised the theme, and state the denominator, such as "of the responses to this question". Avoid language that implies a vote, such as "most people support", because consultation participants are self-selected and a count of mentions is not a measure of support.

How do you handle form letters and organized campaigns?

Keep them, group them, and analyze the shared text once. Record how many submissions used the campaign wording, read each one for anything individuals added, and report the campaign separately from other feedback so its views are visible without being confused with independent responses.