All work
112024SCAD

Open-Ended Survey Response Coder

Coding 30,000+ Arabic open-ended responses in 4 days instead of 6 weeks — at 92% agreement with humans.

Before

Coding 30,000+ Arabic open-ended survey responses to a fixed taxonomy took a team of four about six weeks.

After

An Arabic-first, few-shot LLM coder with confidence thresholds auto-codes the confident majority and routes the rest to a human-in-the-loop UI. It cut the cycle from 6 weeks to 4 days at 92% agreement with the human gold standard.

Challenge

Coding 30,000+ Arabic open-ended responses to a single fixed taxonomy used to take a team of 4 about 6 weeks.

Approach

Few-shot Arabic-first LLM coder with confidence thresholds; high-confidence answers auto-coded, low-confidence ones routed to a human-in-the-loop UI.

How it was built

  1. 1

    Taxonomy + few-shot

    Phase 1

    Encoded the fixed taxonomy as few-shot examples, Arabic-first, so the model codes to the exact categories the survey team uses.

  2. 2

    Confidence thresholds

    Phase 2

    Auto-coded only high-confidence responses and routed the rest to a review UI — automation where it's safe, humans where it's not.

  3. 3

    Human-in-the-loop review

    Phase 3

    Focused the four-person team on edge cases instead of the full 30,000, which is where the six weeks used to go.

Key architecture decisions

Confidence-gated automation

Why · Auto-coding everything would be unsafe; reviewing everything is slow. Thresholds let the model handle the bulk while humans own the ambiguous tail.

Arabic-first prompting

Why · Treating Arabic as the primary language, not a translation target, is what held agreement at 92% on real responses.

Impact

  • Reduced coding cycle from 6 weeks to 4 days
  • 92% agreement with the human gold-standard sample
  • Freed the survey team to focus on edge-case review
10×
speedup
92%
agreement
30K+
responses

What I'd tell someone building this

  • 01 · Confidence thresholds turn 'risky automation' into 'safe automation plus focused review'.
  • 02 · Measure against a human gold standard or you can't claim accuracy.
  • 03 · For Arabic content, design Arabic-first rather than translating to English underneath.

Tech stack

GPT-4Azure OpenAIArabic NLPAngular

Ask anything about Open-Ended Survey Response Coder

AI scoped to this project · GPT-OSS 120B