All work
102024SCAD

Statistical Data-Quality Anomaly Detector

Classical stats to flag it, an LLM to judge it — catching unit-of-measure errors before they hit published statistics.

Before

Monthly economic-indicator submissions occasionally contained unit-of-measure errors that weren't caught until after publication.

After

A hybrid pipeline flags suspect rows with classical outlier detection, then GPT-4 reasons about whether they're real changes or likely data-entry mistakes. It caught 23 publication-blocking issues over 9 months, cut false positives 60% vs the old threshold-only system, and drove data-quality publication delays from 4/year to 0.

Challenge

Monthly economic indicator submissions occasionally contained unit-of-measure errors that weren't caught until publication.

Approach

Hybrid stats + LLM pipeline: classical outlier detection flags suspect rows, then GPT-4 reasons about whether they're real changes vs. likely data-entry mistakes.

How it was built

  1. 1

    Statistical flagging

    Phase 1

    Kept classical outlier detection as the first pass — cheap, explainable, and good at surfacing suspect rows.

  2. 2

    LLM adjudication

    Phase 2

    Layered GPT-4 reasoning on top to distinguish a genuine economic shift from a decimal-point or unit error — the judgement a pure threshold can't make.

  3. 3

    Pre-publication gate

    Phase 3

    Wired it into the pre-release workflow so issues are caught before publication, not discovered weeks later in the wild.

Key architecture decisions

Hybrid stats + LLM, not either alone

Why · Thresholds over-flag; an LLM alone is expensive and unfocused. Stats narrow the candidates, the LLM adds judgement — cutting false positives 60%.

Explain every flag

Why · Analysts need to know why a row was flagged to act on it. Pairing the statistical signal with the LLM's reasoning made flags actionable.

Impact

  • Caught 23 publication-blocking issues across 9 months pre-release
  • Reduced false positives by 60% vs. the prior threshold-only system
  • Cut publication delays linked to data quality from 4/yr to 0
23 issues
caught
-60%
fp
0
delays

What I'd tell someone building this

  • 01 · Use classical methods for recall and the LLM for judgement — don't make the model do both.
  • 02 · False positives are the real enemy of a data-quality tool; tune for them.
  • 03 · Catching an error before publication is worth far more than detecting it after.

Tech stack

PythonGPT-4scikit-learnAzure Functions

Ask anything about Statistical Data-Quality Anomaly Detector

AI scoped to this project · GPT-OSS 120B