All work
152022MoHRE

Regulatory & Policy Change Watcher

Watching 30+ regulator sites every day so a compliance team doesn't have to — and only pinging them when it actually matters.

Before

The compliance team manually scanned 30+ regulator websites for changes affecting labour-services applications. It was slow, easy to miss things, and impossible to do consistently every day.

After

A daily crawler + LLM diff summariser posts only material policy changes — with citations — to a Teams channel. Review effort dropped ~80%, and it has run for 3+ years with under 5 false-positive flags total.

Challenge

Compliance team had to manually scan 30+ regulator websites for changes that affected labour-services applications.

Approach

Daily crawler + LLM diff summariser that posts only material policy changes — with citations — to a Teams channel.

How it was built

  1. 1

    Source mapping

    Phase 1

    Catalogued the 30+ regulator sources that actually affect labour-services work and how each publishes changes, so the crawler watches the right pages rather than everything.

  2. 2

    Change detection

    Phase 2

    Built a daily crawler that snapshots each source and diffs against the prior version — the cheap, deterministic layer that catches that something changed before any LLM is involved.

  3. 3

    Materiality summarisation

    Phase 3

    An LLM summarises each diff and judges whether it's material to labour services, with citations back to the source — turning raw diffs into a decision-ready brief.

  4. 4

    Signal-only alerting

    Phase 4 — Present

    Only material changes are posted to a Teams channel. Tuning the materiality bar down to near-zero false positives is what earned the team's trust to actually read every alert.

Key architecture decisions

Deterministic diffing before the LLM

Why · Detecting that a page changed is a cheap, reliable job for classical diffing. The LLM is reserved for the hard part — judging whether the change matters.

Citations on every alert

Why · Compliance can't act on an unverifiable summary. Linking straight to the changed source is what makes the alert usable, not just informative.

Optimise for precision over recall of noise

Why · An alerting system people ignore is worse than none. Keeping false positives under 5 in three years is why the channel still gets read.

Impact

  • Compliance review effort cut by ~80%
  • Caught two high-impact regulatory changes before manual review would have
  • Running for 3+ years with under 5 false-positive flags total
-80%
effort
100%
recall
<5
falsepos

What I'd tell someone building this

  • 01 · Split the cheap deterministic step (did it change?) from the expensive reasoning step (does it matter?).
  • 02 · For alerting, precision beats recall — one noisy week and people mute the channel.
  • 03 · Citations turn an LLM summary from 'interesting' into 'actionable'.
  • 04 · Longevity is the real proof: a tool running quietly for 3+ years says more than a launch metric.

Tech stack

PythonGPT-4Microsoft TeamsAzure Logic Apps

Ask anything about Regulatory & Policy Change Watcher

AI scoped to this project · GPT-OSS 120B