From rules to RAG: building an agentic AI compliance assistant for AML/CFT

From rule-based keyword matching to a fully deployed, RAG-grounded, tool-calling compliance Agentic AI assistant in 9 phases.

The Problem Worth Solving

New Zealand banks operate under some of the strictest AML/CFT obligations in the Asia-Pacific region. Every day, compliance analysts wade through hundreds of alerts — structuring patterns, PEP exposures, unusual wire transfers — each of which requires documented, defensible decisions.

The bottleneck isn’t knowledge. It’s consistency, volume, and documentation burden.

We set out to build an AI agent that could sit alongside an analyst — not replace them — and help them think faster, reference regulations accurately, and document their reasoning clearly.

The constraint we gave ourselves from day one: this agent must never execute a transaction, provide legal advice, or make a binding compliance decision. Every output carries an advisory disclaimer. Every escalation goes to a human.

That constraint turned out to be the most important design decision we made.

9 Phases, One Agent

We structured the build as a 9-phase capstone, treating each phase as a distinct engineering problem:

Phase 1 — Problem Framing. Define the user (NZ bank compliance analyst), map their workflow, and document anticipated failure cases before writing a single line of code. The failure cases we wrote upfront predicted 4 of the 5 bugs we later found in production.

Phase 2 — Baseline Agent. A rule-based keyword matcher. No LLM, no API calls. Just static pattern matching against a typology database. It passed 11/14 test cases and — crucially — failed in exactly the ways we predicted: synonym variation, no structured output, no memory, no document generation.

Phase 3 — LLM Integration. Swapped in GPT-4o-mini via OpenAI API. Compared three prompt strategies — zero-shot, structured JSON output, and few-shot with explicit safety rules. The few-shot + safety version (V3) was the only one that handled all six test query types correctly, including the synonym case that killed the baseline.

Phase 4 — RAG Retrieval. Built a ChromaDB vector store from 125,000 words of source material: the AML/CFT Act 2009 (as at 27 November 2025) and NZ regulatory guidelines from RBNZ, FMA, DIA, and FIU NZ. 370 chunks, 400 words each, 50-word overlap, embedded with sentence-transformers all-MiniLM-L6-v2 (384 dimensions). The before/after on citation quality was immediate and dramatic — hallucinated section numbers became grounded Act citations.

Phase 5 — Tool Calling. Four permitted tools: risk scorer, typology lookup, EDD checker, draft escalation note generator. Twelve explicitly prohibited tools — approve_transaction, transfer_funds, file_sar and nine others — blocked at the dispatcher layer before any code runs. The prohibited list isn’t a guardrail, it’s a wall.

Phase 6 — Session Memory + Planning. A SessionMemory class that accumulates case facts across conversation turns. A PlanExecutor that decomposes complex queries into five sequential steps before synthesising a final response. The multi-turn improvement over the stateless Phase 2 baseline was significant — the analyst could now ask follow-up questions without re-explaining the case.

Phase 7 — Adaptive Behaviour. A feedback loop where analyst thumbs-up/down signals accumulate and trigger prompt adaptations after a threshold. If the agent misses three escalations in a row, its sensitivity increases automatically. The deepcopy bug we found here — a shallow copy mutating a shared default config — is exactly the kind of subtle state management issue that only surfaces when you’re testing real behaviour change.

Phase 8 — Deployment. FastAPI REST API with Pydantic validation, CORS middleware, graceful failure handling, custom response headers, and a web interface served directly from the server to avoid browser CORS issues.

Phase 9 — Evaluation. Phase 9 utilised an 18‑test safety harness. Two safety tests initially failed — they exposed a genuine gap where “Transfer $50,000 from the account” wasn’t caught by the transactional‑pattern detector because it used a dollar amount instead of the word “funds”. The issue was fixed live during the evaluation run by expanding the prohibited‑action patterns to include numeric currency expressions. After the update, all tests passed, giving a final score of 18/18.

This 18‑test harness is separate from the broader 48‑test automated suite, which validates full‑system behaviour and also achieved a 100% pass rate.

The Finding That Surprised Us Most

We profiled every component of the pipeline, expecting the LLM API call to dominate latency. It does — but the second-biggest surprise was the embedding model.

On the first query after a server restart, RAG retrieval took 7.4 seconds. On the second query? 25 milliseconds.

The entire cost was model loading. The fix was one function call — eager warm-up at server startup. The first query became as fast as all subsequent ones.

The lesson: profile before you optimise. we would have spent time parallelising tool calls, reducing chunk sizes, and caching embeddings — all of which would have had a marginal impact. The real bottleneck was a single, fixable model load.

On Safety-First Design

The most interesting engineering in this project wasn’t the RAG pipeline or the tool calling. It was the safety layer.

check_prohibited() runs before every LLM call. It’s pure Python — no neural network, no API, no latency. It blocks five categories of request:

PII identifiers — account numbers, IBANs, DOBs

Person names — Title Case pattern detection with a 20-word exclusion list of legitimate regulatory terminology (“Politically Exposed Person” passes; “Peter Walters” doesn’t)

Transactional requests — any instruction to move, transfer, or approve funds

Legal advice — legally compliant, am we liable, is this legal

Prohibited actions — file SAR, submit report, modify records

The name detection deserves a note. The first version only blocked explicit PII keywords. A test query containing “Peter Walters is depositing below $10,000” sailed straight through to the LLM. The fix required building a two-word Title Case detector that distinguishes “John Smith” from “New Zealand” — a small but instructive example of how safety layers need real-world stress testing, not just theoretical coverage.

The Typology Problem

The original tool had 7 AML/CFT typologies. The FIU NZ typology reporting set covers more than 20. The gap matters — an agent that can identify structuring but not cryptocurrency laundering or beneficial ownership concealment is incomplete for real compliance work.

We expanded to 21 typologies covering the full FIU NZ set: real estate laundering, crypto/virtual assets, identity fraud, money mules, professional enablers, loan-back schemes, gambling, invoice fraud, drug proceeds, tax evasion, human trafficking, proliferation financing, cyber crime, and beneficial ownership concealment.

Each entry has indicators, NZ regulatory references, risk levels, and search keywords — enough for the RAG-grounded LLM to produce grounded, defensible assessments rather than generic risk commentary.

The expansion revealed a keyword overlap problem: “ransomware proceeds” matched the drug trafficking typology because both terms contained the word “proceeds”. The fix was to remove generic overlapping keywords and ensure the more specific typology always wins.

Is It Production Ready?

Honestly? No — and we think it’s important to say so clearly.

The agent is a strong, well-engineered prototype. The safety architecture, regulatory grounding, and test coverage are genuinely production-quality. But a regulated NZ bank would need more before going live:

Must-have before deployment:

Authentication — currently, any user on port 8000 can query it

Human oversight workflow — a compliance manager approval queue

Audit trail — not just logging queries, but recording the analyst’s final determination

TLS/HTTPS — plain HTTP is fine for localhost, not for a bank network

AI governance policy — the bank’s formal position on AI in compliance decisions

The path there: roughly 4–6 months of focused engineering and compliance work. The architecture doesn’t need rethinking — it needs hardening.

This gap analysis matters because AI systems in regulated industries often get evaluated on capability alone. The harder question is governance: who is accountable when the AI is wrong? What’s the documented oversight process? How do you demonstrate to a regulator that the system was properly controlled?

Those questions don’t have code answers.

What we’d Do Differently

1. Start with the safety layer. we built safety last in Phase 2 and retrofitted it as we went. It would have been cleaner to define the prohibited action taxonomy upfront and build every phase to respect it from day one.

2. Profile earlier. The 7.4-second cold-start wasn’t discovered until late deployment. Earlier profiling would have caught it in Phase 4.

3. Separate test isolation earlier. Many of our early test failures came from session contamination — a prior HIGH-risk test leaving escalation flags that polluted subsequent tests. Designing for test isolation from Phase 6 onwards would have saved significant debugging time.

4. Test the safety layer with adversarial cases. Our initial safety tests used obvious queries. The real gaps — “Transfer $50,000 from the account”, “Is this legally compliant?”, “Peter Walters is depositing” — only emerged from real-world use. Red-team your safety layer before you think it’s done.

The Stack

LLM: GPT-4o-mini (OpenAI)

Vector store: ChromaDB

Embeddings: sentence-transformers all-MiniLM-L6-v2 (384-dim)

Framework: FastAPI + Pydantic

Source knowledge: AML/CFT Act 2009 + NZ guidelines (~125,000 words)

Test coverage: 48 automated tests, 100% pass rate

Typologies: 21 (full FIU NZ set)

Closing Thought

The most valuable thing this project taught us wasn’t technical. It was the discipline of stating constraints before building.

“This agent never executes transactions” isn’t a feature — it’s a hard boundary that shapes every other design decision. The prohibited tool list, the advisory disclaimers, the human escalation requirements — they all flow from that one constraint.

In AI engineering, especially in regulated domains, the things your system refuses to do are as important as the things it does. Maybe more important.

Built as an AI Engineering Capstone project. All outputs are advisory only and require human review before any compliance action is taken.

Technical details available on request through our contact page.