01
Problem
Business problem
The regulatory auditor application lets enterprise subscribers assign and complete health & safety audits across their sites. But regulations are dense. An auditor running a corporate audit or a chemical-handling safety audit had to figure out on their own which requirements applied and which regulations were behind them. That made audits slow, dependent on individual expertise, and often left the regulatory basis for a requirement unclear.
User problem
An auditor working through an audit for their site needed two answers — which requirements apply at the given site, and which regulation backs each one? Finding the right checklists and the exact citations behind them took real time and a wrong answer meant an audit that wouldn't hold up. Auditors needed answers tied to real regulatory references they could click and verify.
02
User journey
03
RAG architecture
04
PRD excerpt
- User
- Auditor or compliance lead completing regulatory H&S audits inside the regulatory auditor application.
- Problem
- Determining which requirements apply to a given audit, and the regulations behind them, requires slow manual navigation through checklists.
- Goal
- Help auditors quickly find the applicable requirements and their codified regulatory citations, grounded in the subscriber's regulatory content.
- Non-goal
- Replace regulatory interpretation or auditor judgment; the assistant surfaces the applicable requirements and citations, it doesn't adjudicate compliance.
- Success metrics
- Auditors locate the right checklists faster
- Every surfaced requirement carries its correct, clickable citation
- Answers stay grounded in the subscriber's own regulatory content
05
Example answer
Citation-first. Refusal-aware. Human-reviewed.
Auditor question
“What Chemical Storage & Segregation regulations apply to my site?”
Assistant answer
The applicable audit requirements are:
- Chemical containers properly labeled, stored, and closed when not in use.
- Chemicals segregated and stored by compatibility, per OSHA General Industry Standards (29 C.F.R. Part 1910).
These are covered in the “Chemical Storage & Segregation” checklist within the Chemical Safety topic.
Design principle
No citation, no answer.
If retrieval can't anchor a claim, the assistant says so and routes the user to a human reviewer or a clarifying question. Precision over recall — silent confidence is a release blocker.
06
Trust, safety & quality gates
| Gate | What it checks | Pass criteria |
|---|---|---|
| Citations are data, not model output | Citations stored alongside requirements, surfaced deterministically | Model never generates a regulatory reference |
| Grounding | Every answer traces to a real requirement in the content | Answer maps to retrieved requirement(s) |
| Tenant scoping | Retrieval bound to the subscriber's own regulatory content | Zero cross-tenant content |
| Refusal behavior | Declines when content isn't in the subscriber's library | "Not found" rather than fabrication |
07
Evaluation
A failure in any dimension halted release.
Every answer was evaluated against a six-dimension rubric. A failure in any single dimension halted release.
| Dimension | What we checked | Pass | Fail |
|---|---|---|---|
| Retrieval accuracy | Did it pull the right requirement? | ✓Correct requirement surfaced | ✗Wrong/irrelevant requirement returned |
| Summary fidelity | Does the summary reflect the requirement? | ✓Faithful to source meaning | ✗Overstates, omits, or misreads |
| Citation correctness | Is the citation tied to that requirement? | ✓Citation matches | ✗Mismatched or missing citation |
| Grounding | Is every answer traceable to a real requirement? | ✓Links back to source | ✗Unsupported claim / no source |
| Tenant isolation | Did the user see only their own content? | ✓Correctly scoped | ✗Leakage across subscribers |
| Refusal behavior | Does it decline when content isn't in the library? | ✓“Not found in library” | ✗Fabricates an answer anyway |
QUALITY GATES (ALL MUST BE MET FOR RELEASE)
- Summary fidelity verified against source on a review sample
- Zero citation mismatches in the tested set
- No cross-tenant content leakage
- Graceful refusal when a requirement isn't in the subscriber's library
08
Tradeoffs, outcome, next
Tradeoffs
Precision over magic
- Stored citations over AI generated ones: Every citation is saved in the database next to its requirement instead of being written by the model on the fly. That meant more upfront data work, but it made a made-up citation impossible. In a regulated audit, that's non-negotiable.
- Keyword search + SQL over embeddings: Simpler to build, easy to debug, nothing new to maintain. The cost - search is pickier about exact wording. Acceptable here, because regulatory content uses consistent, controlled language.
- Subscriber-specific content over generic content: Outputs displayed only from subscriber's data and never pooled across customers. Less 'collective wisdom', but customer data stays fully separated.
Outcome
Deployed impact
- Auditors spent less time hunting for and locating the right audit checklists, and could more quickly identify which checklists to focus on for a given audit.
- Every surfaced requirement carried its clickable regulatory citation, giving auditors a defensible basis they could verify.
- Deployed inside the regulatory auditor application across enterprise subscribers.
What I'd improve next
Next bets
- Agentic next-audit suggestions — have the system recognize a logged-in auditor's previously completed audits and proactively suggest the checklists they should complete next time they're in the tool, rather than waiting to be asked.