Overview & context

Retrieval-Augmented Generation (RAG) combines large language models (LLMs) with live or indexed data retrieval, while agentic systems extend this with autonomous decision loops. This architecture increases capability but also multiplies governance risks — from citation errors to unintended system actions. Zen AI Governance enforces specific controls for transparency, provenance, sandboxing, and oversight escalation.

Architecture & data flow

User Query → RAG Pre-processor → Retriever (Vector DB / Search API) → LLM Generation  
     ↓ provenance tags + metadata  
     ↓ moderation filter + context logging  
     ↓ oversight check + audit record  
     → Response UI or Agentic Action (with escalation triggers)

• Every retrieval must record source URL, document ID, timestamp, and confidence score.
• Context windows and memory stores are limited to prevent unreviewed long-term accumulation.
• Output moderation applied both pre- and post-generation (toxicity, privacy leakage, hallucination).

Key risks & failure modes

• Provenance loss: Missing or unverifiable citation trails in RAG retrievals.
• Hallucination: Generation of false or fabricated facts without grounding evidence.
• Autonomy drift: Agent acts outside authorised goals or context window.
• Prompt injection: Malicious user modifies agent goals or bypasses safeguards.
• Escalation delay: Failure to detect unsafe or uncertain conditions in time.

Provenance & citation controls

• Embed source metadata (document IDs, URLs, authorship) with every RAG output.
• Maintain a provenance ledger linking retrieved chunks to stored embeddings.
• Use confidence thresholds (e.g., cosine similarity ≥ 0.80) to validate retrieved data.
• Display citations visibly in user output — clickable links or footnotes.
• Version control for retrieval indices and embeddings to preserve traceability.

Sandboxing & containment

• Agentic systems operate in secure sandboxes with restricted APIs and rate limits.
• Prohibit direct file system or network write access unless authorised via governance gate.
• Each agent action logged: timestamp, command, target, result, outcome code.
• Implement “Safe Action Lists” — pre-approved function calls only.
• Kill-switch endpoints allow human operators to instantly pause or terminate execution.

Escalation & oversight triggers

• Define escalation tiers:
  • Tier 1 — Auto-flag: uncertainty > 25 % or missing source → request review.
  • Tier 2 — Critical: hallucination, offensive output, or privacy breach → suspend service + CAPA.
  • Tier 3 — Regulatory: repeated or systemic failure → notify AI Governance Board within 24 h.
• Escalations logged with Evidence IDs and CAPA references in AIMS.
• Oversight Officers authorised to rollback or isolate RAG models pending review.

Audit templates & evidence

Audit ID: AIMS-RAG-AUD-2025-07
Model: SupportKnowledgeBot v2.3
Run Date: 2025-11-10
Sample Size: 500 prompts
Hallucination Rate: 1.4 %
Citation Coverage: 98 %
Escalations: 2 (Tier 1)
Corrective Action: Retraining index embeddings
Status: Closed ✅
  

Alignment with NIST / ISO / EU AI Act

• NIST AI RMF: Govern – policy enforcement, Map – context & risk, Measure – trust metrics, Manage – CAPA.
• ISO/IEC 42001: §8.2 (Information), §8.3 (Operation Control), §9 (Performance Evaluation).
• EU AI Act: Articles 14 (Human Oversight), 15 (Accuracy/Robustness), 52 (Transparency Obligations).

Common pitfalls & mitigation

• No source linkage: Always store provenance metadata with responses.
• Uncontrolled agents: Use sandbox APIs and explicit whitelists.
• No escalation path: Pre-define tiered triggers and authorities.
• Drift of retrieval index: Periodically re-embed and re-verify data sources.

Implementation checklist

• RAG provenance ledger implemented and auditable.
• Sandbox & kill-switch controls verified.
• Escalation tiers configured and tested quarterly.
• Audit template applied and evidence stored in AIMS.
• All results reported to AI Governance Board.

© Zen AI Governance UK Ltd • Regulatory Knowledge • v1 12 Nov 2025 • This page is general guidance, not legal advice.