What is the PII and secrets leakage checklist for?

It is a structured audit of engineering, logging, RAG, tool, and incident-response controls to reduce accidental exposure of personal data and credentials in AI applications before and after launch.

How often should I run this checklist?

Run it before shipping LLM features and re-run after major changes such as adding RAG, agents, or a new model provider. Quarterly reviews help catch configuration drift in logs and retention.

What is the minimum bar for production?

At minimum: server-side scanning on every LLM path, block on unsafe secrets, redact or block PII per policy, scan full assembled prompts, and avoid logging raw message bodies in production.

Does passing the checklist guarantee zero leakage?

No. Detection has false negatives and obfuscated data may evade scanners. The checklist reduces common engineering gaps; it does not replace legal review or enterprise DLP where required.

What synthetic data should I use for checklist tests?

Use user@example.com, 555-010-0200, 4111111111111111, sk-test_ prefixes, AKIA0000000000000000, and password=ExampleSyntheticPassword123—never real customer data or live credentials.

PII and Secrets Leakage Checklist for AI Applications

This PII and secrets leakage checklist is an actionable audit you can run before shipping—or monthly while operating—any LLM feature that accepts user text, indexes documents, calls tools, or logs prompts, so sensitive data is less likely to reach model providers, vector stores, observability backends, or other users. Work through each section, mark status, assign an owner, and track remediation; no checklist replaces legal review, but it catches the engineering gaps that cause most accidental exposures.

Use PII & Secrets Detection (POST /api/v1/security/pii-secrets) as your primary text scanner. Validate with PII Checker. Integration reference: docs.

How to use this checklist

Pass — Control implemented and verified in staging/production
Partial — Started but gaps remain (note them)
Fail — Not implemented; treat as blocker for production
N/A — Justify in writing (e.g., no RAG in v1)

Record request_id samples from test scans in your security ticket system for traceability.

1. Data inventory and scope

#	Item	Pass criteria
1.1	List all text paths entering LLM context	Diagram covers user input, history, RAG, tools, system vars
1.2	Classify which features may process PII	Written policy per feature flag / SKU
1.3	Identify regulated data types (PCI, PHI, etc.)	Legal/compliance sign-off documented
1.4	Map third parties receiving prompts	Provider names, data processing terms reviewed
1.5	Define retention for chat logs	TTL configured; deletion tested

2. Input boundary controls

#	Item	Pass criteria
2.1	Server-side PII/secrets scan on every chat message	Cannot reach LLM without scan
2.2	Block on `unsafe` verdict (secrets)	User sees non-echoing error; no provider call
2.3	Redact on `suspicious` verdict (PII) when policy requires	`redacted_text` forwarded; see Redact PII Before LLM
2.4	Scan full assembled prompt, not only latest turn	History + RAG + tools included
2.5	Privacy filter module documented	Owner team named; see LLM Privacy Filter
2.6	Fail-closed behavior defined for scanner outage	Documented in runbook
2.7	Max input length enforced (≤32,000 chars per API limit)	Split or reject oversized payloads

3. Secrets-specific controls

#	Item	Pass criteria
3.1	API keys never in system prompts or few-shot examples	PR review checklist
3.2	`.env` / PEM uploads blocked or scanned in document chat	File type allowlist
3.3	Paste-code UI warns on secret patterns	Manual UX test
3.4	Rotation runbook for exposed keys	Linked from on-call docs; see Prevent API Key Leak
3.5	Private keys trigger block + incident	Test with synthetic PEM
3.6	Credential pairs detected in forms	See Detect Credentials in User Input

4. RAG and document ingestion

#	Item	Pass criteria
4.1	Chunks scanned before embedding	Quarantine or redact on findings
4.2	PDF/HTML extraction re-scanned post-parse	Hidden text cases tested
4.3	Internal wikis/repos secret-scanned before index	CI or ingestion job
4.4	Retrieval results scanned before prompt assembly	Or pre-embedded clean store only
4.5	Vector store access restricted	IAM / tenant isolation verified

5. Tool and agent paths

#	Item	Pass criteria
5.1	Tool outputs minimized (least fields)	Schema review
5.2	Tool outputs scanned before model re-entry	Automated test
5.3	Agents cannot exfiltrate env vars via prompts	Sandbox has no secret env
5.4	Browser/fetch tools do not pull raw `.env` URLs	Blocklist tested
5.5	High-impact tools require human approval	Documented exceptions

6. Logging and observability

#	Item	Pass criteria
6.1	Raw prompts not logged at INFO in production	Log config audit
6.2	Privacy scan logs `request_id` + categories only	No raw PII in SIEM
6.3	Error traces strip Authorization headers	Synthetic curl test
6.4	Debug modes time-limited and gated	Cannot enable via query param alone
6.5	Third-party analytics exclude message bodies	Tag manager / SDK review

7. Output and display

#	Item	Pass criteria
7.1	Model output scanned before show (when PII echo risk)	Optional but documented
7.2	Multi-tenant session isolation tested	User A cannot see User B context
7.3	Support admin views redacted by default	Role-based access
7.4	Export/download flows respect redaction	CSV export test

8. Redaction strategy

#	Item	Pass criteria
8.1	Redaction vs masking vs tokenization decided	See comparison article
8.2	Placeholder format consistent (`[EMAIL]`, etc.)	Matches API docs
8.3	Stored chat history policy defined (raw vs redacted)	DB column audit
8.4	Credit card handling aligned with PCI scope	PAN not in general chat; see Detect Credit Card Numbers

9. Enterprise alignment

#	Item	Pass criteria
9.1	App-level scan complements corporate DLP (if any)	Roles clear; see PII Detection vs DLP
9.2	Vendor DPAs cover LLM subprocessors	Legal file current
9.3	Employee AI acceptable-use policy published	Training completion tracked

10. Testing and CI

#	Item	Pass criteria
10.1	Synthetic fixtures in CI (`user@example.com`, `sk-test_...`)	Pipeline green
10.2	Node or Python integration tests	See Node.js / Python guides
10.3	Regression test for each leaked-secret incident	Ticket → test case
10.4	Staging uses distinct API keys from production	Key inventory
10.5	Manual spot-check via PII Checker before major releases	Release template

Synthetic test vectors (never use real data):

user@example.com
555-010-0200
4111111111111111
sk-test_abcdefghijklmnopqrstuvwxyz123456
AKIA0000000000000000
password=ExampleSyntheticPassword123!

Expected: appropriate verdict and findings per API docs.

11. Incident response

#	Item	Pass criteria
11.1	Playbook for secret in prompt	Rotation steps within 24h target
11.2	Provider log inquiry process documented	Contacts listed
11.3	Customer notification criteria defined	Legal involvement trigger
11.4	Post-incident test added to CI	Closed-loop verified

12. Limitations acknowledgment

#	Item	Pass criteria
12.1	Team acknowledges regex limits (obfuscation, images)	Documented in risk register
12.2	No "100% secure" marketing claims	Public copy review
12.3	Periodic re-audit scheduled (e.g., quarterly)	Calendar invite exists

Detection reduces risk; it does not guarantee zero leakage.

Quick scorecard

Count Pass items in sections 2–7 (core engineering):

Score	Interpretation
90%+ Pass	Strong baseline for production
70–89%	Ship with documented gaps and timeline
Below 70%	High leakage risk—prioritize sections 2, 3, 4

Next steps after the audit

Implement missing server-side scan — Detect PII in Text API
Build privacy filter module — LLM Privacy Filter
Train team on secrets in prompts — Prevent API Key Leak
Add PII & Secrets Detection to staging and production with monitoring

Re-run this checklist after major feature launches (RAG, agents, new model provider).