False Positives vs False Negatives in AI Security

In AI security, false positives and false negatives describe how often your guardrails misclassify benign content as risky, or miss real attacks. Confusing these terms — or swapping them for imprecise labels like "accuracy" — leads to wrong policy decisions, broken user experience, and silent security gaps.

This guide defines the vocabulary correctly, connects it to precision and recall, and explains how product teams should trade off errors when routing IdenticAPI safe/suspicious/unsafe (or allow/review/block) verdicts.

For operational tuning, see AI Guardrails False Positives. For measurement methodology, see Measure AI Guardrail Quality.

Definitions (use these consistently)

Consider a binary decision: "should this content be treated as a security risk?" Your ground truth comes from labeled test data or reviewer adjudication.

False positive rate (FPR) = FP / (FP + TN) — how often benign traffic is wrongly blocked.

False negative rate (FNR) = FN / (FN + TP) — how often attacks slip through. This is also 1 − recall when "positive" means "attack detected."

Do not say "false positive" when you mean "the model was wrong" without specifying direction. A missed injection is a false negative, not a false positive.

Precision and recall (not "accuracy")

Accuracy = (TP + TN) / (TP + TN + FP + FN) is misleading for security workloads where attacks are rare and benign traffic dominates. A detector that always returns safe can show high accuracy while missing every attack.

Use these instead:

Precision matters when blocking — low precision means frustrated users and review-queue noise.

Recall matters when allowing — low recall means attacks reach the model, tools, or users.

There is no free lunch: tightening thresholds usually increases recall (fewer missed attacks) at the cost of precision (more false blocks), and vice versa.

Three-verdict systems change the math

IdenticAPI detectors return three outcomes, not binary pass/fail:

In evaluation, decide how to label outcomes:

• Conservative: count suspicious on attacks as TP; count suspicious on benign as FP
• Lenient: count suspicious on benign as acceptable (review is not block)

Pick one convention per report and stick to it. Mixed conventions make precision/recall incomparable across teams.

The review state exists precisely to absorb borderline cases — converting them into hard blocks inflates false positives; converting them into allows inflates false negatives.

Product impact: false positives vs false negatives

Cost of false positives

• Users abandon chat after repeated refusals
• Support volume increases ("why was I blocked?")
• Teams disable guardrails → false negatives surge
• Review queues back up; reviewers approve without reading

Common FP triggers: security training text, quoted attack strings, technical HTML discussion, credential-shaped test keys in developer prompts. See keyword filter limitations.

Cost of false negatives

• Prompt injection reaches tools or data (OWASP LLM01)
• Secrets enter provider logs (API key leaks)
• Unsafe HTML executes in browsers (XSS)
• Agent executes destructive actions (excessive agency)

False negatives are often silent until an incident — you need labeled adversarial tests and production sampling to detect them.

Risk-tiered defaults

Document your risk acceptance. Fail-closed on scanner outage trades availability for fewer FNs during downtime.

Threshold trade-offs in practice

Guardrail APIs return structured signals — not only a single score:

{
  "request_id": "req_abc123",
  "verdict": "suspicious",
  "risk": "medium",
  "findings": [
    { "category": "instruction_override", "reason": "Instruction-like phrasing detected" }
  ],
  "usage_units": 1
}

Your policy layer maps verdict, risk, and findings[].category to allow, review, or block. Threshold tuning happens in policy — not by asking the API to return binary yes/no.

Effective tuning steps:

1. Measure per category — injection FP rate differs from PII FP rate
2. Use review as a buffer — route suspicious + medium risk to review, not block
3. Surface-specific overrides — support bot vs docs search need different maps
4. Version policy in git — audit changes when block rate shifts

Anti-pattern: mapping both suspicious and unsafe to block "to be safe." That maximizes recall on paper while maximizing user-visible false positives in practice.

Evaluation datasets that support precision/recall

You cannot compute precision or recall without labeled data. Build three corpora:

Benign corpus

Realistic user messages (anonymized or synthetic). Label: allow.

Include edge cases your product actually sees — typos, emails in tickets, "ignore the typo" phrasing.

Adversarial corpus

Attacks relevant to your architecture. Label: block or review.

Include direct injection, indirect RAG poison, secret paste, unsafe output markup, and agent abuse scenarios from prompt injection testing.

Borderline corpus

Security education, quoted attacks, mild profanity in adult products. Label: review or allow per policy — not "wrong" if reviewers disagree; document adjudication rules.

Maintain corpora in version control. Re-run when detector_version changes or you switch models.

What to report

Do not publish a single "99% accurate" marketing number without defining the corpus, verdict mapping, and category breakdown. See evaluate AI guardrails.

Review states reduce both error types

A well-designed review workflow:

• Catches false negatives that would have been suspicious if you only binary-blocked unsafe
• Rescues false positives — reviewers approve benign suspicious cases
• Produces labeled feedback for the next policy iteration

Requirements:

• Users never see raw flagged content while pending review
• Reviewers log true positive vs false positive overrides
• SLAs prevent queue stagnation

See Allow, Review, Block Security Model and Block vs Review for AI Output.

Common terminology mistakes

Summary

• False positive: benign flagged — user friction, guardrail fatigue
• False negative: attack missed — incident risk, often invisible without tests
• Precision guides block decisions; recall guides allow decisions
• Review absorbs ambiguity — use it instead of over-blocking suspicious
• Measure with labeled corpora per category; avoid aggregate "accuracy" for rare attacks

IdenticAPI returns verdict, risk, findings, and request_id so your policy layer can tune trade-offs without retraining detectors. The vocabulary is standard; the product choices are yours.