How to Add Output Safety Checks in Next.js

Add AI output safety checks in Next.js by calling IdenticAPI from Route Handlers or Server Actions after the LLM returns and before any assistant text is sent to the client — then route on safe, suspicious, or unsafe verdicts and render only encoded or sanitized content in React components.

Never expose API keys in Client Components or call moderation from the browser in production. The App Router makes server-side gating straightforward.

Architecture overview

Client Component (chat UI)
    │
    ▼ POST /api/chat
Route Handler (server)
    ├── call LLM provider
    ├── POST IdenticAPI output-safety
    ├── route verdict
    └── return safe message JSON

Related concepts: How to Moderate LLM Output, AI Content Moderation API Guide.

Environment variables

Add to .env.local (never commit secrets):

IDENTICAPI_API_KEY=idapi_live_your_key_here
LLM_API_KEY=your_llm_key_here

Read IDENTICAPI_API_KEY only in server modules (app/api/**, server/ utilities, Server Actions marked "use server").

Output safety helper (server-only)

Create lib/output-safety.ts:

// lib/output-safety.ts — server-only
const IDENTICAPI_BASE = "https://www.identicapi.com";

export type OutputSafetyResult = {
  verdict: "safe" | "suspicious" | "unsafe";
  risk: string;
  findings: Array<{ category: string; reason: string }>;
  reasons: string[];
  request_id?: string;
};

export async function checkOutputSafety(text: string): Promise<OutputSafetyResult> {
  const apiKey = process.env.IDENTICAPI_API_KEY;
  if (!apiKey) {
    throw new Error("IDENTICAPI_API_KEY is not configured");
  }

  const res = await fetch(`${IDENTICAPI_BASE}/api/v1/security/output-safety`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ text }),
    cache: "no-store"
  });

  if (!res.ok) {
    throw new Error(`Output safety check failed: ${res.status}`);
  }

  return res.json() as Promise<OutputSafetyResult>;
}

export function actionForVerdict(verdict: OutputSafetyResult["verdict"]) {
  switch (verdict) {
    case "safe":
      return "deliver" as const;
    case "suspicious":
      return "review" as const;
    case "unsafe":
      return "block" as const;
    default:
      return "review" as const;
  }
}

export const FALLBACK_MESSAGE =
  "I can't share that response. Please rephrase your question or contact support.";

See AI Output Safety documentation for full response schema.

Route Handler with moderation gate

app/api/chat/route.ts:

import { NextResponse } from "next/server";
import { checkOutputSafety, actionForVerdict, FALLBACK_MESSAGE } from "@/lib/output-safety";

export const runtime = "nodejs";

type ChatRequest = { message: string };

async function callLlm(userMessage: string): Promise<string> {
  // Replace with your provider (OpenAI, Anthropic, etc.)
  return `Synthetic assistant reply for: ${userMessage}`;
}

export async function POST(req: Request) {
  let body: ChatRequest;
  try {
    body = await req.json();
  } catch {
    return NextResponse.json({ error: "Invalid JSON" }, { status: 400 });
  }

  const userMessage = body.message?.trim();
  if (!userMessage) {
    return NextResponse.json({ error: "message is required" }, { status: 400 });
  }

  let assistantText: string;
  try {
    assistantText = await callLlm(userMessage);
  } catch {
    return NextResponse.json({ error: "LLM unavailable" }, { status: 502 });
  }

  let moderation;
  try {
    moderation = await checkOutputSafety(assistantText);
  } catch {
    // Fail-closed: do not deliver unmoderated LLM output
    return NextResponse.json({
      role: "assistant",
      content: FALLBACK_MESSAGE,
      moderated: true,
      verdict: "unknown"
    });
  }

  const action = actionForVerdict(moderation.verdict);

  if (action === "block") {
    return NextResponse.json({
      role: "assistant",
      content: FALLBACK_MESSAGE,
      moderated: true,
      verdict: moderation.verdict,
      request_id: moderation.request_id
    });
  }

  if (action === "review") {
    // Queue for human review in your system; customer sees cautious fallback
    await queueForReview({ userMessage, assistantText, moderation });
    return NextResponse.json({
      role: "assistant",
      content: FALLBACK_MESSAGE,
      moderated: true,
      verdict: moderation.verdict,
      request_id: moderation.request_id
    });
  }

  return NextResponse.json({
    role: "assistant",
    content: assistantText,
    moderated: true,
    verdict: moderation.verdict,
    request_id: moderation.request_id
  });
}

async function queueForReview(payload: unknown) {
  // Persist to your review store — implementation-specific
  void payload;
}

This implements block vs review routing server-side.

Client Component (safe rendering)

Render assistant content as text — not raw HTML:

"use client";

import { useState } from "react";

type ChatMessage = { role: "user" | "assistant"; content: string };

export function ChatPanel() {
  const [messages, setMessages] = useState<ChatMessage[]>([]);
  const [input, setInput] = useState("");
  const [loading, setLoading] = useState(false);

  async function send() {
    const text = input.trim();
    if (!text || loading) return;

    setInput("");
    setMessages((m) => [...m, { role: "user", content: text }]);
    setLoading(true);

    try {
      const res = await fetch("/api/chat", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ message: text })
      });
      const data = await res.json();
      setMessages((m) => [...m, { role: "assistant", content: data.content }]);
    } finally {
      setLoading(false);
    }
  }

  return (
    <div>
      {messages.map((msg, i) => (
        <p key={i}>
          <strong>{msg.role}:</strong> {msg.content}
        </p>
      ))}
      <input value={input} onChange={(e) => setInput(e.target.value)} />
      <button onClick={send} disabled={loading}>
        Send
      </button>
    </div>
  );
}

If you need Markdown or HTML, sanitize server-side first — safe AI-generated HTML and XSS prevention.

Streaming considerations

Next.js streaming responses (SSE or ReadableStream) should not forward raw LLM tokens to the client before moderation completes. Patterns:

1. Buffer completion in the Route Handler; moderate; then stream approved text
2. Stream plain "typing" indicators until moderation passes

See Moderating AI Chatbot Responses.

Server Actions alternative

For form-based flows, call checkOutputSafety inside a Server Action after LLM inference — same helper, same verdict routing. Keep "use server" files free of client imports.

CSP headers

In next.config.ts or middleware, set Content-Security-Policy on chat routes:

const csp = [
  "default-src 'self'",
  "script-src 'self'",
  "object-src 'none'",
  "base-uri 'self'",
  "frame-ancestors 'none'"
].join("; ");

Apply via headers() config or middleware NextResponse.next({ headers }).

Prototyping

Use the AI Output Safety Checker to validate sample outputs before wiring the Route Handler. Product overview: AI Output Safety.

Unified Guard (optional)

If you also screen user input and want one orchestrated call, evaluate Unified Guard — compare scope in AI Guardrails vs Content Moderation.

Testing

Add integration tests (Vitest + fetch mock) asserting:

• unsafe moderation → client receives FALLBACK_MESSAGE only
• Moderation API failure → fail-closed fallback, not raw LLM text
• safe → assistant content delivered

Include synthetic XSS strings from prevent XSS guide.

Checklist

Work through AI Output Safety Checklist before production. Cross-reference Python integration if your LLM runs in a separate service — moderate at the boundary before data enters the Next.js UI layer.

Limitations

Server-side moderation in Next.js reduces client exposure but does not alone guarantee safety:

• Verdicts are risk signals — maintain human review for suspicious
• Sanitization and CSP still required for rich text
• Edge runtime fetch timeouts may differ — test your deployment target

Keep API keys server-only, moderate every completion, and render defensively. Documentation covers authentication, rate limits, and response fields.