LLM Observability and Evals

Shipping an LLM feature without observability means debugging production failures blind. Standard app monitoring does not capture what makes AI features fail: prompt regressions, token blowouts, and degraded retrieval quality.

What to Capture Per Request

Every LLM call should log:

Do not log full prompt/response content unless you have a data retention policy and user consent. Log metadata and hashes for PII-sensitive apps.

Minimal Request Logger (Vercel AI SDK)

The onFinish callback in streamText and generateText gives you usage after the response completes:

// lib/llm-logger.ts
export interface LLMTrace {
  traceId: string;
  model: string;
  promptVersion?: string;
  inputTokens: number;
  outputTokens: number;
  cachedTokens?: number;
  latencyMs: number;
  finishReason: string;
  userId?: string;
  sessionId?: string;
  toolCallCount?: number;
}

export function logTrace(trace: LLMTrace) {
  // Replace with Axiom, Datadog, or your own DB
  console.log(JSON.stringify({ event: "llm_trace", ...trace }));
  // Alert on anomalies
  if (trace.inputTokens > 50_000) {
    console.warn("ALERT: large context window usage", { traceId: trace.traceId });
  }
  if (trace.finishReason === "length") {
    console.warn("ALERT: response hit maxTokens — consider raising cap", trace);
  }
}

// app/api/chat/route.ts
import { anthropic } from "@ai-sdk/anthropic";
import { streamText } from "ai";
import { logTrace } from "@/lib/llm-logger";
import { randomUUID } from "crypto";

export async function POST(req: Request) {
  const { messages, sessionId } = await req.json();
  const traceId = randomUUID();
  const start = Date.now();

  const result = streamText({
    model: anthropic("claude-sonnet-4-20250514"),
    messages,
    maxTokens: 1024,
    onFinish: ({ usage, finishReason, toolCalls }) => {
      logTrace({
        traceId,
        model: "claude-sonnet-4-20250514",
        inputTokens: usage.promptTokens,
        outputTokens: usage.completionTokens,
        latencyMs: Date.now() - start,
        finishReason,
        sessionId,
        toolCallCount: toolCalls?.length,
      });
    },
  });

  return result.toDataStreamResponse();
}

Langfuse (Open-Source LLM Observability)

Langfuse provides traces, spans, prompt versioning, and a scoring UI. It has a self-hosted option and a cloud tier.

npm install langfuse

// lib/langfuse.ts
import { Langfuse } from "langfuse";

export const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY!,
  secretKey: process.env.LANGFUSE_SECRET_KEY!,
  baseUrl: process.env.LANGFUSE_BASEURL, // omit for cloud
});

// app/api/chat/route.ts — wrap requests with Langfuse traces
import { langfuse } from "@/lib/langfuse";

export async function POST(req: Request) {
  const { messages, sessionId } = await req.json();

  const trace = langfuse.trace({
    name: "chat",
    sessionId,
    input: messages.at(-1)?.content,
  });

  const generation = trace.generation({
    name: "claude-chat",
    model: "claude-sonnet-4-20250514",
    input: messages,
  });

  const result = streamText({
    model: anthropic("claude-sonnet-4-20250514"),
    messages,
    maxTokens: 1024,
    onFinish: ({ text, usage, finishReason }) => {
      generation.end({
        output: text,
        usage: {
          input: usage.promptTokens,
          output: usage.completionTokens,
        },
      });
      trace.update({ output: text });
    },
  });

  return result.toDataStreamResponse();
}

Langfuse docs: langfuse.com/docs. Self-hosted: Docker Compose setup.

Braintrust (Evals-First Observability)

Braintrust focuses on eval pipelines — define expected outputs, score completions, track regression over time.

npm install braintrust autoevals

// evals/chat-quality.eval.ts
import { Eval } from "braintrust";
import { Factuality, Levenshtein } from "autoevals";
import { anthropic } from "@ai-sdk/anthropic";
import { generateText } from "ai";

await Eval("chat-quality", {
  data: () => [
    {
      input: "How do I reset my password?",
      expected: "Go to Settings → Security → Reset Password.",
    },
    {
      input: "What payment methods do you accept?",
      expected: "We accept Visa, Mastercard, and PayPal.",
    },
  ],

  task: async (input) => {
    const { text } = await generateText({
      model: anthropic("claude-sonnet-4-20250514"),
      system: "You are a support agent for Acme Corp. Answer concisely.",
      prompt: input,
    });
    return text;
  },

  scores: [Factuality, Levenshtein],
});

Run evals in CI to catch prompt regressions before deploy:

npx braintrust eval evals/chat-quality.eval.ts

Braintrust docs: braintrustdata.com/docs.

OpenTelemetry for AI

The OpenTelemetry Semantic Conventions for LLMs provide standard span attributes for gen_ai.* operations. Use this when you already have an OTel pipeline and want LLM calls in the same trace:

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("ai-service");

export async function callLLM(messages: CoreMessage[]) {
  return tracer.startActiveSpan("gen_ai.chat", async (span) => {
    span.setAttribute("gen_ai.system", "anthropic");
    span.setAttribute("gen_ai.request.model", "claude-sonnet-4-20250514");
    span.setAttribute("gen_ai.request.max_tokens", 1024);

    try {
      const { text, usage } = await generateText({
        model: anthropic("claude-sonnet-4-20250514"),
        messages,
        maxTokens: 1024,
      });

      span.setAttribute("gen_ai.usage.input_tokens", usage.promptTokens);
      span.setAttribute("gen_ai.usage.output_tokens", usage.completionTokens);
      span.setStatus({ code: SpanStatusCode.OK });
      return text;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      span.end();
    }
  });
}

Prompt Versioning

Track which prompt was active when a failure occurred. Simple approach: store prompt templates in your codebase with version identifiers.

// lib/prompts.ts
export const PROMPTS = {
  supportAgent: {
    version: "v3",
    system:
      "You are a support agent for Acme Corp. Answer only questions about orders and billing. " +
      "For technical issues, direct users to docs.acme.com.",
  },
  codeReviewer: {
    version: "v2",
    system:
      "You are a senior engineer reviewing code for correctness, security, and performance. " +
      "Be specific about line numbers and concrete improvements.",
  },
} as const;

Include promptVersion in every trace log. When you change a prompt, bump the version and monitor quality metrics for 24–48 hours.

Eval Methodology for RAG

Beyond production monitoring, build an offline eval set before shipping a RAG feature:

1. Build a spot-check set: 20–30 questions with known answers in your docs
2. Run retrieval with topK = 5: does any returned chunk contain the answer?
3. Target ≥80% recall@5 before launch
4. Run monthly as docs change

See RAG for Codebases — Evaluating Retrieval Quality for the full methodology.

Monitoring Checklist

Before launching an AI feature in production:

• [ ] Token usage logged per request (input + output + cached)
• [ ] Latency logged (first token + total response time)
• [ ] finishReason monitored — alert on length spikes
• [ ] Per-user rate limits enforced
• [ ] Prompt version tracked in logs
• [ ] Eval baseline established (even 10 spot-check questions is better than none)
• [ ] Alerts configured for input token spikes (retry loop) and output spikes (runaway generation)
• [ ] Data retention policy documented for logs containing user content

Related

• Cost, Latency, and Tokens
• LLM APIs and Tool Calling
• RAG for Codebases
• Security and Prompt Injection
• Anthropic API for Web Developers