Sponsored

Stop vibe-debugging.

Every exception, every backtrace, grouped so you see patterns, not noise.

Start free trial
  1. ai
  2. /foundations
  3. /evaluating-output

Evaluating Model Output

Models optimize for plausible text, not correct software. Evaluation means matching review depth to risk — not reviewing every line the same way.

Last reviewed: June 2026

This framework complements Verifying AI Output — use that checklist at merge time.

Prerequisites

Read How LLMs Work first. This page applies that mental model to review decisions.

Trust vs Verify Spectrum

ZoneExamplesReview depth
Low riskComment edits, typo fixes in docsSkim diff
MediumNew utility function, test additionsRead diff + run tests
HighAuth, payments, SQL, crypto, infraLine-by-line + security review
Never trust aloneDependencies, env vars, security middlewareVerify against docs + grep registry

Rule: Higher blast radius → less trust in fluency, more verification in tooling.

Evaluation Framework

For each AI-generated change, answer four questions:

  1. Can I verify with automation? (build, lint, test, typecheck)
  2. Does this touch user data or money?
  3. Did the model introduce new external dependencies or env vars?
  4. Would I understand this code in a interview?

If any answer is "yes" for questions 2–4, escalate review beyond skim.

flowchart TD
    change[AI-generated change] --> auto{Automated checks pass?}
    auto -->|No| reject[Reject — paste errors to model]
    auto -->|Yes| risk{High risk path?}
    risk -->|Yes| senior[Senior + security review]
    risk -->|No| human[Human diff review]
    human --> merge[Merge]
    senior --> merge

Signals of Reliable Output

SignalWhy it helps
Matches existing patterns in repoLower novelty → lower surprise
Small, focused diffEasier to audit
Tests included and passingBehavior checked
No new importsFewer phantom dependency risks
Cites file paths you @-mentionedGrounded in provided context

Signals to Distrust

SignalAction
New package you did not requestVerify on npm; check download stats
process.env.* you never definedBlock merge until env documented
Confident explanation, failing testsTrust compiler over narrative
Large refactor beyond askRevert; narrow scope
Dead documentation linksTreat cited APIs as suspect

See Common AI Coding Mistakes for full catalog.

Change-Type Rubric

ChangeMinimum verification
CSS tweakVisual check in browser
React componentRender + a11y spot check
API routecurl test + auth check
Database migrationDry-run on staging; backup plan
Config / CIDiff against known-good template

Worked Example

Task: "Add a formatCurrency helper."

StepEvaluation
Model adds src/lib/formatCurrency.tsRead implementation — does it handle null?
Model adds testsRun npm test -- formatCurrency
Model imports intl polyfill you do not useReject — remove dependency
Tests pass, no new depsApprove after quick read

Task: "Fix login redirect bug."

StepEvaluation
Model edits auth middleware + 8 other filesStop — revert extras
Re-prompt with file boundaryRe-evaluate smaller diff
Touches session cookie flagsSecurity reviewer required

For Teams

Encode rubric in Team AI Policy:

  • Define high-risk paths (auth/, payments/, infra/)
  • Require ai-assisted label for traceability
  • Track post-merge bugs on AI PRs — tune rules, do not ban tools outright

Reference: NIST AI RMF for organizational risk framing.