Testing AI-Generated Code

AI-generated tests often look complete while asserting nothing meaningful. Testing AI code means verifying behavior the model cannot hand-wave away.

Prerequisites

Read Evaluating Model Output and Verifying AI Output.

What Models Fake in Tests

Minimum Test Bar by Change Type

Workflow

1. Run existing suite before merge — npm test or project equivalent
2. Read new tests line by line — every expect must mean something
3. Break the implementation deliberately — test should fail
4. Check coverage only where it matters — 100% coverage with weak asserts is worthless

npm test
npm run lint
npm run build

Reference: Jest docs, Vitest docs, Testing Library.

Worked Example: Hollow Test

it("formats currency", () => {
  const format = jest.fn().mockReturnValue("$1.00");
  expect(format()).toBe("$1.00");
});

import { formatCurrency } from "./formatCurrency";

it("formats USD with two decimals", () => {
  expect(formatCurrency(1099, "USD")).toBe("$10.99");
});

it("returns em dash for null", () => {
  expect(formatCurrency(null, "USD")).toBe("—");
});

Property-Based and Fuzz Checks

For parsers, validators, and serializers, add property tests when feasible:

import fc from "fast-check";

it("round-trips JSON-safe strings", () => {
  fc.assert(
    fc.property(fc.jsonValue(), (value) => {
      expect(parse(serialize(value))).toEqual(value);
    })
  );
});

Libraries: fast-check, Hypothesis (Python).

Integration and E2E

AI often skips integration tests. Require them when:

• Auth middleware changes
• Database queries change
• External webhooks involved

Run E2E sparingly but on critical paths — Playwright, Cypress.

CI Integration for Agent Workflows

When agents run in CI (Agentic Workflows):

Prompts That Produce Better Tests

Write tests for formatCurrency in src/lib/formatCurrency.ts.
Requirements:
- Test real imported function, no mocks of formatCurrency
- Cover null, 0, negative, and large values
- Use project's existing test runner (Vitest)
- Run tests and paste output

For Teams

• Add "tests assert behavior, not mocks" to Project Rules
• Require regression test for every AI-assisted bugfix in Team AI Policy
• Track escaped defects on ai-assisted PRs — tune prompts, not blame individuals

Related

• Verifying AI Output
• Evaluating Model Output
• Common AI Coding Mistakes
• Pre-Merge Verification cheat sheet
• Agentic Workflows