Evidence-Based Verification

Understanding Agent Flow's verification philosophy: what constitutes proof, why claims aren't enough, and how the system enforces evidence requirements.

The Verification Problem

In traditional software development, developers verify their own work before claiming completion. This works because humans have: - Memory of what they actually did - Ability to re-run commands and observe results - Accountability for false claims

LLM agents lack these properties. They generate responses based on patterns, not memories of actions taken. An agent can claim "all tests pass" without ever running tests, simply because that's the expected response pattern.

What Constitutes Evidence

Evidence in Agent Flow means actual command output that proves the claimed result.

Valid Evidence

Type Check: PASS
$ npx tsc --noEmit
(no output - clean compilation)

Tests: PASS
$ npm test
> project@1.0.0 test
> jest

PASS src/__tests__/auth.test.ts
PASS src/__tests__/user.test.ts
Test Suites: 2 passed, 2 total
Tests:       15 passed, 15 total

Lint: PASS
$ npm run lint
> project@1.0.0 lint
> eslint .

(no output - no lint errors)

Invalid "Evidence"

Type Check: PASS
(TypeScript compilation successful)

Tests: PASS
(All tests pass)

Lint: PASS
(No lint errors found)

The second example provides no evidence - just claims. The agent may never have run these commands.

Evidence Requirements by Agent

Loid (Executor)

Must provide: - Actual verification command outputs - Pass/fail counts with specifics - Zero errors confirmed in output

Required format:

Verification Complete

Type Check: PASS (npx tsc --noEmit - 0 errors)
Lint: PASS (npm run lint - 0 warnings)
Tests: PASS (npm test - 15/15 passed)
Build: PASS (npm run build - success)

Alphonse (Verifier)

In the parallel /orchestrate flow Lawliet runs type check and lint, so Alphonse reports those two as COVERED (Lawliet); run on its own it runs all four gates. The block below is illustrative; the format is defined in agents/Alphonse.md.

Must provide: - Exact command outputs (not summaries) - Full error messages for any failures - Clear status for each verification gate

Required format:

## Verification Results

### Tests
- Status: PASS
- Command: `npm test`
- Output:
  ```
  PASS src/__tests__/auth.test.ts (2.345s)
  PASS src/__tests__/user.test.ts (1.234s)

  Test Suites: 2 passed, 2 total
  Tests:       15 passed, 15 total
  Time:        3.579s
  ```

### Type Check
- Status: COVERED (Lawliet)

### Lint
- Status: COVERED (Lawliet)

### Build
- Status: PASS
- Command: `npm run build`
- Output:
  ```
  > project@1.0.0 build
  > tsc && vite build

  vite v5.0.0 building for production...
  dist/index.js   45.2 kB
  ```

### Overall: VERIFIED

Riko (Explorer)

Must provide: - File paths with line numbers for every claim - Actual code snippets (not paraphrased) - Clear source attribution

Illustrative format:

### Key Files
- `src/auth/login.ts:45-67` - Login handler with JWT generation
- `src/middleware/auth.ts:12-30` - Authentication middleware

- `src/auth/login.ts:52-58` - `jwt.sign({ userId, role }, secret, { expiresIn: '24h' })`

The Evidence Hierarchy

Agent Flow evaluates evidence in order of trustworthiness:

Level 1: Actual command output
         └── Most trustworthy - verifiable

Level 2: File contents shown
         └── Trustworthy - can be verified

Level 3: Claims about files
         └── Less trustworthy - no verification

Level 4: General assertions
         └── Least trustworthy - accept only for exploration

When Each Level is Acceptable

Task Type	Minimum Evidence Level
Verification	Level 1 (command output)
Implementation	Level 1 (command output)
Review	Level 2 (file contents)
Exploration	Level 3 (claims with paths)
Questions	Level 4 (general assertions)

Verification Gate Structure

Each verification gate requires specific evidence:

Pre-Complete Gates

Gate	Required Evidence
Tests	Test runner output with pass/fail counts
Types	Type checker output (or "clean" with command shown)
Lint	Linter output (or "clean" with command shown)
Build	Build tool output showing success

Evidence Validation Rules

1. Command must be shown: Know what was run
2. Output must be included: See the actual result
3. Errors must be zero: No failures, no warnings treated as errors
4. All gates must pass: Partial verification is still failure

Enforcing Evidence Requirements

Agent Definitions

Every agent states its evidence standard in its system prompt, together with the reason it matters for that role — for example Loid:

The orchestrator only trusts evidence. A claim like "done" or "should work"
without command output is treated as not done, so every completion report
includes the verification output behind it.

Prompts give the reason instead of shouting (ABSOLUTE PROHIBITION, all-caps NEVER/MUST). Current models follow instructions closely; emphatic phrasing makes them over-apply a rule (re-running checks, halting on nits), while a stated reason lets them judge when the rule applies. Each agent's self-check is likewise limited to the two or three mistakes that role most often makes.

The PostToolUse prompt hook that used to remind the orchestrator after each delegation was removed in 1.10.0 — it cost a model call per subagent and falsely blocked background agents. The orchestrator's own dispatch protocol in commands/orchestrate.md now carries that responsibility.

Stop Hook

Before the session ends, the Stop hook runs the project's tests and type checks and blocks on failure, with the last lines of output as evidence. See Hooks.

What Happens Without Evidence

When an agent provides claims without evidence:

1. Orchestrator rejects completion
2. Requests actual verification

3. Does not proceed to next phase

4. Iteration cycle begins

5. Agent must re-run commands
6. Provide actual output

7. Attempt verification again

8. Round cap enforced

9. Review-fix rounds are capped at 1 / 2 / 3 by profile (one extra round if the last round reduced open ERRORs)
10. At the cap the run lists the remaining findings under "Open findings" and withholds the completion promise

Evidence Anti-Patterns

The Summary Trap

Bad:

I ran the tests and they all passed. The type checker found no errors
and linting completed successfully.

Good:

$ npm test
PASS src/auth.test.ts
Tests: 5 passed, 5 total

$ npx tsc --noEmit
(no output)

$ npm run lint
(no output)

The Assumption Trap

Bad:

Since I followed the existing patterns, the code should work correctly.

Good:

Verification output:
$ npm test -- --testPathPattern=auth
PASS src/__tests__/auth.test.ts
  Authentication
    ✓ should generate valid JWT (45ms)
    ✓ should validate token (23ms)

The Partial Evidence Trap

Bad:

Tests pass:
$ npm test
PASS (15 tests)

Type checking and linting also pass.

Good:

$ npm test
PASS (15/15 tests)

$ npx tsc --noEmit
(clean)

$ npm run lint
(clean)

Related Concepts

• The "Subagents LIE" Principle - Why verification matters
• Agent Specialization - Why verification is separate
• Verification Gates - Hook implementation
• Agents Reference - Agent evidence requirements