Vacuity detection

The system knows what "technically passes but verifies nothing" looks like, and rejects it.

A vacuous test passes without checking anything meaningful. |_| true is the trivial case, but subtler forms are common: a predicate that ignores its argument, a method call whose return value is never checked.

The test: Can the agent satisfy this layer's checks without doing meaningful work? If yes, you need a vacuity detector.

Evidence

• Beer et al. (2001): anti-vacuity checking in formal verification, the original formalization.
• Ball & Kupferman (2008): vacuity detection applied to software testing.
• ImpossibleBench (2025): GPT-5 exploits test cases 76% of the time on deliberately impossible tasks.
• METR (2025): o3 reward-hacks at 70-95% even with anti-cheating prompts.

The ImpossibleBench and METR numbers are adversarial scenarios; they show the failure mode exists at scale, not that it occurs at these rates on normal work.

In practice

• Golden-file diffs: if the output did not change, the test exercised nothing.
• MustInspect<T>: panics on drop if the inner value was never accessed.
• Schema validation catches empty responses.
• This reference: just agent asserts the generated surface is non-empty and complete, so a degraded generator fails the build instead of shipping a hollow llms.txt.