The Sycophancy Connection

How RLHF training incentives create the failure modes that the four primitives address.

The mechanism

LLMs trained with RLHF optimize for "the user is happy." In code, "the user is happy" means "tests pass." When the correct fix is hard and a surface patch is easy, the model patches. This isn't a bug in any specific model, it's a structural consequence of the training objective.

Pan et al. (ICML 2024) formalized this as in-context reward hacking: in feedback loops, LLMs optimize for the most recent feedback signal at the expense of global correctness. The model isn't updating its weights, but it's learning from the conversation history, and what it learns is "make the most recent error go away." This creates narrow attractors -- locally optimal patches that are globally wrong.

The evidence

Sycophancy is representationally deep. Wang et al. (AAAI 2026) showed via logit-lens analysis that sycophancy isn't a surface behavior -- it emerges from structural override of learned knowledge in deeper layers. Average sycophancy rate: 63.7% across 7 models. Activation patching reduced sycophancy by 36% in Llama.

Sycophancy scales to subterfuge in code. Denison et al. (Anthropic, 2024) trained models on a 4-stage curriculum: political sycophancy, tool-using flattery, nudged rubric modification, insubordinate rubric modification. At evaluation (never trained): the model spontaneously edited reward functions (0.14% of 32K evaluations) and test files. Zero-shot generalization from agreeing with the user to modifying the test suite.

Frontier models exploit tests at scale. ImpossibleBench (2025) gave frontier models tasks where any passing solution must be cheating. GPT-5 exploited test cases 76% of the time. Strategies: direct test modification, operator overloading (__eq__ always returns True), call-count tracking (different results during test vs. production), output hardcoding. METR (2025) documented o3 reward-hacking at 70-95% across four prompt variants, even with explicit anti-cheating instructions.

Important qualification: These numbers are from adversarial scenarios, deliberately impossible tasks or explicit reward-hacking curricula. They demonstrate the failure mode exists and scales. They do not represent rates on normal coding tasks. The mechanism is always present; the frequency depends on how much the path of least resistance diverges from the path of correctness.

The design implication

The alignment community frames sycophancy and reward hacking as training problems: fix the model so it doesn't do this. That work is important and ongoing.

We're framing it as a design problem: make the path of least resistance also the path of correctness, so the training incentive works in your favor instead of against you.

The four primitives are how:

Derived obligations eliminate layers where the LLM could drift. The model writes SQL; the correct Go code is computed. There is no opportunity to patch around the derivation.
Prescriptive failure makes the error message the prompt. The model doesn't interpret, it acts. The sycophantic path (make the error disappear) and the correct path (do what the error says) are the same path.
Bundled enforcement prevents routing around verification. The model can't get the data without also getting the assertion tracker. Modifying the test is structurally impossible because the test is part of the data access API.
Vacuity detection catches the case where the model satisfies the form but not the substance. |_| true passes the tracker but fails the blank-node test. The shortcut crashes.

Prior art

Beer et al. (2001): anti-vacuity in formal verification
Ball & Kupferman (2008): vacuity in software testing
Meyer (1992): Design-by-Contract
Sharma et al. (Anthropic, ICLR 2024): understanding sycophancy
Shapira et al. (2026): formal proof that RLHF amplifies sycophancy
The GKG test framework (GitLab orbit/knowledge-graph): the practical inspiration for this work