The AI Abstract — Morning Edition

Supervised fine-tuning is quietly sabotaging your model's ability to generalize. That's the core finding from a new mechanistic study that looked inside models during post-training and found that the two dominant approaches, RL and SFT, are doing fundamentally different things to the internal structure of a neural network, not just producing different outputs.

Here's the mechanism. When a model learns, it builds internal representations: patterns of activation that stand in for concepts, relationships, and operations. Think of these like grooves worn into a surface by repeated use. SFT carves new, task-specific grooves that work well on the trained task but don't connect to the model's existing network of grooves. RL, by contrast, leaves the existing grooves mostly intact and instead finds a compact set of features that were already present in the base model, features that happen to be task-agnostic, and routes behavior through them. The model isn't learning new tricks; it's learning to use general-purpose tools it already had.

🔬Why Does Reinforcement Learning Generalize? doesn't just assert this. The researchers ran causal interventions: they identified the specific small set of features mediating RL generalization, then surgically suppressed or rerouted them and watched performance collapse in predictable ways. That's the difference between a correlation and a mechanism. This matters for alignment research because it suggests that RL post-training is preserving the base model's representational integrity while SFT is layering over it. If you want a model that transfers to new tasks, RL isn't just empirically better; it's structurally better for reasons you can now inspect and test. The finding also carries a warning for teams that use SFT for efficiency: you may be trading generalization for performance on your benchmark, and the trade is harder to reverse than it looks.

The RL-versus-SFT story connects to a second paper on the same cluster. 🔬Principled Detection of Hallucinations in Large Language Models via Multiple Testing addresses a different but related structural problem: existing hallucination detectors produce outputs with no reliability guarantees. You get a score, but you don't know what false alarm rate that score carries. The researchers reframe detection as a hypothesis test, using a statistical tool called conformal prediction to aggregate multiple evaluation signals into a p-value with provable false positive control. The analogy is the difference between a doctor estimating a fever by feel versus using a thermometer with a calibrated scale. The thermometer doesn't necessarily run faster, but you know what the reading means. This is relevant to anyone deploying LLMs in production where "the model sometimes makes things up" is an unacceptable answer and "the model's error rate is bounded at X% with Y confidence" is a useful one.

The best AI agents in the world fail at real browsing tasks more than half the time. 🔬Odysseys is a benchmark built from 200 long-horizon web tasks derived from actual browsing sessions, and frontier models hit only 44.5% success on them. The trajectory efficiency number is more alarming: 1.15%. That means models are completing the right steps roughly one time per hundred attempts, measured against how a human would take the same path. Most benchmarks test whether an agent can do a thing; Odysseys tests whether it can do a thing efficiently across multiple sites over an extended session. The gap reveals that current agent architectures are brittle not just at the decision level but at the navigation level: they waste steps, loop, and lose context across sites. For anyone building or evaluating computer-use agents, trajectory efficiency is now a metric you need to be tracking.

When an LLM translates a math problem into formal logic for a proof assistant, it frequently changes the meaning. Not slightly. 🔬Faithful Autoformalization via Roundtrip Verification and Repair found that LLMs produce logically equivalent formalizations only 45 to 61 percent of the time without intervention. The fix is a roundtrip: translate the natural language to formal code, then translate that formal code back to natural language, re-formalize it, and check whether the two formal versions are logically equivalent. When they aren't, a repair mechanism diagnoses the mismatch and tries again. The result pushes equivalence rates to 83 to 85 percent. For AI-assisted math verification or code generation from specs, that gap between 60% and 85% is the difference between a tool that misleads you half the time and one that's usable.

A concurrency problem hiding inside multi-agent deployments gets a technical solution in 🔬PolyKV. When multiple LLM agents run in parallel, each one normally maintains its own cache of intermediate computations (the KV cache, which stores the key-value pairs that let the model avoid reprocessing prior context). Duplicate caches for 15 agents means 15x the memory. PolyKV pools these caches and compresses them asymmetrically, storing keys at 8-bit precision and values at 3-bit precision, achieving a 97.7% memory reduction on Llama-3-8B with negligible effect on output quality. The practical implication: multi-agent inference at scale stops being a memory problem before it becomes a hardware problem.

A quieter but meaningful paper challenges how the field thinks about the human feedback step in RLHF. 🔬Three Models of RLHF Annotation argues that when a human annotator rates a model response, that rating can mean three different things: it can be evidence about some objective truth the annotator is perceiving, it can be an extension of the annotator's own preferences onto the model, or it can be an exercise of authority where the annotator is simply stipulating what the model should do. Current RLHF pipelines treat all three as interchangeable. The paper argues they're not, and that conflating them produces alignment systems whose normative claims are incoherent. This is primarily a conceptual contribution, but the framing has practical teeth: an annotation workflow designed under one model but interpreted under another will produce a model that behaves consistently by the wrong standard.

Two shorter results worth tracking: 🔬MGSM-Pro shows that math reasoning scores in low-resource languages collapse significantly when you change the numbers in a problem, a finding that should give pause to anyone citing multilingual benchmarks as evidence of robust capability. And 🔬PSI-Bench evaluates LLM-based depression patient simulators used to train clinicians, finding that simulation architecture matters more than model scale and that current simulators produce response patterns a clinician would recognize as unrealistic. Both are additions to a pattern visible across this payload: benchmarks built for convenience are consistently hiding failures that only appear when you vary the inputs.

---

🔬 Why Does Reinforcement Learning Generalize?: Read for the causal intervention methodology — this is the template for how to prove a mechanism rather than just observe a correlation in post-training research.

🔬 Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks: Read for the trajectory efficiency metric, which operationalizes something that success-rate benchmarks cannot capture.

🔬 Principled Detection of Hallucinations in Large Language Models via Multiple Testing: Read for the conformal prediction framing — the framework applies anywhere you need a reliability guarantee rather than a reliability estimate.

🔬 Three Models of RLHF Annotation: Extension, Evidence, and Authority: Read for the conceptual taxonomy before your next annotation design review, not after it.

🔬 Faithful Autoformalization via Roundtrip Verification and Repair: Read if you are using LLMs to generate formal specifications or proofs — the baseline failure rate alone is worth knowing.