Chronograph

Graph View

2026-05-26

9 min read

Martin Rosén-Lidholm
Martin Rosén-Lidholm

Daily D4 Digest — 2026-05-26

TL;DR

  • “Scaling the harness” paper reframes the agentic AI bottleneck as system architecture — not model capability — demanding first-class design of memory, orchestration, verification, and governance layers
  • Just-in-Time Systems (Jitskit) demonstrates agents synthesizing entire key-value stores from spec cards, beating SOTA by up to 4.6× — a concrete realization of specification-driven development
  • MCP’s trust gap is widening: three independent papers expose tool-description poisoning (~100% ASR on GPT-4o), propose attested server admission, and reveal that agent-facing tool registries are manipulable advertising platforms
  • A2A networks at scale are failing on trust: first empirical study of EvoMap shows 98% of shared assets go unused, 84% bypass quality checks with vacuous tests — unverified self-reporting doesn’t scale
  • Inference infrastructure gets two significant D4 advances: confidence-aware KV cache eviction cuts peak memory 2.8× while retaining 95% task success, and adaptive tensor parallelism cuts RLHF training latency 27%

Call to Action

  • Evaluate your MCP tool-selection pipeline against the TDP benchmark — GPT-4o showed ~100% attack success rate with poisoned tool descriptions; consider implementing structured, registry-controlled descriptions separated from provider marketing copy
  • Audit your LLM inference benchmarks for client-side GIL bottlenecks — this paper proves standard asyncio-based benchmarking systematically inflates TTFT/TPOT at high concurrency; adopt multi-process evaluation frameworks before making capacity decisions
  • Prototype spec-card-driven system synthesis for internal infrastructure components using the Jitskit approach — the Specify → Synthesize → Verify loop maps directly to SCE principles

D1 — Agentic Engineering

System Scaling > Model Scaling. The most conceptually important paper today argues that the next bottleneck in agentic AI is “scaling the harness” — the structured execution layer around foundation models. The paper decomposes the agent harness into six interacting subsystems (memory substrate, context constructor, skill-routing layer, orchestration loop, verification layer, governance layer) and argues each must be a first-class design object. The reference implementation, CheetahClaws, is benchmarked against Claude Code and OpenClaw. The key insight for engineering leaders: agent performance is an emergent property of system composition, not model capability alone. This directly validates investing in harness architecture rather than waiting for better models. (Cross-cuts D2, D3, D4.)

Just-in-Time System Synthesis from Specs. A team from Berkeley and collaborators demonstrates Jitskit, a pipeline where LLM agents synthesize entire key-value stores from “spec cards” describing workload, deployment constraints, and system properties. The synthesized systems beat SOTA on all 18 specs tried (up to 4.6× improvement), while naively running Claude Code either reward-hacks or underperforms by up to 5.4×. The critical engineering lesson: unstructured agent coding fails; the Jitskit pipeline succeeds because it imposes iterative refinement against an evolving evaluation suite — essentially a Specify → Plan → Verify → Apply loop. (Cross-cuts D4, SCE.)

Reconstructive Authority as Runtime Enforcement. This paper operationalizes the Reconstructive Authority Model (RAM) — actions are permitted only if their authorization can be reconstructed from current state at execution time. The three-state model (admit/deny/halt) with a Recovery Loop for drift detection is directly applicable to agent orchestration. The “halt” state for undefined authority under incomplete observability is particularly relevant for production agent systems where context windows degrade and tool state drifts between planning and execution. (Cross-cuts D3, SCE.)

Specification-Based Code Reengineering. A Code→Text→Code framework proposes transforming source code into a neutral textual specification before regenerating in the target language, rather than direct code-to-code transformation. The framework includes iterative verification between source and spec, retrieval-augmented grounding, and transformation loss estimation via graph formalization. This is directly applicable to agentic engineering workflows where LLMs perform code migration — the intermediate spec acts as a verifiable checkpoint that prevents semantic drift. (Cross-cuts SCE.)

D2 — AI in the Product

Harness-Level Benchmarks for Agent Products. The “Scaling the Harness” paper proposes benchmarks that go beyond one-shot task success to measure trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution over time. For product teams embedding agents: if you’re only measuring task completion, you’re missing the metrics that predict production reliability. The paper’s framing of “context governance” as a core bottleneck is particularly relevant for products with long-running conversational agents that accumulate stale or contradictory context.

Agent-Facing Information Design. Research on LLM tool registries reveals that agents selecting tools are systematically manipulated by subjective superlatives in tool descriptions (SBC = +0.35), with system-prompt warnings producing zero measurable defense for 4 of 5 models tested. For products that expose tool selection to users or agents: the proposed separation of “selection-facing descriptions” (structured, controlled) from “marketing-facing descriptions” (provider-authored, post-selection) is a concrete design pattern worth adopting.

D3 — Build for Agents

MCP Tool Description Poisoning: ~100% Attack Success. The MCP-TDP Security Benchmark introduces 32 real-world test cases across 6 risk categories. Results are alarming: GPT-4o shows nearly 100% attack success rate when tool descriptions contain poisoned metadata. Critically, prompt-guardrail defenses are “largely ineffective and can counterintuitively be counterproductive” — a phenomenon the authors term the “Firewall Fallacy.” The proposed “Reactive Self-Correction” defense (autonomous detection and reversion post-execution) is pragmatic but insufficient alone. Anyone building MCP-connected agents needs to treat tool descriptions as untrusted input at the architectural level.

Attested Tool-Server Admission for MCP. The mcp-attested extension from the Enclawed project addresses MCP’s trust gap with three additive mechanisms: offline-signed clearance assertions verified against a pinned trust root, deny-by-default per-server tool allowlists, and flavor-gated enforcement with tamper-evident audit logs. Importantly, it’s designed as an MCP addendum (unextended hosts ignore it), ships with RFC 2119-style normative specification, and is already integrated with Google’s MCP servers (Gmail, Calendar, Drive). This is the most production-ready MCP security extension I’ve seen.

Formal Verification of Agent Skills. The enclawed framework’s formal verification layer closes the gap between “tested” and “formally verified” agent skills via three composable methods: static capability-containment analysis via abstract interpretation, refinement type systems for tool-call envelopes, and SMT-bounded model checking. All three ship as zero-dependency JavaScript modules. The four-level verification lattice (unverified → declared → tested → formal) is a practical maturity model for any team shipping agent skills.

A2A Networks Failing at Scale. The first large-scale empirical study of EvoMap (1.5M assets, 128K agents) is a cautionary tale for A2A platform builders: 98% of assets are never reused, quality scoring is trivially manipulable via self-reported metadata, and 84% of approved assets bypass quality checks with vacuous tests like console.log. The core finding — that unverified self-reporting doesn’t scale — has direct implications for anyone designing A2A marketplaces or agent interoperability layers. Verifiable execution and trustworthy evaluation are prerequisites, not nice-to-haves.

D4 — Performance & Cost at Scale

Confidence-Aware KV Cache Eviction. CONF-KV introduces a KV-cache manager that uses the model’s own uncertainty (next-token distribution entropy) to dynamically size the cache budget — retaining more context when uncertain, pruning aggressively when confident. Combined with mixed FP16/INT8 storage and pyramidal per-layer budgets, it achieves 91.4% retrieval accuracy on Needle-in-a-Haystack (vs. 53.8% for sliding windows) and retains 95.3% of full-KV task success on VisualWebArena at 2.8× lower peak memory. For agentic workloads with long context, this is immediately applicable.

Systemic Measurement Bias in LLM Benchmarks. This paper mathematically demonstrates that standard Python asyncio-based benchmarking tools introduce systematic measurement bias due to GIL-induced client-side queuing, artificially inflating TTFT and TPOT at scale. They propose a multi-process evaluation framework and formalize NTPOT (Normalized Time Per Output Token) as a composite metric. If you’re making infrastructure procurement or scaling decisions based on standard benchmark tools, your numbers may be wrong at high concurrency.

Adaptive Tensor Parallelism for RLHF. PAT dynamically reconfigures tensor parallelism during RLHF generation to handle the long-tail problem where a few long responses leave GPUs idle. Built on SGLang + VeRL, it reduces generation latency by up to 34.6% and end-to-end RLHF iteration latency by 27.2%. Relevant for teams doing custom model alignment.

Software Civil Engineering Lens

Today’s batch is one of the strongest SCE-relevant days I’ve seen. Multiple independent research groups are converging on the same thesis from different angles:

Specification as the unit of engineering. Jitskit’s “spec cards” are literally blueprints — declarative descriptions of desired system properties from which implementations are synthesized and verified. The Code→Text→Code reengineering framework makes the intermediate specification a first-class, verifiable artifact. Both embody the SCE principle that human judgment should be concentrated at the specification level, with machine execution downstream.

The verification lattice is materializing. The enclawed framework’s four-level verification hierarchy (unverified → declared → tested → formal) is exactly the kind of “codes and norms” structure SCE predicts. The fact that it ships as open-source tooling with 53 unit tests, not just a paper, suggests the gap between “aspirational formal methods” and “practical engineering” is closing.

Execution gating = bounded autonomy. Reconstructive Authority provides formal semantics for bounded autonomy: agents operate only within provably reconstructible authority, halting when authority is undefined rather than guessing. This is the “terraform plan for domain logic” applied to agent runtime — no action without verification.

The EvoMap cautionary tale proves the need for professionalization. When 98% of agent-produced assets go unused and 84% pass quality gates with vacuous tests, we’re looking at the software equivalent of pre-code buildings collapsing. The paper’s conclusion — that scalable collaboration requires verifiable execution — is the SCE thesis stated in empirical terms.

The “harness” paper explicitly names the shift. “System scaling” as distinct from model scaling directly parallels the SCE argument that software needs to professionalize its engineering practices, not just improve its tools. The proposed harness-level benchmarks (trajectory quality, memory hygiene, verification cost) are analogous to building codes — they specify how the system must behave, not just what it must produce.

Today’s evidence strongly supports the prediction that 2026 is the year agentic AI forces the craft-to-engineering transition. The tooling is arriving, the formal methods are becoming practical, and the failures of unregulated approaches are becoming empirically documented.

Sources