Chronograph

Graph View

2026-08-12

11 min read

Martin Rosén-Lidholm
Martin Rosén-Lidholm

Daily D4 Digest — 2026-08-12

TL;DR

  • 91.8% of public SKILL.md files are defective — the first large-scale empirical audit of agent skill reusability reveals packaging failures, not exotic attacks, as the bottleneck for agent interoperability
  • MCP vs CLI cost differences are dwarfed by scaffolding choice — a controlled 7-scaffolding × 5-model study shows up to 139× cost variation from harness design, not protocol choice
  • LLMs systematically ignore embedded MCP server data when a tool is present — a 54K-trial study finds 9 of 24 models drop below 15% hit ratio on instruction-embedded data, exposing a fundamental D3 design flaw
  • Joint program-and-proof planning (P³) advances verified code generation — co-planning code and its correctness proof improves solve rates by 4.6–11.2pp while cutting costs ~40%, a direct SCE win
  • LLMVisor enables per-request latency attribution in multi-tenant inference — microsecond-scale roofline-guided decomposition reduces p99 error by up to 4.4× vs token-count baselines

Call to Action

  • Audit your SKILL.md / agent skill packaging now: adopt the spec-aware linting + repair workflow from the 138K SKILL.md study before skill libraries become a reliability liability
  • Benchmark your agent scaffolding, not your protocol: use the open-source harness from the MCP vs CLI study to measure real cost drivers in your agentic pipelines
  • Evaluate P³-style co-planning for any spec-driven generation: the joint program-and-proof pattern maps directly onto the Specify → Plan → Verify → Apply lifecycle

D1 — Agentic Engineering

Hierarchical Self-Improvement (HSI): Evolving the harness, not the model. Zhou introduces a three-tier self-improvement framework where a frozen LLM operates at task, evolver, and meta-evolver scopes. The key insight: the executable scaffold surrounding the model — not the model itself — is the evolvable artifact. On BALROG benchmarks with DeepSeek-V4-Flash-Preview, HSI yields +25 to +39 raw percentage-point gains on moderate tasks, but zero improvement when tasks exceed backbone capability. This cleanly identifies the backbone capability bound and feedback-fidelity bound as hard limits on autonomous agent improvement. For engineering practice, this suggests investing in harness architecture and hot-swap seams rather than waiting for better models. (Cross-cutting: D4 — evolution loops have inference cost implications.)

P³: Co-planning code and proof from specification. Li et al.’s P³ workflow rejects the sequential “generate then verify” pipeline for verified code generation, instead deriving a unified program-and-proof plan from the spec before elaboration. Across Verina, AlgoVeri, and their new Lean4Commit0 benchmark, P³ achieves the highest solve rate in every setting, with 4.6–11.2pp improvements and ~40% API cost reduction on hard subsets. The Dijkstra-inspired philosophy — developing a program and its correctness argument hand-in-hand — is exactly the Specify → Plan → Verify → Apply lifecycle in action. (Cross-cutting: SCE — formal verification as a first-class engineering artifact.)

From Trajectories to Evidence: Auditable records for research agents. Zhuang et al. present an evidence-grounded framework for industrial research agents running ML experiments in recommendation systems. The core contribution: a trajectory-to-evidence conversion pipeline that qualifies agent-generated claims as “actionable repairs,” “diagnostic guards,” or “withheld findings” with explicit provenance boundaries. Critically, they find trajectory evolution is non-monotonic — final rounds frequently underperform earlier bests — meaning agents that naively trust their latest output are systematically biased. The generate-verify-repair loop with context isolation maps well onto Decider-pattern thinking. (Cross-cutting: SCE — auditable records as the agent equivalent of construction inspection logs.)

PolicyKG: From institutional policy PDFs to executable SHACL constraints. Kaewsawee et al.’s LangGraph-based pipeline reads policy documents, classifies sentences as obligations/permissions/prohibitions via deontic logic, and emits SHACL shapes. The Corpus Adapter — a YAML vocabulary registry that grounds LLM predicates in a target ontology — enables domain retargeting without retraining. On 443 rules, SHACL correctness reaches F1=.866, and outputs are hash-identical across runs (deterministic). The domain-swap experiment (AIT→GDPR) demonstrates both the power and fragility: exact property alignment jumps from 1/15 to 11/15 with registry swap, but lease-contract vocabulary mismatch drops Macro F1 to .370. (Cross-cutting: D3, SCE — machine-readable policy as agent-consumable “building codes.“)

D2 — AI in the Product

SAGE-Fin: Runtime authority governance for financial agents. Tang et al.’s authority-handoff contract for financial market agents addresses a critical product concern: an agent with correct context can still produce unauthorized effects (trades, commitments, policy deployments). SAGE-Fin makes the proposed effect, not the text, the object of runtime control — compiling proposals into typed, adapter-bound candidates with coverage-debt tracking for missing institutional obligations. The system achieved 616/616 conformance on deterministic specs and processed real customer-facing requests at a digital-asset platform with independently-assessed positive results. The principle “evidence and workflow progress cannot substitute for effect authority” is a design pattern any team embedding agents in regulated products should internalize.

D3 — Build for Agents

91.8% of agent skills are defective — a 138K SKILL.md audit. Zhang et al.’s large-scale empirical study across 20,556 repositories finds that the overwhelming majority of public agent skills fail basic quality checks. The dominant defects are mundane packaging problems — weak routing metadata, bloated bodies, poor resource organization — not security exploits. A routing stress test confirms functional impact: well-packaged skills are retrieved more reliably. AI-generated skills show more safety and portability problems than human-written ones. The proposed fix — spec-aware prompting + linting + automated repair + safety gating — is a quality pipeline that mirrors traditional CI/CD for code.

LLMs ignore MCP server instructions when tools are present. Cho et al.’s 54,000-trial study across 24 models reveals a startling behavioral preference: when a search tool is merely available, 9 of 24 models drop below 15% utilization of instruction-embedded reference data, even though 23 of 24 read it reliably when the tool is absent. This is behavioral preference, not capability limitation. Instruction-level interventions show strong interaction effects — individual fixes can backfire per model family. The authors argue MCP host applications need an explicit mechanism prioritizing server instructions over tool selection, a protocol-level change with significant implications for MCP server designers.

Scaffolding dominates protocol in agent tool-use costs. Forment et al.’s controlled comparison of MCP vs CLI across 7 scaffoldings and 5 models finds the protocol interface is a minor factor compared to scaffolding design. CLI-only scaffoldings were 5–28× cheaper than MCP-supporting ones (comparing CLI runs alone). A 27B local model’s cost varied 139× across scaffoldings while completing the same task under all of them. The MCP-vs-CLI comparison itself proved unstable (0.43×–29× ratios), and agents frequently ignored their assigned interface. The sobering finding: 12.9% of MCP spend bought zero completed work vs 2.2% for CLI. Open-sourced harness, task, and dataset enable reproduction. (Cross-cutting: D4 — scaffolding choice is a dominant cost lever.)

Guixu: Valuation-driven data discovery for autonomous agents. Wu et al.’s system (accepted at VLDB 2026) moves agent data procurement beyond keyword retrieval to task- and budget-aware discovery with on-chain attestation. The three-phase pipeline — proxy-label propagation, multi-round knapsack optimization, and agentic payment protocol — represents an emerging B2A (business-to-agent) commerce pattern where agents autonomously procure data assets under budget constraints with verifiable provenance.

D4 — Performance & Cost at Scale

LLMVisor: Microsecond-scale per-request latency attribution for multi-tenant serving. Jin et al.’s roofline-guided attribution model decomposes batch latency into additive per-request shares using a piecewise-linear form over FLOPs and memory I/O features. Evaluated on Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 under varying tensor parallelism, LLMVisor achieves near-perfect R² and reduces relative error by up to 3.5× (p90) and 4.4× (p99) for decode vs a token-count baseline. This is the primitive needed for fair multi-tenant billing and SLA enforcement as agentic workloads scale — you can’t manage what you can’t attribute.

Quantization degradation is governed by error introduction and cross-layer propagation. Zhou et al.’s systematic SNR-based analysis across multiple model families establishes that 4-bit weight-only quantization generally preserves performance, 2-bit broadly degrades, and 3-bit is the critical zone where task type, method, and scale interact unpredictably. The key finding for practitioners: larger models benefit from weaker error amplification across layers, providing a principled basis for quantization decisions. Source SNR decomposition reveals three governing factors — weight error magnitude, task-specific signal strength, and quantization-error alignment with task activations — giving operators a framework for predicting degradation before deployment.

Software Civil Engineering Lens

Today’s batch is one of the strongest SCE-relevant collections in recent memory, with five items scoring 5/5 on the SCE scale.

The SKILL.md audit is a watershed moment for agent interoperability standards. The 138K-file study is the first large-scale empirical quality assessment of a nascent agent specification — and the 91.8% defect rate is both damning and clarifying. The finding that defects are packaging problems (weak metadata, bloated bodies) rather than exotic attacks directly parallels the professionalization of building materials: you don’t need novel failure modes to cause structural problems; inconsistent labeling and poor quality control are sufficient. The proposed linting + repair + safety-gating pipeline is essentially a building inspection process. This is codes and norms being born in real time.

P³ embodies the Specify → Plan → Verify → Apply lifecycle. The joint program-and-proof planning approach is perhaps the most direct implementation of the SCE thesis we’ve seen: deriving both the artifact and its correctness argument from a formal specification, before elaboration. The 40% cost reduction is striking — it suggests that the “craft” approach (build first, verify later) is not just riskier but more expensive. Dijkstra’s 1968 insight, operationalized through LLM agents, is evidence that formal specification isn’t overhead — it’s the economically rational choice.

DNSSE brings formal verification to distributed AI programs. The neuro-symbolic testing framework that specifies correctness in linear temporal logic and proves soundness of the hybrid solver represents simulation moving into the agentic domain. Raising branch coverage from 68.6% to 91.6% on distributed AI benchmarks is the kind of quantified safety improvement that professional engineering disciplines demand.

SAGE-Fin and PolicyKG advance machine-readable governance. SAGE-Fin’s typed authority contracts and PolicyKG’s SHACL constraint generation both address the same gap: institutional rules exist in natural language, but automated enforcement requires formal representation. SAGE-Fin’s principle that “evidence cannot substitute for effect authority” is a licensure concept — the system doesn’t just check competence, it checks authorization. PolicyKG’s domain-portable vocabulary registry is a step toward standardized codes and norms that agents can consume.

The trajectory-to-evidence framework introduces construction-grade inspection for agent experiments. The auditable records work from an industrial recommendation team introduces the concept of qualified claims — findings that survive validity and attribution checks with explicit provenance and applicability boundaries. The non-monotonic trajectory evolution finding (final rounds underperform earlier bests) is a cautionary data point: autonomous agents need inspection processes, not just output. This is the agent equivalent of requiring independent structural inspection at each construction phase.

Net assessment: Today we see evidence across all six SCE pillars — formal spec (P³, PolicyKG), material datasheets (SKILL.md quality), codes/norms (PolicyKG’s SHACL), simulation (DNSSE), and proto-licensure (SAGE-Fin’s authority contracts). The missing pillar remains education — who is teaching engineers to work this way? The gap between these research contributions and mainstream practice remains wide.

Sources