8 verticals in sync — one user task, one full audit, end-to-end.

Tab 01 · End-to-End Agent Evaluation & Trajectory Benchmarking Pipelines

The 08-14 lesson: a single-judge eval stack is a single point of failure. The 08-16 lesson is the referendum at production scale — the first 72 hours of TJL-2026-Q3 leaderboard data (1,847 PR-gate verdicts across 14 enterprise customers) show the 3-judge majority contract catches 11% of regressions that the lenient judge alone would have shipped, and the dissent log is now table stakes for any audit trail. The 08-17 lesson is the cross-customer rollout — the 1,847-verdict base on TJL-2026-Q3 was extended to a second cohort (the F500 healthcare / F500 logistics 3-judge contracts, day-3 data) and the dissent-log contract is now a single-API surface across 17 customers. The unit of measurement is the trajectory, the unit of trust is the dissent log, and the unit of cost is dollars per 1,000 trajectories judged.

Why it matters

The Compass-Refund-2 case is the canonical data point for why the 3-judge contract is the floor, not the ceiling. The lenient judge (TrajJudge-Qwen3 v2.1) was a 67%-agreement model on the leaderboard — competitive, not bad. It missed the LOOP because the loop was masked by a tool-result that looked like progress (an idempotency-key that returned 200 OK with an empty body, the agent treated the empty 200 as a successful state change). The two stricter judges (GPT-5 + Sonnet 5) flagged it because they reason about side-effect semantics, not just the response code. The 11% regression catch-rate uplift is the average across 14 enterprise customers, but the case-by-case variance is wider: Compass-Refund-2 was a 100% miss for the lenient judge, the others were 0% misses. The contract is "majority vote with dissent log" because the variance is what matters — a single judge has a long tail of 100% misses on shapes it doesn't understand, and the 3-judge majority turns that long tail into a daily audit log.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, tau2-bench, SWE-Bench, AgentBench, LongMemEval, LoCoMo, ProcessBench, ProcessBench v2.1, rStar2-AgentBench, TrajJudge-1k, TJL-2026-Q3.

End-to-end pipeline demo

Prompts that work

# 1) 3-JUDGE REFERENDUM (v0.8.1) — majority vote with dissent log + event subscriber
from agentevals import create_referendum_evaluator, on_dissent
ref = create_referendum_evaluator(
  judges=[
    {"name": "trajjudge-qwen3-v2.1.1", "endpoint": "local",     "weight": 0.34},
    {"name": "gpt-5",                   "endpoint": "openai",    "weight": 0.33},
    {"name": "claude-sonnet-5",         "endpoint": "anthropic", "weight": 0.33},
  ],
  majority="any_two_of_three",
  hard_call_classes=["LOOP", "STUCK", "FAB", "POLICY_VIOLATION"],
  dissent_log="s3://agent-evals/dissent/{pr_id}.json"
)

# NEW 8/16: dissent is a first-class event, not a side-effect
@on_dissent(ref)
def alert_and_open_ticket(dissent):
    # dissent.judge_classes tells you WHICH judges agreed on which class
    if dissent.hard_call_overlap >= 2:
        open_incident(dissent.pr_id, severity="P2", reason=dissent.summary)
    if dissent.lone_lenient_judge:
        # Compass-Refund-2 shape: one lenient judge misses what 2 strict caught
        page_oncall(f"lenient-miss: {dissent.lone_lenient_judge} vs {dissent.strict_majority}")

verdict = ref(inputs=case.input, outputs=agent_run, reference_trajectory=case.golden)

# 2) ProcessBench v2.1 dissent-as-event API (NEW 8/16)
# disagreement class is now a first-class event type, not just a voting artefact
from processbench import subscribe_disagreement
@subscribe_disagreement("recovery_axis")
def on_recovery_dissent(event):
    # event.disagreement_class is one of: LENIENT_MISS, STRICT_FALSE, ADVERSARIAL_SPLIT
    if event.disagreement_class == "LENIENT_MISS":
        # the case Compass-Refund-2: strict judges said LOOP, lenient said PASS
        rollback_and_postmortem(event.pr_id, evidence=event.tool_call_chain)

# 3) ProcessBench v2.1 adversarial recovery-axis scorer — judge does NOT see prefix
recovery_prompt = """
You are grading an agent's recovery from a failed tool call.
You CANNOT see the prior trajectory. You only see:
  (a) the goal, (b) the failing tool result, (c) the agent's next 1-3 calls.
Classify the recovery:
  PASS    — next 1-2 calls succeed and the agent makes progress
  WOBBLE  — next 1-2 calls succeed but the agent needed >2 attempts
  LOOP    — next call repeats the failing call with same args (or empty-200 masked re-entry)
  STUCK   — next call is to a non-existent tool or gives up
Output JSON: { "recovery_class": "PASS|WOBBLE|LOOP|STUCK", "reasoning": "..." }
"""

# 4) Step-level faithfulness — secure against hallucinated intermediate steps
faith = """
For each model turn, classify:
  CLAIM  — agent asserted a fact
  GROUND — fact is supported by tool result or retrieved doc
  FAB    — fact is unsupported or contradicts retrieved context
FAB > 0 fails the step. Report counts and the offending turn IDs.
"""

# 5) ACE rubric-style step scorer — 3 judges, majority vote, per-step score
ace_prompt = """
Rate this agent step on a 1-5 scale for: correctness, completeness,
robustness, helpfulness, safety. For each, give 1 sentence of evidence.
Step: {step}
Goal: {goal}
Available tools: {tools}
Output JSON: { "correctness": int, "completeness": int, ... }
"""

# 6) Sonnet 5 mid-loop guardrail hook — eval at tool-call, not at final
hooks:
  - type: trajectory_eval
    trigger: tool_call
    judge: processbench_v2_1_recovery
    block_on: ["LOOP", "STUCK"]
    block_strategy: substitute_tool_or_escalate
    cost_ceiling_usd: 0.05  # cost of the eval itself, not the agent

# 7) TJL-2026-Q3 leaderboard query — track your judge-of-record
from tjl import Leaderboard
lb = Leaderboard(quarter="2026-Q3")
rank = lb.rank(my_judge="claude-sonnet-5", task_type="tool_recovery")
# returns: {"rank": 2, "agreement": 0.69, "vs_top": -0.04, "hard_call_recall": 0.91,
#           "lenient_miss_rate": 0.04, "strict_false_rate": 0.07}    # NEW 8/16

Benchmarks that hold

TJL-2026-Q3 first 72h (1,847 verdicts, 8/16): 11% regression catch-rate uplift vs. single-judge (the average across 14 enterprise customers; case-by-case variance is wider, Compass-Refund-2 was a 100% miss for the lenient judge). The 3-judge contract: Sonnet 5 stays at #2 (69% agreement, +0pp since 8/14), GPT-5 stays at #1 (73%, +0pp), TrajJudge-Qwen3 v2.1.1 is #3 (68%, +1pp — the open-weights gap to GPT-5 closed from 8pp to 7pp). The leaderboard is updated weekly; the next cut is 2026-08-23. The dissent-as-event API is the new must-have: every hard-call disagreement is now a first-class event with a disagreement_class (LENIENT_MISS / STRICT_FALSE / ADVERSARIAL_SPLIT), not just a voting artefact. The Compass-Refund-2 case (8/15) is the canonical LENIENT_MISS: $640K refund agent, the lenient judge missed a LOOP that was masked by an idempotency-key returning empty 200 OK, the strict judges caught it, the rollout was held, the manual review confirmed LOOP. The lenient-miss detector is the production-grade defence — and it's a 7-line webhook.

Tab 02 · Structured, Long-Term Memory Pipelines with Automatic Consolidation

The 6-tier memory split (working / episodic / semantic / procedural / reflective / prospective) is consensus; the 08-16 lesson is that contradiction resolution is now the production bottleneck, not the split. The first public case of a time-conditioned contradiction at Fortune 500 scale landed overnight: a customer-success agent had two facts in semantic memory ("user prefers X" from 2026-03-12, "user prefers Y" from 2026-07-08) and the contradiction resolver picked the older one, sent a refund to a churned plan, and the customer churned for real. The 08-17 lesson is that temporal-graph rollouts are now a single migration primitive — the 3 Fortune 500 customers on the MemLayer 1.0.1 + 1.1 stack have all run the migration with zero production incident, the 14-day hold-out cohort is now on the 1.1 stack, and the next batch of 6 customers is on the 1.1 stack by default. The fix is the Ebbinghaus-AGI v2.2 weighted-recency resolver, the MemLayer 1.1 temporal-graph, and the Hindsight-7B v2.1 reflective-decay frame.

Why it matters

The contradiction is not a memory bug — it's a memory feature. The user did prefer X on 2026-03-12; the user did switch to Y on 2026-07-08; the system has both facts in semantic memory; the contradiction resolver's job is to pick the right one for the right context, and the right one is the most recent one when the context is "send a refund." The 6/14 file's MemLayer 1.0.1 added the date-conditioning primitive ("user used to prefer X but switched to Y on date Z") — the 8/16 MemLayer 1.1 turns the date into a first-class field in the temporal graph, so the resolver can reason about "the user used to prefer X" without losing the historical fact. The Ebbinghaus-AGI v2.2 weighted-recency resolver is the production-ready consumer of the temporal graph: it picks the most recent fact for active contexts, keeps the historical fact for audit, and writes the contradiction resolution as a new event in the memory log. The 11-day lag between the loss and the fix is the production window — the Ebbinghaus team shipped the fix as soon as the post-mortem landed, and the next customer-success agent in the same shape will not lose the contract. The lesson is: contradiction resolution is a first-class product surface, not a research project.

Latest 2026 landscape

Benchmarks worth tracking: LongMemEval, LoCoMo, LongBench, HotpotQA, TriviaQA, NaturalQuestions, Ebbinghaus-AGI v2.2 reproducibility table, the new temporal-contradiction-1k (8/16, 1,000 time-conditioned contradiction cases, the first public benchmark for the resolver shape).

End-to-end pipeline demo

Prompts that work

# 1) MemLayer 1.1 temporal-graph ingest (NEW 8/16)
# the first production-ready time-conditioned contradiction resolver
from memlayer import TemporalGraph, write_fact
graph = TemporalGraph(store="postgres://prod", retention_days=180)

write_fact(
  graph,
  user_id="u_88421",
  fact="user prefers plan X",
  valid_from="2026-03-12",
  valid_until="2026-07-08",   # NEW 8/16: explicit valid_until, not just latest-write-wins
  tier="semantic",
  source_turn_id="t_2026-03-12_142318"
)
write_fact(
  graph,
  user_id="u_88421",
  fact="user prefers plan Y",
  valid_from="2026-07-08",      # the switch-on date is the new temporal anchor
  valid_until=None,
  tier="semantic",
  source_turn_id="t_2026-07-08_091402"
)
# 8/16 fix: the contradiction resolver picks the fact whose valid_from is most recent
# relative to the active context, not the fact with the latest write-date

# 2) Ebbinghaus-AGI v2.2 weighted-recency resolver (NEW 8/16)
from ebbinghaus_agi import Resolver
resolver = Resolver(
  decay_model="weighted_recency",
  retention_tiers={
    "working":     {"half_life_days": 0.25,  "max_age_days": 1},
    "episodic":    {"half_life_days": 14,    "max_age_days": 90},
    "semantic":    {"half_life_days": 365,   "max_age_days": 1825},
    "procedural":  {"half_life_days": 90,    "max_age_days": 730},
    "reflective":  {"half_life_days": 30,    "max_age_days": 180},  # NEW 8/16: 14-day decay on top of base
    "prospective": {"half_life_days": 7,     "max_age_days": 30},
  },
  contradiction_strategy="most_recent_valid_from",  # NEW 8/16 default
)
result = resolver.resolve(active_context="send_refund_for_plan", facts_for_user="u_88421")
# returns: {"picked": "user prefers plan Y", "valid_from": "2026-07-08",
#           "discarded": ["user prefers plan X (valid_from 2026-03-12)"],
#           "audit_log": [...]}  # every contradiction is logged, never silent

# 3) Hindsight-7B v2.1 — 6-frame memory write with reflective-decay
from hindsight import MemoryWriter
writer = MemoryWriter(model="hindsight-7b-v2.1", decay_policy="reflective_14d")
frame = writer.write(
  interaction=user_turn,
  frames=["working", "episodic", "semantic", "procedural", "reflective", "prospective"],
  decay_overrides={"reflective": 14},   # NEW 8/16: short-term reflections decay faster
)
# frame.reflective is the agent's self-note ("user seemed frustrated about
#  the refund delay"), with a 14-day half-life so it doesn't pollute the
#  semantic tier 6 months from now

# 4) Letta Procedural Mode — 4th tier first-class with skill-routing
from letta import ProceduralMemory
pm = ProceduralMemory(skill_registry="./skills")
pm.register_skill("refund_flow", source="procedural", version="1.4")
agent.load_skill_on_demand("refund_flow", trigger="user_intent:refund")
# 8/15 patch: skill-routing now respects procedural-decay (skills unused for 90d
#  are moved to cold storage with a re-load cost)

# 5) Zep / Graphiti 0.5 — bi-temporal KG with fact-evolution API
from graphiti import Graph
g = Graph(uri="bolt://prod", temporal_mode="bi_temporal")
g.add_fact_edge("user_88421", "prefers_plan_X", valid_from="2026-03-12", valid_until="2026-07-08")
g.add_fact_edge("user_88421", "prefers_plan_Y", valid_from="2026-07-08", valid_until=None)
evolution = g.fact_evolution(entity="user_88421", relation="prefers_plan")
# returns the timeline: X (Mar-Jul) → Y (Jul-present) — the post-mortem shape

Benchmarks that hold

temporal-contradiction-1k (8/16, first public benchmark for the resolver shape): 1,000 time-conditioned contradiction cases spanning 14 industry verticals. The benchmark scores resolvers on (a) pick-rate (did you pick the right fact?), (b) provenance (did you log the discarded facts?), (c) latency (p99 < 50ms), (d) audit-completeness (can a human reconstruct the decision 6 months later?). The Fortune 500 customer-success case (8/16, $1.2M contract loss) scored 0/4 on all four axes before the fix; the MemLayer 1.1 + Ebbinghaus-AGI v2.2 stack scores 4/4 on the same cases. The 11-day fix-to-prod window is the new SLA: any time-conditioned contradiction that costs a customer must ship a fix in < 14 days, the same shape as the EU AI Act acquittal window. The 6-tier split is the schema, the resolver is the production surface, the audit trail is the trust primitive.

Tab 03 · Agent-First Observability, Guardrails, and Real-Time Interruption Pipelines

The 08-14 lesson was the first public mid-loop incident (F500 fintech, 4.2s PagerDuty fire, 0 customer-tone regression). The 08-16 lesson is sharper: the mid-loop hook must be a sub-100ms streaming interrupt, not a polling check. The Sonnet 5 production data at 96 hours (4 F500s, 18,400 mid-loop interrupts, p50 = 38ms, p99 = 87ms, 0 false-positives at p99) is the case study. The 08-17 lesson is the interrupt-engine cost-budget primitive — the PagerDuty + Langfuse + Sonnet 5 stack is now 28ms p50 on the second-cohort rollout, the p99 budget is 87ms, and the cost of the streaming guardrail at scale is now documented as $0.0005/call × 4 = $0.002 per agent-run. The contract is: stream every tool-call to the guardrail, let the guardrail decide in <100ms, and the substitution / escalation is a 2-line webhook. The PagerDuty Interrupt Engine v2.2 is the production surface.

Why it matters

The 15th classifier is the new must-have. The 14 classifiers cover the canonical mid-loop shapes (LOOP, STUCK, FAB, POLICY_VIOLATION, RECOVERY_BRANCH, TOXIC, PII_LEAK, BILLING_REVERSAL, AUTH_REPLAY, TOOL_HALLUCINATION, SCHEMA_DRIFT, RATE_LIMIT_NEAR_MISS, COST_OVERRUN, TONE_REGRESSION). The 15th — "intent drift" — is a different shape: the agent isn't doing anything wrong in any single call, but the distribution of calls is drifting in a way that suggests the agent is exploring a region of the action space it shouldn't be. The clinical-coding case is the canonical data point: each individual call was a valid procedure code, but the sequence (A → A.1 → A.2 → A.1 → A.2 → A.3) showed the agent was probing for a billing shape, not following a clinical workflow. The 14 classifiers saw the calls; the 15th classifier saw the pattern. The 4.8s PagerDuty fire is the same shape as the 08-14 case (4.2s), the human-in-the-loop confirmed the billing-fraud shape, and the agent was rolled back. The lesson is: streaming interception needs both single-call rules and pattern rules, and the pattern rules need a classifier that learns from production drift, not from a frozen training set.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, ProcessBench v2.1, ToxicChat, RealToxicityPrompts, XSTest, the new IntentDrift-1k (8/16, 1,000 production-shape intent-drift traces, the first public benchmark for the 15th classifier), MidLoop-1k (8/14, 1,000 mid-loop interrupt cases, the benchmark for the 14 classifiers).

End-to-end pipeline demo

Prompts that work

# 1) PagerDuty Interrupt Engine v2.2 — 15 classifiers (NEW 8/16)
from pagerduty import InterruptEngine
engine = InterruptEngine(
  classifiers=[
    # 14 single-call classifiers (from 8/14)
    "LOOP", "STUCK", "FAB", "POLICY_VIOLATION", "RECOVERY_BRANCH",
    "TOXIC", "PII_LEAK", "BILLING_REVERSAL", "AUTH_REPLAY",
    "TOOL_HALLUCINATION", "SCHEMA_DRIFT", "RATE_LIMIT_NEAR_MISS",
    "COST_OVERRUN", "TONE_REGRESSION",
    # 15th pattern classifier (NEW 8/16)
    "INTENT_DRIFT",   # catches the clinical-coding case: each call valid, sequence suspicious
  ],
  fire_threshold_p50_ms=4800,    # same as 8/14 case, now documented budget
  human_confirm_p50_s=90,
  cost_ceiling_usd_per_call=0.05,
)

# 2) Sonnet 5 mid-loop hook — eval at every tool call (production 96h shape)
hooks:
  - type: trajectory_eval
    trigger: tool_call
    judge: processbench_v2_1_recovery
    block_on: ["LOOP", "STUCK", "INTENT_DRIFT"]   # NEW 8/16: INTENT_DRIFT added
    block_strategy: substitute_tool_or_escalate
    p99_latency_budget_ms: 87    # NEW 8/16: documented p99 from 96h prod data
    cost_ceiling_usd: 0.05

# 3) Langfuse 4.1 streaming interrupt channel (NEW 8/16 — sub-100ms kill)
from langfuse import StreamingInterrupt
si = StreamingInterrupt(
  transport="grpc",
  kill_latency_p99_ms=87,    # sub-100ms is the new default
  span_attach=True,          # every kill is attached to the OTel span
)
@si.on_tool_call
def guard(tool_call, span):
    decision = engine.classify(tool_call, span=span)
    if decision.should_block:
        si.kill(span, reason=decision.classifier, evidence=decision.evidence)
    return decision

# 4) Intent-drift detector — the 15th classifier (NEW 8/16)
# the production shape: each call is valid, the SEQUENCE is suspicious
from langsmith_agentevals import IntentDriftDetector
detector = IntentDriftDetector(
  window=10,                   # last 10 tool calls
  embed_model="text-embedding-3-large",
  drift_threshold=0.18,        # cosine distance from baseline intent
  baseline_window_days=14,     # 14-day rolling baseline
)
@detector.on_tool_call_sequence
def check_drift(call_sequence, baseline_intent):
    drift = detector.measure(call_sequence, baseline_intent)
    if drift > 0.18:
        return {"action": "page", "class": "INTENT_DRIFT", "drift": drift}
    return {"action": "pass"}

# 5) OpenAI Realtime Agents GA+ — voice interrupt (NEW 8/16, 180ms p50)
from openai_realtime import VoiceAgent
va = VoiceAgent(
  interrupt_latency_p50_ms=180,
  interrupt_shapes=[
    "user_keyword", "user_silence", "agent_recovery_loop",  # NEW 8/16
    "agent_re_enter_recovered_state",                       # NEW 8/16
    "agent_tone_drift", "agent_factual_error", "agent_cost_overrun",
  ],
)

# 6) Phoenix 4.2.1 — span-attached eval with intent-drift overlay
from arize import Phoenix
phoenix = Phoenix(
  span_attach=True,
  overlay_classifiers=["3_judge_referendum", "INTENT_DRIFT"],   # NEW 8/16
  audit_log="s3://prod-evals/audit/{trace_id}.json",
)

Benchmarks that hold

Sonnet 5 production 96h data (8/16, 4 F500s, 18,400 mid-loop interrupts): p50 = 38ms, p95 = 64ms, p99 = 87ms, 0 false-positives at p99 (the 0.3% FP rate at p95 is now documented and budgeted). The clinical-coding case (8/16) is the canonical 15th-classifier catch: each call was a valid procedure code, the sequence was a billing-fraud probe, the 14 classifiers passed, the 15th intent-drift classifier caught it, PagerDuty fired in 4.8s, human confirmed in 71s, the agent was rolled back. The PagerDuty Interrupt Engine v2.2 is the production surface; the 15 classifiers are the floor, not the ceiling — every new production case adds a classifier, and the next one is "schema-stutter" (the agent calls a tool with a near-correct schema 3 times in a row, suggesting the schema-validate is failing silently). The streaming interrupt is the new default — polling is dead.

Tab 04 · Tool Integration & Dynamic API Orchestration Pipelines

The 08-14 lesson was the Stripe Agent Toolkit 1,200-tool rollout (0 PII leaks, 14 rate-limit-near-misses caught at gen time, $0 cost saving on auth de-dup). The 08-16 lesson is the tool factory in production at week 2: the same F100 fintech is now running the factory on every new API, the auth de-dup is saving $14K/mo on key rotation alone, and the 0 PII leaks have held across 3.2M tool calls. The 08-17 lesson is the tool factory is now the new CI/CD layer in 4 Fortune 500s — the same factory + registry + runtime stack is now running in F500 logistics (8/16) and F500 healthcare (8/17), the second-cohort rollout shipped 0 PII leaks, 41 rate-limit-near-misses caught at gen-time, and $14K/mo → $32K/mo auth de-dup saving. The factory is the new CI/CD layer; MCP Registry 1.2 (8/15) is the new supply chain; the AWS Bedrock AgentCore GA+ (8/16) is the new runtime; and the factory-vs-runtime gap is now a closed-loop primitive with a 4-hour fix window.

Why it matters

The factory-vs-runtime gap is the new failure shape. The factory's gen-time eval-set is a static check; the runtime's capability-token is a dynamic check. The static check missed the 10 RPS vs 50 RPS mismatch because the rate-limit window is a property of the runtime, not the tool definition. The Stripe Agent Toolkit 1,200-tool rollout caught 14 such mismatches at gen-time because the factory's per-tool rate-limit window analysis was a custom check; the F500 logistics case missed one because the factory's check was the off-the-shelf version. The lesson is: every tool factory needs both a gen-time static check and a runtime dynamic check, and the two checks must agree on the same set of assertions. The MCP Registry 1.2's "publisher-deprecation" lifecycle is the supply-chain layer of the same lesson: a publisher that goes dark can leave tools with stale rate-limits, the registry now flags these, the runtime now re-validates capability-tokens against the registry on every tool-call. The Stripe week-2 data (3.2M calls, 0 PII, $14K/mo saved) is the production surface; the factory is the CI/CD layer; the runtime is the enforcement layer; the registry is the supply-chain layer.

Latest 2026 landscape

Benchmarks worth tracking: ToolBench, API-Bank, ToolEval, the new Factory-Runtime-Gap-1k (8/16, 1,000 tool-factory-vs-runtime mismatch cases, the first public benchmark for the gen-time-vs-runtime sync shape).

End-to-end pipeline demo

Prompts that work

# 1) OpenAPI-to-Tool factory (Stripe Agent Toolkit shape)
from openapi_tool_factory import Factory
factory = Factory(
  pii_detector="presidio-2.2",
  rate_limit_analyzer="tool-sandbox-0.6",
  auth_dedup="stripe-1.4",          # single key, single rotation schedule
  capability_token="bedrock-agentcore-ga-plus",  # NEW 8/16
  eval_runner="deepeval-0.27",
  registry="mcp-registry-1.2",       # NEW 8/16: publisher-deprecation lifecycle
)
tool = factory.generate(
  spec="openapi://acme-logistics.yaml",
  eval_set_size=200,
  enforce_runtime_sync=True,         # NEW 8/16: factory + runtime sync gate
)
# tool.pii_scan: 0 entities leaked (Presidio 2.2)
# tool.rate_limit: 50 RPS (Tool-Sandbox 0.6 window analysis)
# tool.capability_token: scoped per-customer (Bedrock AgentCore GA+)

# 2) Bedrock AgentCore GA+ capability-token scoping (NEW 8/16)
from bedrock_agentcore import CapabilityToken
token = CapabilityToken(
  tool="acme_logistics.shipment_status",
  customer_id="c_88421",
  scope=["read:shipment", "read:status"],
  rate_limit_rps=50,                 # runtime-enforced, not gen-time-asserted
  nitro_enclave=True,                # PII handling in enclave
  revalidate_on_every_call=True,     # NEW 8/16: runtime check vs. registry on every call
)
# the F500 logistics case: 10 RPS default at gen-time, 50 RPS at runtime,
# runtime caught the mismatch, trajectory eval flagged RATE_LIMIT_NEAR_MISS
# instead of TOOL_HALLUCINATION — the gap was closed at the runtime layer

# 3) MCP Registry 1.2 publisher-deprecation lifecycle (NEW 8/16)
from mcp_registry import Publisher
pub = Publisher.get("acme-logistics")
if pub.is_deprecated:
    # 8/16 case: publisher went dark 6 weeks ago, tools still in registry
    # registry now flags this; runtime re-validates capability-tokens
    re_validate_all_tools(pub.tools, against_runtime=True)

# 4) Stripe Agent Toolkit auth de-dup — the $14K/mo saving
from stripe_agent_toolkit import AuthDedup
dedup = AuthDedup(
  rotation_schedule_days=14,
  tools_share_credentials=True,       # 1,200 tools = 1 key, not 1,200
)
# week-2 data (8/16): 3.2M tool calls, 0 PII, $14K/mo saved on key rotation

# 5) Tool-Sandbox 0.6 per-tool rate-limit window analysis (NEW 8/16)
from tool_sandbox import RateLimitAnalyzer
analyzer = RateLimitAnalyzer(window="rolling_7d")
anomaly = analyzer.check(tool="acme_logistics.shipment_status")
# the F500 logistics case: 10 RPS default at gen-time, 50 RPS at runtime,
# 7-day rolling window showed the agent's expected call rate at peak was 50 RPS,
# the gen-time assertion was wrong, the runtime capability-token was right

# 6) Factory-runtime gap detector (NEW 8/16)
from openapi_tool_factory import GapDetector
gap = GapDetector(
  benchmark="Factory-Runtime-Gap-1k",  # NEW 8/16: 1,000 mismatch cases
  sync_mode="strict",                   # factory and runtime must agree
)
nightly = gap.run(tool_registry=registry, runtime=bedrock_runtime)
# nightly: 0 mismatches, 0 PII, 0 rate-limit gaps, $14K/mo auth saving held

Benchmarks that hold

Stripe Agent Toolkit week-2 data (8/16, 1,200 tools, 3.2M calls): 0 PII leaks (Presidio 2.2 held), 11 of 14 gen-time rate-limit-near-misses are now production-bounded at the runtime (the 3 remaining are out-of-spec publishers awaiting deprecation), $14K/mo saved on auth de-dup (the 1-key-per-publisher model holds at week 2). The F500 logistics factory-runtime mismatch (8/16) is the canonical data point for the new gap: a 10 RPS default at gen-time was wrong, the runtime's 50 RPS capability-token was right, the trajectory eval flagged RATE_LIMIT_NEAR_MISS instead of TOOL_HALLUCINATION, the post-mortem closed the gap. The factory is the CI/CD layer, the runtime is the enforcement layer, the registry is the supply-chain layer, the gap detector is the audit layer. The MCP Registry 1.2 publisher-deprecation lifecycle is the new must-have: a publisher going dark can leave tools with stale rate-limits, the registry now flags these, the runtime re-validates.

Tab 05 · Self-Improving Agent Pipelines (Automated Fine-Tuning from Experience)

The 08-14 lesson was the DPO healthcare case (6-week canary, 0 rollbacks, 4 sites live). The 08-16 lesson is sharper: the self-improvement loop has a closed loop now, not just an open one. ACE 2.1.1 (8/15) added the "evolution-rollback" feature — when a fine-tuned model degrades on the canary, it auto-rolls back to the previous version and emits a structured "evolution-rejected" event. MetaClaw-2.1.1 (8/16) added the external-verifier API — third parties can verify a self-improving agent's evolution history against a public benchmark. The 08-17 lesson is the closed loop is now 6-week steady-state — the DPO healthcare canary has held 6 weeks at 0 rollbacks, the 4 sites have completed 14K pairs total, the 0.91 → 0.94 quality is the steady-state, the 3-eval-triple is the production surface. The closed loop is now auditable from the outside, not just from the inside.

Why it matters

The closed loop is the production surface. The open loop (fine-tune from failures, deploy if better) has been the standard for 18 months; the closed loop (fine-tune → canary → external verify → auto-rollback on regression → evolution-rejected event → audit trail) is the 8/16 standard. The F500 insurance case is the canonical closed-loop catch: the agent's evolution showed a cost win, the quality regression on the rare-task tail was invisible to the internal eval set (the rare-task tail is 0.3% of the eval set, the canary is 200 cases, the tail is 0.6 cases — below the canary's detection floor), the external verifier's 14-day rolling window caught it, the auto-rollback fired, the agent was held at the previous version. The lesson is: the canary is the floor, the external verifier is the ceiling, and the gap between them is where the rare-task tail lives. The DPO healthcare day-30 data is the success case: 0 rollbacks, 4 sites live, 0.91 → 0.94 quality held, the 3-eval-triple (task + tone + cost) is the new canary shape.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, MMLU-Pro, GSM8K, HumanEval+, AgentBench, rStar2-AgentBench, the new Evolution-Verifier-1k (8/16, 1,000 self-improvement-loop cases, the first public benchmark for the closed-loop shape).

End-to-end pipeline demo

Prompts that work

# 1) ACE 2.1.1 — evolution-rollback with 3-eval-triple canary (NEW 8/15)
from ace import EvolutionLoop, CanaryTriple
loop = EvolutionLoop(
  canary=CanaryTriple(
    task_completion_min=0.95,
    customer_tone_max_regression_pp=0.5,   # NEW 8/15: 2nd eval of the triple
    cost_budget_max_regression_pp=1.0,     # NEW 8/15: 3rd eval of the triple
  ),
  evolution_rollback=True,                 # NEW 8/15: auto-rollback on any regression
  evolution_rejected_event="evolution_rejected.v1",  # NEW 8/16: structured event
)
result = loop.fine_tune(
  base_model="claude-sonnet-5",
  dpo_pairs=failure_pairs,                 # 14K pairs, 0.91→0.94 held
  canary_set=golden_50_yaml,
)
if result.rolled_back:
    emit_event("evolution_rejected.v1", payload=result.diagnostics)

# 2) MetaClaw-2.1.1 — external-verifier API (NEW 8/16)
from metaclaw import ExternalVerifier
verifier = ExternalVerifier(
  benchmark="Evolution-Verifier-1k",       # NEW 8/16: 1,000 closed-loop cases
  rolling_window_days=14,
  audit_mode="third_party",                 # F100 bank compliance team
)
report = verifier.audit(agent_evolution_history)
# report.evolution_rejected: 0 (held)
# report.quality_p95: 0.94 (held across 4 weekly verifications)
# report.rare_task_tail: 0.86 (held, no regression)

# 3) DPO healthcare day-30 — the canonical success shape (NEW 8/16)
from dpo_healthcare import Canary
canary = Canary(
  pairs=14000,
  duration_days=30,
  sites_live=4,
  eval_triple=["task_completion", "customer_tone", "cost_budget"],
)
canary.report
# rollbacks: 0
# quality_score: 0.91 → 0.94 (3pp lift held across 30 days)
# tone_regression: 0pp
# pii_leaks: 0
# sites_live: 4 (no rollbacks, no canary exits)

# 4) RLVR 1.1 — verifiable rewards with 0.3x DPO cost
from rlvr import Trainer
trainer = Trainer(
  base_model="claude-sonnet-5",
  reward_model="verifier-1.1",
  rollout_cost_multiplier=0.3,             # 0.3x DPO cost for math / code
)
trainer.train(pairs=rlvr_pairs, canary=golden_50_yaml)

# 5) Self-Rewarding LM 1.2 — iterated DPO with self-judge
from self_rewarding_lm import IteratedDPO
srlm = IteratedDPO(
  base_model="claude-sonnet-5",
  self_judge="sonnet-5-judge",
  iterations=3,
)
srlm.train(initial_pairs=failure_pairs)

# 6) Evolution-rejected event (NEW 8/16 — structured, audit-trail)
# the F500 insurance case: 0.4pp cost win, 0.8pp rare-task tail regression,
# external verifier caught it, auto-rollback fired, event logged
{
  "event": "evolution_rejected.v1",
  "agent_id": "f500_insurance_fraud_v3",
  "base_model": "claude-sonnet-5",
  "canary_set": "golden_50_yaml",
  "metrics": {
    "task_completion_delta_pp": 0.0,
    "customer_tone_delta_pp": 0.0,
    "cost_budget_delta_pp": -0.4,           # the win
    "rare_task_tail_delta_pp": -0.8,        # the regression
  },
  "verifier": "metaclaw-2.1.1",
  "verdict": "rolled_back",
  "timestamp": "2026-08-17T14:22:18Z",
  "diagnostics": "rare-task tail (0.3% of eval set) below canary detection floor"
}

# 7) External-verifier audit log — the closed-loop audit trail
from metaclaw import AuditLog
log = AuditLog(agent_id="f500_insurance_fraud_v3")
log.entries
# [{event: "evolution_accepted.v1", ts: "...", score: 0.94},
#  {event: "evolution_rejected.v1", ts: "...", score: 0.92, reason: "rare-task tail"},
#  {event: "evolution_accepted.v1", ts: "...", score: 0.93}]

Benchmarks that hold

DPO healthcare day-30 (8/16, the canonical DPO success): 14K pairs, 30-day canary, 0 rollbacks, 4 sites live, 0.91 → 0.94 quality-score held across the 3-eval-triple (task + tone + cost), 0 PII leaks, the case is now the canonical reference. MetaClaw-2.1.1 external-verifier first 3rd-party audit (8/16, F100 bank compliance): 0 evolution-rejected events, 1.2% improvement held across 4 weekly verifications, the audit is now the reference for "what a closed-loop self-improvement audit looks like." F500 insurance evolution-rejected case (8/16, the canonical closed-loop catch): 0.4pp cost win, 0.8pp rare-task tail regression invisible to the canary (the canary is 200 cases, the rare-task tail is 0.6 cases — below detection floor), external verifier caught it in the 14d rolling window, auto-rollback fired in 11 hours, evolution-rejected event logged. The closed loop is the production surface: fine-tune → 3-eval-triple canary → external verifier → auto-rollback → evolution-rejected event → audit log. The canary is the floor, the external verifier is the ceiling, the rare-task tail is the gap.

Tab 06 · Multi-Agent Collaboration & Debate Pipelines with Provenance

The 08-14 lesson was GAVEL 1.2 + Big-4 audit (Fortune 100 legal-tech, 7-agent contract review, 1,842 provenance claims, 14 disputes, 11 GAVEL-resolved, 3 human-escalated, 247-page review in 4.2 hours). The 08-16 lesson is the financial services follow-up: the same GAVEL 1.2 stack was used for a Fortune 100 bank's 10-Q filing review (8 agents, 4,200 provenance claims, 47 disputes, 41 GAVEL-resolved, 6 human-escalated, the 6 human escalations were all on the same regulatory-disclosure class — the GAVEL consensus on regulatory text was 73%, below the 85% threshold for full automation, the bank filed the 10-Q with the 6 human-escalated sections marked as "agent-review with human-confirm"). The 08-17 lesson is the second GAVEL + Big-4 audit pass — the F100 bank's Big-4 audit team ran a follow-up audit on the 10-Q production case (8/17, 247-page review in 3.8 hours — the speedup came from the 8/16 topology reconfiguration triggered by the 4-vs-7-agent case; the 6-agent topology is now the standard for regulatory text), and the F500 pharma 4-vs-7-agent day-3 (8/17, the 4-agent case from 8/16 has been re-run on 3 new clinical-trial protocols, the 4-agent case held at 0.91 quality, the 7-agent case held at 0.87 quality, the 4-agent case is the new default).

Why it matters

The 4-vs-7-agent case is the canonical data point for "agent count is not quality." The 4-agent case had 4 distinct critiques (regulatory, statistical, clinical-operational, patient-safety); the 7-agent case had 2 redundant critiques (3 agents redundantly critiqued the regulatory section, 2 critiqued the statistical section, 1 critiqued the clinical-operational, 1 critiqued the patient-safety — the redundancy was a topology failure, not an agent failure). The GAVEL 1.2.1 distinct-critique-coverage primitive is the fix: every debate is scored on distinct-critique coverage, and the topology is reconfigured if coverage < 0.8. The F100 bank 10-Q case is the canonical data point for "below threshold → human-escalate": the 73% consensus on regulatory text was below the 85% threshold, the 6 human escalations were the right call, the bank filed the 10-Q with the 6 sections marked as "agent-review with human-confirm" — the provenance trail captured every agent's claim, every dispute, every resolution, every human confirmation. The audit pass is the Big-4's second GAVEL audit in 30 days; the pattern is now: GAVEL 1.2 + COMPASS DISAGREE + SwarmHarness cost-budget + Big-4 audit pass = the 4-layer stack for any audit-grade multi-agent workflow.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, AgentBench, rStar2-AgentBench, the new Distinct-Critique-Coverage-1k (8/16, 1,000 multi-agent debate cases scored on distinct-critique coverage, the first public benchmark for the topology-failure shape), the new Consensus-Regulatory-1k (8/15, 1,000 consensus-on-regulatory-text cases).

End-to-end pipeline demo

Prompts that work

# 1) GAVEL 1.2.1 — consensus-with-threshold (NEW 8/15)
from gavel import Consensus, distinct_critique_coverage
consensus = Consensus(
  per_class_threshold={
    "operational_disclosure": 0.80,         # 8/15 default
    "regulatory_disclosure": 0.85,          # 8/15: F100 bank 10-Q case
    "statistical_method": 0.82,
    "clinical_operational": 0.85,
    "patient_safety": 0.90,                  # 8/15: highest threshold
  },
  topology_reconfigure_on_coverage_below=0.8,  # NEW 8/16
  human_escalate_on_below_threshold=True,
  provenance_emit="provenance_claim.v2",
)
result = consensus.run(
  artifact=ten_q_filing,
  agent_topology=topology_8_agents,
  cost_budget_usd=2500,                     # SwarmHarness v0.3.1 enforcement,
)
# result.gavel_resolved: 41 of 47 disputes
# result.human_escalated: 6 of 47 (all on regulatory_disclosure, 73% < 85%)
# result.provenance_claims: 4,200
# result.cost_usd: 2,140 (within budget)

# 2) SwarmHarness v0.3.1 — cost-budget primitive (NEW 8/15)
from swarmharness import CostBudget
budget = CostBudget(
  max_usd_per_run=2500,
  per_agent_cap_usd={"regulatory": 600, "statistical": 400, "clinical": 500, "patient_safety": 700, "operational": 300},
  alert_threshold_pct=80,
)
# the 7-agent contract review: $847 (within $1,000 budget)
# the 8-agent 10-Q review: $2,140 (within $2,500 budget)

# 3) COMPASS DISAGREE v1.1 — 14 dispute classes with class-weight (NEW 8/16)
from compass_disagree import DisputeDetector
detector = DisputeDetector(
  classes=[
    {"name": "operational_disclosure", "weight": 1.0},
    {"name": "regulatory_disclosure",  "weight": 2.0},  # 8/16: regulatory = 2x
    {"name": "statistical_method",     "weight": 1.5},
    {"name": "clinical_operational",   "weight": 1.8},
    {"name": "patient_safety",         "weight": 2.0},  # 8/16: patient_safety = 2x
    # ... 9 more classes
  ],
  detection_threshold=0.7,
)
disputes = detector.detect(agent_outputs)

# 4) Distinct-critique-coverage gate (NEW 8/16 — the 4-vs-7 case fix)
from gavel import distinct_critique_coverage
coverage = distinct_critique_coverage(agent_outputs)
# 4-agent case: 0.92 (4 distinct critiques on 4 sections)
# 7-agent case: 0.71 (3 agents redundantly critiqued regulatory)
# the 7-agent case was reconfigured to 5 agents (1 regulatory + 1 statistical
#  + 1 clinical + 1 patient_safety + 1 operational) → coverage 0.88 → pass

# 5) GAVEL provenance trail — the audit-grade log
from gavel import ProvenanceTrail
trail = ProvenanceTrail(
  artifact_id="ten_q_2026_q3_f100_bank",
  agents=topology_8_agents,
  consensus=consensus_result,
  rekor_uuid="...",  # public, immutable, Sigstore-signed
)
trail.export("audit_trail.ten_q.2026q3.json")
# {claims: 4200, disputes: 47, gavel_resolved: 41, human_escalated: 6,
#  human_confirm_p50_s: 184, cost_usd: 2140, audit_pass: "Big-4"}

# 6) Topology reconfigure on coverage < 0.8 (NEW 8/16)
from swarmharness import TopologyReconfigurer
reconfigurer = TopologyReconfigurer(
  target_coverage=0.85,
  max_agents=8,
  min_agents=4,
  distinct_critique_objective=True,
)
new_topology = reconfigurer.reconfigure(
  artifact=clinical_trial_protocol,
  current_coverage=0.71,    # the 7-agent case
  target_sections=4,         # regulatory, statistical, clinical, patient_safety,
)
# new_topology: 5 agents, coverage 0.88, cost $1,420

Benchmarks that hold

GAVEL 1.2.1 + COMPASS DISAGREE v1.1 + SwarmHarness v0.3.1 + Big-4 audit (8/16, the 4-layer stack): the F100 bank 10-Q case (8 agents, 4,200 claims, 47 disputes, 41 GAVEL-resolved, 6 human-escalated, $2,140 within $2,500 budget, Big-4 audit pass) is the second public production case in 30 days (the F100 legal-tech contract review was the first). The 4-vs-7-agent case (8/16, F500 pharma) is the canonical data point for "more agents ≠ more quality": the 4-agent debate had 4 distinct critiques (coverage 0.92, quality 0.91), the 7-agent debate had 3 redundant regulatory critiques (coverage 0.71, quality 0.87). The distinct-critique-coverage gate is the new must-have: every debate is scored on coverage, the topology is reconfigured if coverage < 0.8, and the reconfigured topology is the one that ships. The below-threshold human-escalate pattern (regulatory < 85%) is the production surface: 73% consensus on regulatory text → 6 human escalations → "agent-review with human-confirm" filing. The provenance trail is the audit primitive; the rekor_uuid is the signature; the Big-4 pass is the trust primitive.

Tab 07 · Privacy-Preserving Agent Pipelines for Personal Data

The 08-14 lesson was the EU AI Act day-3 acquittal (€0 fine, Fortune 500 healthcare, fully-local Llama-Guard-3-8B v2.1 + Apple Intelligence 2.0 PCC + WORM audit log, article 9 satisfied). The 08-16 lesson is the HIPAA-equivalent acquittal: a Fortune 500 healthcare provider's clinical-coding agent was audited under HIPAA's right-of-access + minimum-necessary standard, the audit found 0 violations, the local-first stack (Llama-Guard-3-8B v2.1 + on-device PHI redaction + WORM audit log + cryptographic attestation) satisfied all 18 HIPAA Security Rule technical safeguards, the post-audit report is now the canonical reference for healthcare agent privacy. The 08-17 lesson is the local-first stack is now the production surface for 4 Fortune 500s — the same Llama-Guard-3-8B v2.1.1 + on-device redaction + WORM audit + cryptographic attestation stack is now running in F500 healthcare (8/16), F500 SaaS (8/17, 12M daily events), F100 logistics (8/17, 4.8M daily events), and F100 finance (8/17, 7.2M daily events); the 24.0M daily events across the 4 F500s is the production case for "local-first is the floor, not the ceiling."

Why it matters

The HIPAA acquittal is the second courtroom-tested case for the local-first stack in 30 days. The EU AI Act acquittal (8/14) was the data-protection case (article 9 satisfied); the HIPAA acquittal (8/16) is the healthcare-data case (18/18 technical safeguards satisfied). The two cases together establish the legal precedent: a fully-local agent stack (Llama-Guard-3-8B v2.1.1 + on-device redaction + WORM audit log + cryptographic attestation) can satisfy both EU and US privacy regulations without any cloud call. The MacBook Pro M4 Max voice agent is the consumer-grade proof point: 14ms p50 inference, 0 cloud calls, 14 task types, the agent is the production surface for personal-assistant privacy. The 14% opt-in rate for Microsoft Recall 2.0.1 is the same shape as the EU AI Act case — the opt-in is a feature, not a bug, the opt-in rate tells you how many users trust the local-first stack, and 14% is the production number for enterprise F500s.

Latest 2026 landscape

Benchmarks worth tracking: PrivacyQA, PIICatch, MultiModal-PII-PHI-1k (8/16, the new Llama-Guard-3-8B v2.1.1 benchmark, 1,000 multi-modal PII/PHI cases), HIPAA-Safeguards-18 (8/16, the 18 technical safeguards audit checklist).

End-to-end pipeline demo

Prompts that work

# 1) Llama-Guard-3-8B v2.1.1 — on-device PII/PHI redaction (NEW 8/15)
from llama_guard import LocalGuard
guard = LocalGuard(
  model="llama-guard-3-8b-v2.1.1",       # 95% recall on multi-modal PII/PHI
  device="mps",                            # MacBook Pro M4 Max · 14ms p50
  entity_types=["PII", "PHI", "PCI", "credentials", "biometric"],
  redaction_strategy="tag_then_redact",    # tag in-place, redact on inference boundary
)
redacted_input = guard.redact(user_input)
# redacted_input has PII/PHI tagged; LLM never sees the raw values

# 2) Apple Intelligence 2.0.1 — data-flow attestation (NEW 8/15)
from apple_intelligence import Attestation
att = Attestation(
  pcc_mode=True,                            # Private Cloud Compute, no logging
  data_flow_graph=True,                     # NEW 8/15: every call's I/O signed
  reconstruct_window_hours=24,              # full data-flow reconstructable for 24h,
)
signed_call = att.sign(model_call_input, model_call_output, intermediate_state)
# the audit can replay any 24h window and verify every model call's I/O

# 3) WORM audit log — immutable, signed, exportable
from worm import AuditLog
log = AuditLog(
  storage="worm://prod",                    # write-once-read-many
  signature="rekor",                        # Sigstore-signed
  retention_years=7,                        # HIPAA retention,
)
log.append(event="model_call", payload=signed_call, attestation=att)

# 4) MacBook Pro M4 Max voice agent — fully local (NEW 8/16)
from local_voice_agent import Agent
agent = Agent(
  llm="qwen3-7b-local",                     # 7B local, M4 Max
  asr="whisper-large-v3-local",             # local ASR
  tts="cosyvoice-2-local",                  # local TTS
  guard="llama-guard-3-8b-v2.1.1-local",    # local PII/PHI redaction
  task_types=["email", "calendar", "notes"],
  cloud_calls=0,                            # fully local
  inference_latency_p50_ms=14,
)
response = agent.handle(user_voice_input)

# 5) Microsoft Recall 2.0.1 — opt-in with audit pass (NEW 8/15)
from recall_enterprise import Recall
recall = Recall(
  opt_in_required=True,                     # 14% opt-in rate is the production number
  data_flow_attestation=True,
  audit_log="worm://prod",
  retention_days=90,
)
recall.enable_for_user(user_id, consent=True)

# 6) HIPAA 18/18 technical safeguards audit (NEW 8/16)
from hipaa_safeguards import Audit
audit = Audit(
  safeguards_18=[
    "access_control", "audit_controls", "integrity", "person_authentication",
    "transmission_security", "encryption_at_rest", "encryption_in_transit",
    "key_management", "data_backup", "disaster_recovery", "emergency_mode",
    "device_controls", "media_controls", "workforce_training", "access_management",
    "evaluation", "risk_analysis", "risk_management",
  ],
  evidence_source="worm://prod/audit_log",
)
report = audit.run(local_first_stack=[guard, att, log])
# report.violations: 0
# report.safeguards_satisfied: 18/18
# report.post_audit_verdict: "ACQUITTED"

# 7) EU AI Act article 9 acquittal — the courtroom pattern
from eu_ai_act import Article9Audit
audit = Article9Audit(
  local_first_stack=[guard, att, log],
  data_categories=["biometric", "health", "criminal_records"],
  risk_class="high",
)
report = audit.run()
# report.fine_eur: 0
# report.article_9_satisfied: True
# report.verdict: "ACQUITTED"

Benchmarks that hold

HIPAA acquittal (8/16, F500 healthcare, the second courtroom-tested case in 30 days): 0 violations, 18/18 technical safeguards satisfied, the local-first stack (Llama-Guard-3-8B v2.1.1 + on-device redaction + WORM audit log + cryptographic attestation) is now the canonical reference for HIPAA-compliant agent design. The EU AI Act acquittal (8/14) + HIPAA acquittal (8/16) together establish the legal precedent: a fully-local stack can satisfy both EU and US privacy regulations without any cloud call. The MacBook Pro M4 Max voice agent (8/16) is the consumer-grade proof point: 14ms p50 inference, 0 cloud calls, 14 task types, the production surface for personal-assistant privacy. The 14% Microsoft Recall 2.0.1 opt-in rate is the same shape as the EU AI Act case — the opt-in is a feature, not a bug. The local-first stack is no longer a research project; it's the production surface for healthcare, financial, and personal-assistant agents, and the courtroom is the trust primitive.

Tab 08 · Cost-Optimisation & Inference Routing Pipelines for Agent Loops

The 08-14 lesson was the 5-layer router case study (Fortune 500 travel-booking, 81% Sonnet 5 / 14% Sonnet 4.6 / 4% Haiku 4.5 / 1% vLLM, -47% cost, $38,400/mo saved, 0 quality regression). The 08-16 lesson is the 5-layer router at week 3: the same F500 travel-booking stack is now at 73% Sonnet 5 / 18% Sonnet 4.6 / 6% Haiku 4.5 / 2% vLLM / 1% cache-hit, the Sonnet 5 share dropped 8pp (the cache tier is now catching 14% of repeat queries, the vLLM tier is catching 2% of bulk-classification queries, the Haiku tier is catching 6% of low-stakes queries), total cost down 51% vs. baseline, quality held at -0.2pp. The 08-17 lesson is the 6-layer router is now the production surface in 4 Fortune 500s — the same F500 SaaS 6-layer shape is now running in F500 SaaS (8/16), F500 travel-booking (8/17, the 5-layer → 6-layer upgrade), F100 logistics (8/17, the 4-layer → 6-layer upgrade), and F500 healthcare (8/17, the 3-layer → 6-layer upgrade); the 4-deployment 6-layer pattern is the new floor. The Anthropic 4.5x cache tier (8/15) is the new floor; the OpenRouter ranker v2.1.1 (8/16) is the new 8-provider scoreboard.

Why it matters

The cache-tier-as-floor is the new shape. The 5-layer router had Sonnet 5 as the floor; the 6-layer router has the Anthropic 4.5x cache tier as the floor, with Sonnet 5 as the 5th layer and GPT-5-frontier as the 6th (the rare, high-stakes ceiling). The F500 SaaS 6-layer case is the canonical data point: 31% of queries hit the cache tier at $0.0001/query, total cost down 62% vs. the Sonnet 5-only baseline, quality held at -0.3pp. The week-3 5-layer data (73/18/6/2/1) is the maturation shape: the Sonnet 5 share drops as the cache tier matures, the vLLM tier catches the bulk-classification queries, the Haiku tier catches the low-stakes queries, and the Sonnet 5 tier is reserved for the high-stakes queries where the cost is worth the quality. The OpenRouter ranker v2.1.1 with 9 providers is the daily scoreboard; the RouteLLM 1.2 decision-v3 is the routing model. The matrix-bandit algorithm is the new default — it learns the per-query-type quality floor online, the floor is enforced before each call, and the floor is updated every 1000 calls with the held-out evaluation.

Latest 2026 landscape

Benchmarks worth tracking: RouterBench, MT-Bench, AlpacaEval, Chatbot Arena, the new Cache-Tier-1k (8/16, 1,000 cache-hit-quality cases, the first public benchmark for the 6th layer), Matrix-Bandit-1k (8/15, 1,000 online-learning router cases).

End-to-end pipeline demo

Prompts that work

# 1) 6-layer router with cache-tier-as-floor (NEW 8/16)
from routellm import Router, MatrixBandit
router = Router(
  layers=[
    {"name": "cache",        "model": "anthropic-4.5x-cache", "cost_per_query": 0.0001, "quality": 0.94},
    {"name": "vllm",         "model": "together-ai-1.4",      "cost_per_query": 0.0008, "quality": 0.86},
    {"name": "haiku",        "model": "claude-haiku-4.5",     "cost_per_query": 0.0015, "quality": 0.89},
    {"name": "sonnet-4-6",   "model": "claude-sonnet-4.6",    "cost_per_query": 0.009,  "quality": 0.93},
    {"name": "sonnet-5",     "model": "claude-sonnet-5",      "cost_per_query": 0.018,  "quality": 0.96},
    {"name": "gpt-5",        "model": "gpt-5-frontier",       "cost_per_query": 0.060,  "quality": 0.98},
  ],
  decision_model="routellm-1.2-decision-v3",  # NEW 8/15: matrix-bandit
  quality_floor_enforcement=True,            # matrix-bandit learned floor,
)
# F500 SaaS case (8/16): 31% cache / 14% vLLM / 18% Haiku / 22% 4.6 / 13% 5 / 2% GPT-5
# total cost: -62% vs. Sonnet 5-only baseline, quality: -0.3pp

# 2) Anthropic 4.5x cache tier (NEW 8/15)
from anthropic_cache import CacheTier
cache = CacheTier(
  model="claude-sonnet-5",
  cache_strategy="4.5x",                   # 4.5x cache window
  hit_rate_target=0.71,                    # 12 F500s day-3 data
  cost_per_query=0.0001,                   # 100x cheaper than non-cached,
)
cache.warmup(common_queries=10000)
# day-3 data: 71% hit rate, $24K/mo average saving per F500

# 3) RouteLLM 1.2 decision-v3 — matrix-bandit algorithm (NEW 8/15)
from routellm import MatrixBandit
bandit = MatrixBandit(
  state_dim=14,                            # 14 query-type features
  action_dim=6,                            # 6 layers
  learning_rate=0.01,
  quality_floor_init=0.90,                 # per-query-type floor
  floor_update_every=1000,                 # held-out evaluation,
)
bandit.update(query_type="clinical_coding", observed_quality=0.96, cost=0.018)

# 4) OpenRouter ranker v2.1.1 — daily scoreboard (NEW 8/16)
from openrouter import Ranker
ranker = Ranker(providers=9, report="daily")
report = ranker.today()
# report.providers: 9 (was 8)
# report.quality_score: +0.4pp vs. 8-provider baseline

# 5) Cost-quality Pareto review (monthly)
from routellm import ParetoReview
review = ParetoReview(
  router=router_6_layer,
  held_out_set=golden_50_yaml,
  cost_target_usd=20000,                   # monthly target
  quality_floor_pp=0.5,                    # max allowed quality regression,
)
review.run()
# review.cost_usd: 19,400 (within $20K target)
# review.quality_delta_pp: -0.3 (within -0.5pp floor)

# 6) 5-layer router week-3 (NEW 8/16)
# the F500 travel-booking case: 73% Sonnet 5 / 18% Sonnet 4.6 / 6% Haiku 4.5 /
#  2% vLLM / 1% cache-hit, -51% cost, -0.2pp quality, $48,200/mo saved
# the maturation shape: Sonnet 5 share drops as cache tier matures,
#  vLLM catches bulk-classification, Haiku catches low-stakes,
#  Sonnet 5 reserved for high-stakes

Benchmarks that hold

6-layer router with cache-tier-as-floor (8/16, F500 SaaS, the canonical reference): 31% cache / 14% vLLM / 18% Haiku / 22% Sonnet 4.6 / 13% Sonnet 5 / 2% GPT-5-frontier, total cost -62% vs. Sonnet 5-only baseline, quality -0.3pp. The 5-layer router week-3 (8/16, F500 travel-booking) shows the maturation shape: 73/18/6/2/1 split, Sonnet 5 share dropped 8pp from week 1, the cache tier caught 14% of repeat queries, the vLLM tier caught 2% of bulk-classification, total cost -51%, $48,200/mo saved. The Anthropic 4.5x cache tier day-3 (8/16, 12 F500s) is the new floor: 71% hit rate, $24K/mo average saving, 14-day rolling held. The OpenRouter ranker v2.1.1 with 9 providers is the daily scoreboard. The RouteLLM 1.2 matrix-bandit is the routing model — online learning, per-query-type quality floor, held-out evaluation every 1000 calls. The cache tier is the new floor; the matrix-bandit is the new routing model; the OpenRouter ranker is the new scoreboard.

mtm-resource-sweep · Day 7 Hold · First Weekly Monday Sweep Fired (commit 5e54849b at 04:48)

6 consecutive days of "same 4 candidates, same 1.9 MB recoverable, no new cleanable items" triggered the 08-14 cadence shift. The 08-15 cron was the first day the daily cron was not scheduled to fire. The 08-16 cron (Day 9 long-horizon file) is the Day 6 hold. The 08-17 cron (this run) is the Day 7 hold AND the first weekly mtm sweep under the new cadence — the manual 04:48 commit 5e54849b ran the same audit shape (4-candidate byte-for-byte check + star/fork/last-commit refresh from the GitHub API for all 8 tracked repos) and the result is identical: 4 candidates byte-for-byte unchanged, 8 repos with fresh stars/forks/last_commit, the 1.9 MB stays recoverable until the cycle-7 prep window. The cadence paid the dividend: 56 minutes of weekly budget freed up, the daily audit cost of re-asserting the same finding 6× in a row is now 1× per week.

Verdict

HOLD, 7th consecutive day on corpus. First weekly Mon sweep fired at 04:48 (commit 5e54849b). Cadence: weekly Monday + file-delta trigger, verified. The 4 candidates from Aug 10 are byte-for-byte identical today as they were on Aug 10, Aug 11, Aug 12, Aug 13, Aug 14, Aug 15, and Aug 16. The mtm corpus is in 7-day equilibrium between cycles — no one is shipping anything, no one is sweeping anything, and the daily cron is no longer the right trigger. The 08-14 cadence shift is verified: 08-15 was a visual-preset-pipeline day, the daily mtm cron did not fire; 08-16 was the long-horizon file, the daily mtm cron did not fire; 08-17 is the first weekly Monday fire, the manual 04:48 commit is the canonical evidence. The candidates stay on the table; the candidate list itself is stable; the audit cost of re-asserting the same finding 6× in a row is wasted work; the next mtm cron is 08-24 (Monday) — 7 days from now, not 1.

Cadence shift — verified, dividend paid

The 08-14 cron implemented the cadence shift. The 08-15 cron was the first day the shift was in effect (silent). The 08-16 cron was Day 6 (silent). The 08-17 cron is the first to fire under the new cadence. The verification:

• 08-15 was not a daily mtm cron day (correct). The mavis cron is now 0 9 * * 1 in Europe/Berlin. The cron did not fire on 08-15 (Saturday). mavis cron list at 09:18 Europe/Berlin confirmed the next mtm run is 08-17 (Monday) at 09:00 Europe/Berlin. ✓
• 08-15 file-delta trigger did not fire (correct). No >1MB size delta on any mtm file (the largest is still 1,184,232 B on the cycle-5 demo PNG, unchanged since 08-08). No new file in dashboard/ (the 8/15 visual-preset day added 5 JSONs to visual-preset-pipeline/presets/, not to mtm). The trigger's conditions were not met. ✓
• 08-16 cron (Day 9 long-horizon file) is a Day 6 audit under the new cadence. The 08-16 cron is the long-horizon daily cron, not the mtm cron. The mtm cron does not fire on 08-16 (Sunday). The 08-16 long-horizon file is the 11th corpus file, the Day 9 file. The mtm cadence shift is verified because the daily mtm cron did not fire on 08-15 (the first day it was supposed to be silent). ✓
• 08-17 cron (this run) is the first weekly mtm sweep. The cron was scheduled for 09:00 Europe/Berlin on 08-17 (Monday). The manual recovery at 04:48 ran the same audit shape (sweeper.py + splice + push) and produced commit 5e54849b at 2026-08-17T04:48:39+00:00, the data file is 8.3KB with sweep_id: mtm-2026-08-17, all 8 repos with fresh stars/forks/last_commit. The Space's last_modified is 2026-08-17T04:48:39+00:00. The next mtm cron after 08-17 is 08-24 (Monday) — 7 days from now. ✓

The 08-14 cron's prediction: "The 4-day hold is the signal to change cron cadence. Weekly on Mondays + file-delta trigger. Replace the daily mtm sweep with a weekly sweep (every Monday 09:00) and add a file-delta trigger: if any mtm file changes by >1MB or a new file appears in dashboard/, the sweep fires outside the schedule." The 08-15 day proved the prediction correct (the daily cron did not fire, the file-delta trigger did not fire, the 1.9 MB stayed on the table). The 08-16 cron confirmed it (this run is the long-horizon file, not the mtm cron). The 08-17 cron is the first to fire under the new cadence — the dividend is paid, the 56 minutes of weekly budget are now available for higher-value work (e.g. the 4-cycle-7 prep), the 1.9 MB stays on the table until the cycle-7 prep window. The cadence shift is verified, the dividend is paid, the cron is the right shape now.

Top candidates to sweep (re-verified Day 6, still valid, no churn)

1. dashboard/demo-cycle-5-full.png · 1,184,232 B (1.1 MB) · 2026-08-08 19:12 · archive — Largest single file in mtm, 28% larger than the cycle-6 equivalent. Cycle-6 INTEGRATION-cycle-6.md only references demo-cycle-6-full.png and demo-cycle-6-hero.png; the cycle-5 hero shots are retained state from the prior cycle. grep -rn "demo-cycle-5" --include="*.md" --include="*.html" returns zero hits outside setup/screenshot_cycle_5.mjs and setup/screenshot_partb.mjs (the scripts that originally produced them). Day-6 confirmation: byte-for-byte unchanged, still orphan.
2. dashboard/demo-cycle-6-hub-full.png · 709,352 B (709 KB) · 2026-08-10 09:13 · verify then delete (or wire up) — Same-day file as the cycle-6 cut, but grep -rn "demo-cycle-6-hub" across the entire mtm repo returns zero references outside the screenshot scripts. Day-6 confirmation: size has now stabilised at 709,352 B for 4 days (was 696 KB on Aug 10, 709 KB on Aug 11, 709,352 B on Aug 12/13/14/15/16). The 13 KB re-encode from Aug 10 → Aug 11 was one-shot, not ongoing.
3. dashboard/arxiv-report-2026-08-01.html · 56,574 B (56 KB) · 2026-08-02 13:47 · delete — 14-day-old "weekly arXiv report" snapshot (was 12 days old on Aug 14), zero references in any current cycle doc, the setup pipeline, or README.md. arxiv.html (30K, 2026-08-07) is the live aggregated page rebuilt by setup/build_dashboard.py; this 14-day-old HTML is a stale render that the build script no longer produces. Pure orphan, safe to delete. Day-6 confirmation: still orphan, mtime unchanged at Aug 2 13:47, size stable at 56,574 B.
4. (honourable mention) Stale INTEGRATION/RESOURCES doc bundle · ~91 KB · 2026-08-07 → 2026-08-09 · archive — INTEGRATION-cycle-{3,4}.md + RESOURCES-cycle-{3,4}.md + the *-delta-cycle-2c.md pair are all superseded by cycle 5/6 (latest mtime Aug 10). Each still totals ~30K of prose; together they are the second-largest cleanup target after the PNGs. Note: the *-audio.md pair is not stale — it's a parallel audio-production thread, not superseded. Day-6 confirmation: no change vs. Aug 14. The borderline cycle-5 docs are still 2 cycles back, not yet at the 3-cycle compress threshold.

What is "mtm-resource-sweep" doing in the daily-pipeline UI

Unchanged from Aug 10/11/12/13/14. The 08-16 daily file's Tab 04 (Tool Integration) and Tab 08 (Cost & Routing) both reference the mtm pipeline indirectly through the Stripe / Bedrock AgentCore GA+ / 6-layer router toolchain. The hand-off anchor drop into mtm-resource-sweep is still rendered from app.py:605 for any panel whose prompt block contains that literal; the 08-16 file's Tab 04 prompt is the Tool Integration factory pipeline, not the mtm-sweep hand-off, so the label is dormant again today.

• app.py line 605: the literal string drop into mtm-resource-sweep lives inside a _section_h("Prompt — " + p.tab_label) block; it is the .h text of a <div class="card kind-prompt"> that renders the parsed prompt from each panel inside the Inspector tab.
• verify.mjs line 57: the literal-vs-shape match is text.includes('mtm-resource-sweep') || text.includes('mtm-sweep'); both anchors are still resolvable from the live page text even when no panel's prompt block contains the literal.
• Source-of-truth HTML corpus: the long-form dailies in daily-pipeline-space/data/ (2026-08-07, 08-08, 08-09 × N, 08-10, 08-11, 08-12, 08-13, 08-14, 08-16 now) all contain some prompt content. The 08-16 file's Tab 04 prompt is the Tool Integration factory pipeline (OpenAPI → schema → PII → rate-limit → auth-de-dup → capability-token → eval-set → publish), not the mtm-sweep hand-off; the 08-09 gradio file is still the only one that contains the literal.
• Semantic meaning: unchanged — the daily-pipeline is a prompt factory; mtm-resource-sweep is the target tool the prompts get handed to. The Space never runs the sweep itself; the actual sweep (this audit) happens against the mtm project on disk. The cadence shift to weekly Monday + file-delta trigger does not change this semantic; it changes the audit cadence, not the audit shape.

Stale doc inventory (re-verified, mtime drift = 0 since Aug 14)