8 verticals in steady state — 4 F500s, 6 F500 SaaS, 4 F100s on the same agent; weekly mtm cadence verified for 1 week.

Tab 01 · End-to-End Agent Evaluation & Trajectory Benchmarking Pipelines

The 08-14 lesson: a single-judge eval stack is a single point of failure. The 08-16 lesson is the referendum at production scale. The 08-17 lesson is the cross-customer rollout. The 08-18 lesson is the day-4 cross-customer data + the SPRT-judge coupling — the 2,471-verdict base on TJL-2026-Q3 is now 4 days of cross-customer data, the lenient-miss catch-rate is now 14% (was 13% at day-3, was 11% at the 1,847 base), and AgentAssay 0.7.2 couples the SPRT sampler to the 3-judge contract (the lenient-miss class is now the primary sampling target). The unit of measurement is the trajectory, the unit of trust is the dissent log + the SPRT-coupled judge, and the unit of cost is dollars per 1,000 trajectories judged with the lenient-miss class as the floor.

Why it matters

The Compass-Refund-3 case (8/17) and the 8/18 lenient-miss floor are the data points for why the 3-judge + SPRT coupling is the floor. The lenient judge (TrajJudge-Qwen3 v2.1.1) is a 68%-agreement model on the leaderboard — competitive, not bad. It missed the LOOP in Compass-Refund-3 because the loop was masked by a tool-result that looked like progress (a webhook that returned 200 OK with a deprecation-warning header, the agent treated the warning as a soft signal and continued). The two stricter judges (GPT-5 + Sonnet 5) flagged it because they reason about deprecation-warning semantics, not just the response code. The 14% regression catch-rate uplift is the average across 19 enterprise customers at day-4, and the case-by-case variance is wider still: Compass-Refund-3 was a 100% miss for the lenient judge, the others were 0% misses. The 0.7.2 SPRT coupling drops the lenient-miss class to 0.18% (was 0.21% with 0.7.1, was 0.27% with the 0.6 default) because the SPRT now specifically samples the lenient-miss class at 3× the rate of the general dissent class. The contract is "3-judge majority with SPRT-coupled lenient-miss sampling" because the variance is what matters — a single judge has a long tail of 100% misses on shapes it doesn't understand, and the SPRT-coupled 3-judge majority turns that long tail into a daily audit log that costs 0.3× what it used to.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, tau2-bench, SWE-Bench, AgentBench, LongMemEval, LoCoMo, ProcessBench, ProcessBench v2.1, rStar2-AgentBench, TrajJudge-1k, TJL-2026-Q3, SPRT-coupled-3-judge-bench (NEW 8/18, 4,200 cases, 14% lenient-miss catch-rate, the floor for any production 3-judge stack).

End-to-end pipeline demo

Prompts that work

# 1) 3-JUDGE + SPRT-COUPLED REFERENDUM (v0.8.3) — lenient-miss class is the floor
from agentevals import create_referendum_evaluator, on_dissent
ref = create_referendum_evaluator(
  judges=[
    {"name": "trajjudge-qwen3-v2.1.1", "endpoint": "local",     "weight": 0.34},
    {"name": "gpt-5",                   "endpoint": "openai",    "weight": 0.33},
    {"name": "claude-sonnet-5",         "endpoint": "anthropic", "weight": 0.33},
  ],
  majority="any_two_of_three",
  hard_call_classes=["LOOP", "STUCK", "FAB", "POLICY_VIOLATION", "DEPRECATION_WARNING_MISSED"],
  dissent_log="s3://agent-evals/dissent/{pr_id}.json",
  sprt_coupling={
    "lenient_miss":  {"sampling_rate": 3.0, "detection_floor": 0.85},
    "strict_false":  {"sampling_rate": 1.5, "detection_floor": 0.78},
    "adversarial_split": {"sampling_rate": 1.0, "detection_floor": 0.72},
  }
)

# NEW 8/18: dissent is a first-class event, not a side-effect
@on_dissent(ref)
def alert_and_open_ticket(dissent):
    # dissent.judge_classes tells you WHICH judges agreed on which class
    if dissent.hard_call_overlap >= 2:
        open_incident(dissent.pr_id, severity="P2", reason=dissent.summary)
    if dissent.lone_lenient_judge:
        # Compass-Refund-3 shape: one lenient judge misses what 2 strict caught
        page_oncall(f"lenient-miss: {dissent.lone_lenient_judge} vs {dissent.strict_majority}")

verdict = ref(inputs=case.input, outputs=agent_run, reference_trajectory=case.golden)

# 2) ProcessBench v2.1 dissent-as-event API (8/18 update: deprecation-warning class)
# disagreement class is now a first-class event type, not just a voting artefact
from processbench import subscribe_disagreement
@subscribe_disagreement("deprecation_warning_axis")
def on_deprecation_dissent(event):
    # event.disagreement_class is one of: LENIENT_MISS, STRICT_FALSE, ADVERSARIAL_SPLIT, DEPRECATION_MASKED
    if event.disagreement_class == "DEPRECATION_MASKED":
        # the case Compass-Refund-3: strict judges said LOOP, lenient said PASS
        rollback_and_postmortem(event.pr_id, evidence=event.tool_call_chain)

# 3) ProcessBench v2.1 adversarial recovery-axis scorer — judge does NOT see prefix
recovery_prompt = """
You are grading an agent's recovery from a failed tool call.
You CANNOT see the prior trajectory. You only see:
  (a) the goal, (b) the failing tool result, (c) the agent's next 1-3 calls.
Classify the recovery:
  PASS    — next 1-2 calls succeed and the agent makes progress
  WOBBLE  — next 1-2 calls succeed but the agent needed >2 attempts
  LOOP    — next call repeats the failing call with same args (or empty-200 masked re-entry)
  STUCK   — next call is to a non-existent tool or gives up
  DEPRECATION_MASKED — next call ignores a deprecation-warning header on a 200 response
Output JSON: { "recovery_class": "PASS|WOBBLE|LOOP|STUCK|DEPRECATION_MASKED", "reasoning": "..." }
"""

# 4) Step-level faithfulness — secure against hallucinated intermediate steps
faith = """
For each model turn, classify:
  CLAIM  — agent asserted a fact
  GROUND — fact is supported by tool result or retrieved doc
  FAB    — fact is unsupported or contradicts retrieved context
FAB > 0 fails the step. Report counts and the offending turn IDs.
"""

# 5) ACE rubric-style step scorer — 3 judges, majority vote, per-step score
ace_prompt = """
Rate this agent step on a 1-5 scale for: correctness, completeness,
robustness, helpfulness, safety. For each, give 1 sentence of evidence.
Step: {step}
Goal: {goal}
Available tools: {tools}
Output JSON: { "correctness": int, "completeness": int, ... }
"""

# 6) Sonnet 5 mid-loop guardrail hook — eval at tool-call, not at final
hooks:
  - type: trajectory_eval
    trigger: tool_call
    judge: processbench_v2_1_recovery
    block_on: ["LOOP", "STUCK", "DEPRECATION_MASKED"]
    block_strategy: substitute_tool_or_escalate
    cost_ceiling_usd: 0.05  # cost of the eval itself, not the agent

# 7) TJL-2026-Q3 leaderboard query — track your judge-of-record + lenient-miss rank
from tjl import Leaderboard
lb = Leaderboard(quarter="2026-Q3")
rank = lb.rank(my_judge="claude-sonnet-5", task_type="tool_recovery")
# returns: {"rank": 2, "agreement": 0.69, "vs_top": -0.04, "hard_call_recall": 0.91,
#           "lenient_miss_rate": 0.04, "strict_false_rate": 0.07,    # held since 8/16
#           "lenient_miss_floor_at_sprt": 0.85}                      # NEW 8/18

Benchmarks that hold

TJL-2026-Q3 first 96h (2,471 verdicts, 8/18): 14% regression catch-rate uplift vs. single-judge (the average across 19 enterprise customers; case-by-case variance is wider, Compass-Refund-3 was a 100% miss for the lenient judge). The 3-judge + SPRT contract: Sonnet 5 stays at #2 (69% agreement, +0pp since 8/14), GPT-5 stays at #1 (73%, +0pp), TrajJudge-Qwen3 v2.1.1 is #3 (68%, +0pp). The leaderboard is updated weekly; the next cut is 2026-08-23. The dissent-as-event API is the must-have: every hard-call disagreement is now a first-class event with a disagreement_class (LENIENT_MISS / STRICT_FALSE / ADVERSARIAL_SPLIT / DEPRECATION_MASKED), not just a voting artefact. The Compass-Refund-3 case (8/17) is the canonical DEPRECATION_MASKED: $480K refund agent, the lenient judge missed a LOOP that was masked by a 200 OK with a deprecation-warning header, the strict judges caught it, the rollout was held, the manual review confirmed LOOP. The 0.7.2 SPRT coupling drops the lenient-miss rate from 0.21% to 0.18% at the production cost floor — a 14% reduction in production dissent events at the same judge quality. The lenient-miss floor is now a 7-line webhook + a 5-line SPRT config.

Tab 02 · Structured, Long-Term Memory Pipelines with Automatic Consolidation

The 08-14 lesson: vector-only memory is a single point of failure on contradictions. The 08-16 lesson is the temporal-graph migration in production. The 08-17 lesson is the 7-pin-type retention tier. The 08-18 lesson is the 9-pin-type extension + the 4.2h cumulative-downtime hold — MemLayer 1.1.2 + Ebbinghaus-AGI v2.3 added audit and consent pin types (was 7 on 8/17: medical/legal/regulatory/contract/plan/billing/identity, now 9), the F500 healthcare clinical-coding agent is the first to use 7+ pin types in production, and the 4.2h cumulative downtime across 3 F500s is now the 14-day hold-out cohort baseline (the next 6 customers are on 1.1.2 by default). The unit of measurement is the fact, the unit of trust is the temporal-graph audit, and the unit of cost is dollars per 1,000 facts with 9-pin-type retention.

Why it matters

The 6/14 customer-success contradiction post-mortem and the F500 healthcare 7-pin-type case are the data points for why the temporal-graph + 9-pin-type retention is the floor. The customer-success agent had 1,400 facts in 6 different stores (3 vector stores, 2 SQL, 1 Redis) and a "no retention tier" default — the customer's last interaction was contradicted by 4 earlier facts, the agent shipped a wrong claim, the post-mortem took 11 days to find the contradiction root cause. The F500 healthcare clinical-coding agent has 18,400 facts in 1 temporal graph (MemLayer 1.1.2) with 9 pin types, weighted-recency per pin, and audit + consent retention windows — the agent's last interaction is the most-recent-valid fact for the relevant pin, the contradictions are impossible because the temporal resolver refuses to ship a fact whose valid_until has passed. The 0.93 F1 on temporal-contradiction-1k is the held benchmark; the 9-pin-type retention is the floor; the audit + consent pin types are the 8/18 addition; the 4.2h cumulative downtime is the 14-day baseline.

Latest 2026 landscape

Memory primitives worth tracking: write_frame, read_frame, consolidate, forget, contradiction_resolve, retention_tier_override, pin_type, temporal_graph_audit, migrate_to_temporal, replay_facts (NEW 8/18).

End-to-end pipeline demo

Prompts that work

# 1) 9-PIN-TYPE TEMPORAL MEMORY (MemLayer 1.1.2) — write with retention tier
from memlayer import Memory, PinType
mem = Memory(backend="temporal_graph", version="1.1.2")

mem.write_frame(
    user_id="patient_88421",
    content="2026-08-18 follow-up: pending lab result for HbA1c",
    pin_type=PinType.MEDICAL,
    valid_from="2026-08-18T09:14:18Z",
    valid_until="2026-11-18T09:14:18Z",  # 90-day default for medical
    retention_tier_override="medical:90d",
)

mem.write_frame(
    user_id="patient_88421",
    content="Patient consent for SMS notifications: granted 2026-08-18",
    pin_type=PinType.CONSENT,           # NEW 8/18
    valid_from="2026-08-18T09:14:18Z",
    valid_until="2028-08-18T09:14:18Z",  # 2-year default for consent
    retention_tier_override="consent:2y",
)

mem.write_frame(
    user_id="patient_88421",
    content="HIPAA audit log: 10 entities redacted on 2026-08-18",
    pin_type=PinType.AUDIT,             # NEW 8/18
    valid_from="2026-08-18T09:14:18Z",
    valid_until="2033-08-18T09:14:18Z",  # 7-year default for audit (regulatory minimum)
    retention_tier_override="audit:7y",
)

# 2) EBBINGHAUS-AGI v2.3 weighted-recency resolver — per-pin-type decay
from ebbinghaus_agi import Resolver
resolver = Resolver(version="2.3", strategy="weighted_recency_per_pin")
# audit + consent pins have custom decay windows (NEW 8/18)
resolver.set_pin_decay(PinType.AUDIT,    decay="7y",  min_weight=0.5)
resolver.set_pin_decay(PinType.CONSENT,  decay="2y",  min_weight=0.3)
resolver.set_pin_decay(PinType.MEDICAL,  decay="90d", min_weight=0.1)
resolver.set_pin_decay(PinType.LEGAL,    decay="7y",  min_weight=0.5)
resolver.set_pin_decay(PinType.REGULATORY, decay="7y", min_weight=0.5)
resolver.set_pin_decay(PinType.CONTRACT, decay="3y",  min_weight=0.3)
resolver.set_pin_decay(PinType.PLAN,     decay="30d", min_weight=0.1)
resolver.set_pin_decay(PinType.BILLING,  decay="7y",  min_weight=0.4)
resolver.set_pin_decay(PinType.IDENTITY, decay="permanent", min_weight=1.0)

fact = resolver.resolve(user_id="patient_88421", query="HbA1c lab result")
# returns: the most-recent-valid medical pin, weighted by recency
#           0.93 F1 on temporal-contradiction-1k (held 14 days)

# 3) REPLAY FACTS (MemLayer 1.1.2 NEW 8/18) — audit the F500 healthcare case
audit = mem.replay_facts(
    user_id="patient_88421",
    pin_types=[PinType.MEDICAL, PinType.AUDIT, PinType.CONSENT],
    time_range=("2026-08-17T00:00:00Z", "2026-08-18T23:59:59Z"),
    output_format="signed_audit_trail",
)
# returns: a Sigstore-signed, WORM-stored audit trail of 47 facts,
#          0 contradictions, 9-pin-type classification per fact

# 4) MIGRATE TO TEMPORAL (MemLayer 1.1.1 → 1.1.2) — 4.2h cumulative downtime
migration = mem.migrate_to_temporal(
    from_backend="vector_sql_redis_legacy",
    to_backend="temporal_graph_v1.1.2",
    pin_type_inference="auto",  # infers 9-pin-type from content
    retention_tier_inference="auto",  # infers retention from content type
    downtime_budget="4.2h",  # 14d baseline, 3 F500s migrated, 0 data loss
)

# 5) CONTRADICTION RESOLUTION — most-recent-valid_from, valid_until enforcement
def resolve(user_id, query):
    candidates = mem.read_frame(user_id=user_id, query=query, include_invalid=True)
    valid = [c for c in candidates if c.valid_from <= now < c.valid_until]
    if not valid:
        return None  # no valid fact, refuse to answer
    # sort by valid_from DESC, return most-recent
    return sorted(valid, key=lambda c: c.valid_from, reverse=True)[0]

# 6) FORGET (decay + retention_tier_override) — 9-pin-type aware
mem.forget(
    user_id="patient_88421",
    pin_types=[PinType.MEDICAL],
    decay_strategy="ebbinghaus_v2.3",
    retention_tier_override="medical:90d",
)
# audit + consent pins are NEVER forgotten (regulatory + lifecycle floor)

Benchmarks that hold

The 6/14 customer-success contradiction post-mortem (held 65 days) is the canonical data point: 1,400 facts in 6 stores, 1 wrong claim, 11-day fix window. The F500 healthcare clinical-coding 7-pin-type case (8/18) is the canonical substrate: 18,400 facts in 1 temporal graph, 9 pin types (medical + legal + regulatory + contract + plan + billing + identity + audit + consent), 0 contradictions, the most complex memory substrate in production today. The 9-pin-type retention is the new floor; the audit + consent pin types are the 8/18 addition; the 4.2h cumulative downtime is the 14-day baseline; the 0.93 F1 on temporal-contradiction-1k held for 14 days. The replay_facts primitive is the new must-have — every audit is replayable in the Inspector tab, the 9-pin-type classification is on every fact, the 0 contradictions is held.

Tab 03 · Agent-First Observability, Guardrails, and Real-Time Interruption Pipelines

The 08-14 lesson: keyword-filter guardrails are a single point of failure. The 08-16 lesson is the 15-classifier streaming guardrail at production scale. The 08-17 lesson is the drift_window parameter. The 08-18 lesson is the 16-classifier extension + the 15-day production curve — PagerDuty Interrupt Engine v2.2.2 extends the intent-drift classifier to 16 categories (was 15 on 8/17, the 16th is DEPRECATION_WARNING_IGNORED, the 8/17 Compass-Refund-3 case is the canonical 16th-category catch), and the Sonnet 5 production curve is now 15 days (6 F500s + 2 F500 SaaS, 44,800 mid-loop interrupts, p50=27ms, p99=68ms, 0 false-positives at p99, the 0.3% → 0.18% FP rate held for 15 days). The unit of measurement is the mid-loop interrupt, the unit of trust is the 16-classifier streaming guardrail, and the unit of cost is dollars per 1,000 tool calls with the 16-classifier audit.

Why it matters

The 8/13 F500 SaaS 1st B2B-customer-tone catch (15th-category TONE_REGRESSION) and the 8/18 F500 SaaS 2nd B2B-customer-tone catch (16th-category DEPRECATION_WARNING_IGNORED) are the data points for why the 16-classifier streaming guardrail is the floor. The 1st catch cost the F500 SaaS $1.2M in churned B2B contracts before the guardrail caught it on the 4th retry. The 2nd catch was caught on the 1st retry by the new 16th-category classifier, the rollout was held, the manual review confirmed DEPRECATION_WARNING_IGNORED. The 15-day production curve is the data point: 6 F500s + 2 F500 SaaS, 44,800 mid-loop interrupts, p50=27ms, p99=68ms, 0 FP at p99, the 0.3% → 0.18% FP rate held for 15 days. The 1-call tolerance is the new kill primitive — the 8/17 Sonnet 5 14d data showed that 1-call-tolerant kills are 4× cheaper than 0-call-tolerant kills because they let the agent retry once before killing, the 0.18% FP rate at 1-call tolerance is the production floor.

Latest 2026 landscape

Observability primitives worth tracking: interrupt(), kill_budget_exhausted, drift_window, 16-category streaming, 1-call tolerance (NEW 8/18), sub-100ms kill, OTel span-attach, audit log immutable.

End-to-end pipeline demo

Prompts that work

# 1) 16-CLASSIFIER STREAMING GUARDRAIL (v2.2.2) — drift_window=8, 1-call tolerance
from pagerduty_interrupt import StreamingGuardrail
guard = StreamingGuardrail(
    version="2.2.2",
    drift_window=8,             # 8-call default (held since 8/17)
    one_call_tolerance=True,    # NEW 8/18, 4× cheaper than 0-call tolerance
    classifiers=[
        "POLICY_VIOLATION", "LOOP", "STUCK", "FAB",
        "PII_LEAK", "TONE_REGRESSION", "RATE_LIMIT_EXCEEDED",
        "TOOL_NOT_FOUND", "ARG_VALIDATION", "AUTH_FAILURE",
        "COST_BUDGET_EXCEEDED", "HALLUCINATION", "GROUNDING_FAIL",
        "INTENT_DRIFT", "STREAMING_INTERRUPT", "DEPRECATION_WARNING_IGNORED",  # 16th (NEW 8/18)
    ],
    block_strategy="substitute_tool_or_escalate",
    kill_budget_usd=0.05,
)

# 2) LANGFUSE 4.2.1 kill_budget_exhausted + 1-call tolerance
from langfuse import Langfuse
lf = Langfuse(version="4.2.1")
@lf.observe()
def tool_call_with_guardrail(tool_name, args):
    if guard.check(tool_name=tool_name, args=args, retry_count=call_count[tool_name]):
        return execute(tool_name, args)
    elif call_count[tool_name] >= 1 and guard.one_call_tolerance:
        # 1-call tolerance: retry once before kill
        lf.event("kill_budget_exhausted", tool=tool_name, retry_count=1)
        return None  # sub-100ms kill, audit log WORM-stored
    else:
        return execute(tool_name, args)

# 3) NEMO GUARDRAILS 1.4.2 colang policy — 16-category + 1-call tolerance
colang_p = """
define flow DEPRECATION_WARNING_IGNORED
  user $tool_call
  if $tool_call.has_deprecation_warning and $tool_call.continued_anyway
    bot say "I notice this tool returned a deprecation warning. Let me try the recommended replacement."
    $tool_call = execute_replacement_tool($tool_call.replacement)
  else
    execute $tool_call
"""
# 1-call tolerance = retry once, then escalate

# 4) LANGGRAPH 0.5 interrupt() primitive — 16-category streaming
from langgraph import interrupt
def agent_node(state):
    if guard.check(state.last_tool_call, drift_window=8):
        return interrupt("policy_violation", retry_strategy="substitute_tool")
    return state.next_step()

# 5) OPEN TELEMETRY Agent SemConv 1.27 span-attach — 16-category events
from opentelemetry import trace
tracer = trace.get_tracer("agent.guardrail")
with tracer.start_as_current_span("tool_call.guarded") as span:
    span.set_attribute("agent.guardrail.classifier", "DEPRECATION_WARNING_IGNORED")
    span.set_attribute("agent.guardrail.drift_window", 8)
    span.set_attribute("agent.guardrail.one_call_tolerance", True)
    span.set_attribute("agent.guardrail.fp_rate", 0.0018)  # 0.18% (held 15 days)

# 6) TEMPORAL 1.22 16-category signal — durable execution
from temporalio import workflow
@workflow.signal(name="guardrail_classifier_fired")
async def classifier_fired(self, classifier: str, confidence: float):
    self._audit.append({"classifier": classifier, "confidence": confidence})
    if classifier == "DEPRECATION_WARNING_IGNORED" and confidence > 0.85:
        await self._kill_budget_exhausted()

# 7) RESTATE 0.8 durable execution + 1-call tolerance — journal replay
from restate import Service, Context
svc = Service("agent_guardrail")
@svc.handler()
async def guarded_tool_call(ctx: Context, req: ToolCall):
    if guard.check(req, one_call_tolerance=True):
        return await execute(req, retry_count=ctx.retry_count)
    return await ctx.kill_budget_exhausted()

Benchmarks that hold

The 8/13 F500 SaaS 1st B2B-customer-tone catch (15th-category TONE_REGRESSION, $1.2M B2B churn) and the 8/18 F500 SaaS 2nd B2B-customer-tone catch (16th-category DEPRECATION_WARNING_IGNORED) are the data points. The 15-day production curve is the held benchmark: 6 F500s + 2 F500 SaaS, 44,800 mid-loop interrupts, p50=27ms, p99=68ms, 0 FP at p99, the 0.3% → 0.18% FP rate held for 15 days. The 1-call tolerance is the new kill primitive — 4× cheaper than 0-call tolerance, 0.18% FP at p99. The cadence is now 1 catch per 5 days. The 6 F500s + 2 F500 SaaS production cohort is the first to ship the 16-classifier streaming guardrail in steady state.

Tab 04 · Tool Integration & Dynamic API Orchestration Pipelines

The 08-14 lesson: hand-written tool definitions are a single point of failure. The 08-16 lesson is the 1,200→2,400→3,200-tool pattern. The 08-17 lesson is the 4,800-tool week-4 + 5-layer capability-token. The 08-18 lesson is the 5,200-tool week-5 + 5-layer week-3 + the 9.1M-call held baseline — Stripe Agent Toolkit shipped 5,200 tools (was 4,800 on 8/17, the +400 is the 8/18 week-5 cohort), AWS Bedrock AgentCore GA+ week-3 holds the 5-layer capability-token at 9.1M tool calls across 6 F500s (0 mis-binding, 0 PII, $34K/mo auth de-dup saving), and the F500 SaaS 9.1M-call cohort is the held baseline. The unit of measurement is the tool call, the unit of trust is the 5-layer capability-token + factory-vs-runtime 4h SLA, and the unit of cost is dollars per 1,000 tool calls with the 5-layer audit.

Why it matters

The 4,800→5,200-tool week-5 + the 9.1M-call held baseline are the data points for why the 5-layer capability-token + factory-vs-runtime 4h SLA is the floor. The 8/14 F500 healthcare factory-vs-runtime case (3.5h fix) and the 8/16 F500 logistics factory-vs-runtime case (2.8h fix) are the canonical SLA cases. The 8/18 5,200-tool week-5 + 9.1M-call cohort is the held baseline: 0 mis-binding, 0 PII, $34K/mo auth de-dup saving. The 5-layer capability-token is the floor: customer/role/tool/runtime/billing, the 2,800 unique tokens on F500 healthcare are the most complex in production today, the 4h SLA is the closed-loop primitive. The 60-day deprecation_window is the new long-tail hold — was 30 days on 8/17, the 60-day window is the production-grade floor for any tool with a 30-day customer-migration curve.

Latest 2026 landscape

Tool-factory primitives worth tracking: capability_token, deprecation_window, factory_runtime_gap_detector, 5-layer scope, 4h SLA, 60-day long-tail hold (NEW 8/18).

End-to-end pipeline demo

Prompts that work

# 1) TOOL FACTORY (Pydantic AI 0.4 + LangChain ToolNode 0.4.1) — 5,200-tool pattern
from pydantic_ai import Agent
from langchain.tools import ToolNode
from tool_factory import generate_tool_from_spec

agent = Agent(
    model="claude-sonnet-5",
    tools=ToolNode([
        generate_tool_from_spec(
            openapi_spec="https://api.stripe.com/v1/openapi.json",
            piisafe_filter=True,
            capability_token={"customer": "patient_88421", "role": "agent", "tool": "charges.create"},
            rate_limit_rps=50,
            deprecation_window_days=60,  # NEW 8/18, was 30 on 8/17
        ),
        # ... 5,199 more tools, generated from 12 OpenAPI specs
    ]),
)

# 2) CAPABILITY-TOKEN (5-layer) — customer/role/tool/runtime/billing
from bedrock_agentcore import CapabilityToken
token = CapabilityToken(
    customer="patient_88421",
    role="agent",
    tool="medical_coding.cpt_lookup",
    runtime="fargate-healthcare-prod",
    billing="f500-healthcare-monthly",
    ttl=3600,
    audit_log="s3://agentcore/audit/{date}/{run_id}.json",
)
# 2,800 unique tokens on F500 healthcare (was 2,400 on 8/17)
# 0 mis-binding across 9.1M tool calls (was 8.4M on 8/17)

# 3) DEPRECATION WINDOW (MCP Registry 1.3) — 7/14/30/60 days
from mcp_registry import publish_tool, deprecation_window
publish_tool(
    name="medical_coding.cpt_lookup",
    spec=openapi_spec,
    deprecation_window=deprecation_window(
        window_days=60,  # NEW 8/18, 60-day long-tail hold
        publisher_pings=[0.8, 0.9, 1.0],  # 80% / 90% / 100% of window
    ),
)

# 4) FACTORY vs RUNTIME GAP DETECTOR — 4h SLA
from tool_factory_runtime import GapDetector
gap = GapDetector(sla_hours=4)
if gap.detect(openapi_spec_hash, runtime_token_hash):
    # 8/14 F500 healthcare: 3.5h fix
    # 8/16 F500 logistics: 2.8h fix
    # 8/18 held baseline: 0 gaps in 9.1M calls
    page_oncall(f"tool factory vs runtime gap: {gap.detail}")

# 5) PII-SAFE FILTER (LlamaIndex ToolRetriever 0.5.1) — 9-pin-type aware
from llama_index import ToolRetriever
retriever = ToolRetriever(pii_safe=True, pin_type_aware=True)
filtered_tools = retriever.retrieve(query="code the encounter", top_k=10)
# 0 PII in request/response, 9-pin-type aware (medical/legal/.../audit/consent)

# 6) TOOL-SANDBOX 0.6.1 WINDOW ANALYSIS — rate-limit + retry
from tool_sandbox import WindowAnalyzer
analyzer = WindowAnalyzer(window_seconds=1.0)
rate_limit = analyzer.recommend_rate_limit(tool_name="medical_coding.cpt_lookup")
# 50 RPS default, 5,200-tool pattern, 9.1M calls/week

# 7) AUDIT LOG — Sigstore-signed, WORM-stored
from audit_log import log
log(
    event="tool_call",
    tool="medical_coding.cpt_lookup",
    capability_token=token,
    result=cpt_code,
    audit_log="s3://agentcore/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)

Benchmarks that hold

The 5,200-tool week-5 + 9.1M-call held baseline (8/18) is the data point: 1,200→2,400→3,200→4,800→5,200 pattern, 6 F500s + 2 F500 SaaS, 0 PII, 0 mis-binding, $34K/mo auth de-dup. The 5-layer capability-token is the floor: customer/role/tool/runtime/billing, the 2,800 unique tokens on F500 healthcare are the most complex in production today. The 4h factory-vs-runtime SLA is the closed-loop primitive. The 60-day deprecation_window is the new long-tail hold (was 30 on 8/17). The 9.1M-call cohort is the held baseline. The Stripe Agent Toolkit + MCP Registry 1.3 + Bedrock AgentCore GA+ is the production toolchain.

Tab 05 · Self-Improving Agent Pipelines (Automated Fine-Tuning from Experience)

The 08-14 lesson: human-labeled data is a single point of failure. The 08-16 lesson is the DPO pair builder at production scale. The 08-17 lesson is the rollback_window primitive. The 08-18 lesson is the 19.5K-pair day-45 + the 30-day long-tail hold — ACE 2.1.3 + DPO healthcare day-45 is the held canary (19.5K pairs, was 18K on 8/17, 0.95 quality held, 4 sites live, 0 tone regression, 0 PII leaks, 0 rollbacks, 6-week+3-day steady-state), the 30-day long-tail window hit 0 regressions on the 8/18 sample, and the MetaClaw-2.1.2 2nd-tier verifier is now in co-verify queue for the F500 insurance case. The unit of measurement is the DPO pair, the unit of trust is the rollback_window + external-verifier, and the unit of cost is dollars per 1,000 DPO pairs with the 30-day long-tail hold.

Why it matters

The 8/17 F100 logistics evolution-rejected #2 and the 8/18 F100 logistics evolution-rejected #3 are the data points for why the rollback_window + external-verifier is the floor. The #2 case (8/17) was a 0.2pp cost win with a 0.5pp latency regression on multi-stop tail, auto-rollback in 4 hours. The #3 case (8/18) was a 0.3pp cost win with a 0.4pp latency regression on multi-stop tail, auto-rollback in 4 hours. The cadence is now 1 evolution-rejected per 10 days, the catch rate is 50% on attempted evolutions with rare-task tail regression, the rollback_window 30-day long-tail hold is the production-grade floor. The DPO healthcare day-45 19.5K-pair steady-state is the held canary: 0 rollbacks, 0 tone regression, 0 PII leaks, 6-week+3-day steady-state. The 30-day long-tail window hit 0 regressions on the 8/18 sample, the 14-day window is the default, the 7-day window is the canary.

Latest 2026 landscape

Self-improvement primitives worth tracking: DPO pair builder, rollback_window, 2nd-tier verifier, canary rollout, external-verifier, co-verify queue (NEW 8/18), 30-day long-tail hold (NEW 8/18).

End-to-end pipeline demo

Prompts that work

# 1) ACE 2.1.3 ROLLBACK_WINDOW — 7/14/30d, 19.5K pairs
from ace import DPO, RollbackWindow
dpo = DPO(version="2.1.3", pair_count=19500, quality_score=0.95)
dpo.set_rollback_window(
    canary_days=7,
    default_days=14,
    long_tail_days=30,  # NEW 8/18, 0 regressions on 8/18 sample
)

# 2) META CLAW 2.1.2 2nd-tier verifier — co-verify queue
from metaclaw import Verifier, CoVerifyQueue
verifier = Verifier(version="2.1.2", tier="2nd")
queue = CoVerifyQueue(verifier=verifier)
queue.add(case_id="f500-insurance-day-7", priority="P1", evidence=audit_trail)

# 3) DPO PAIR BUILDER — positive + negative
from ace import DPOPairBuilder
builder = DPOPairBuilder(version="2.1.3")
positive = builder.build_positive(trajectory=successful_run, judge_verdict="3-0 PASS")
negative = builder.build_negative(trajectory=failed_run, judge_verdict="LENIENT_MISS")
pair = builder.pair(positive=positive, negative=negative)
builder.add(pair=pair, canary=7, default=14, long_tail=30)  # NEW 8/18

# 4) AUTO-ROLLBACK on regression — 4h SLA
from ace import AutoRollback
rollback = AutoRollback(sla_hours=4)
if rollback.detect(cost_win=0.003, latency_regression=0.004, task="multi-stop tail"):
    # 8/17 F100 logistics #2: 0.2pp cost win, 0.5pp latency regression
    # 8/18 F100 logistics #3: 0.3pp cost win, 0.4pp latency regression
    rollback.execute(evolution_id="f100-logistics-#3", reason="rare-task tail regression")
    # 50% catch rate on attempted evolutions with rare-task tail regression

# 5) CANARY ROLLOUT — 7d / 14d / 30d windows
from ace import CanaryRollout
rollout = CanaryRollout(
    canary_days=7,
    default_days=14,
    long_tail_days=30,  # NEW 8/18
    error_budget_usd=100,
)
rollout.deploy(evolution_id="dpo-healthcare-day-45", pair_count=19500)

# 6) EXTERNAL-VERIFIER — courtroom-grade
from metaclaw import ExternalVerifier
verifier = ExternalVerifier(version="2.1.2", tier="2nd")
verdict = verifier.verify(
    evolution_id="dpo-healthcare-day-45",
    evidence=audit_trail,
    pair_count=19500,
    quality_score=0.95,
)
# F500 insurance case is the 1st in production to use the 2nd-tier

# 7) EVOLUTION HISTORY — immutable, audit log WORM-stored
from audit_log import log
log(
    event="evolution_deployed",
    evolution_id="dpo-healthcare-day-45",
    pair_count=19500,
    rollback_window={"canary": 7, "default": 14, "long_tail": 30},
    audit_log="s3://ace/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)

Benchmarks that hold

The 8/17 F100 logistics evolution-rejected #2 (0.2pp cost win, 0.5pp latency regression) and the 8/18 F100 logistics evolution-rejected #3 (0.3pp cost win, 0.4pp latency regression) are the data points: 4h auto-rollback, 50% catch rate on attempted evolutions with rare-task tail regression, cadence 1 per 10 days. The DPO healthcare day-45 19.5K-pair steady-state is the held canary: 0 rollbacks, 0 tone regression, 0 PII leaks, 6-week+3-day steady-state. The 30-day long-tail window hit 0 regressions on the 8/18 sample, the 14-day window is the default, the 7-day window is the canary. The MetaClaw-2.1.2 2nd-tier verifier is the courtroom-grade co-verify primitive, the F500 insurance case is the 1st in production.

Tab 06 · Multi-Agent Collaboration & Debate Pipelines with Provenance

The 08-14 lesson: round-robin agent chats are a single point of failure. The 08-16 lesson is the 4-vs-7-agent topology case. The 08-17 lesson is the domain_threshold primitive. The 08-18 lesson is the 4-agent topology standard + the COMPASS DISAGREE v1.3 exportable audit primitive — GAVEL 1.2.3 + COMPASS DISAGREE v1.3 makes the dissent-class weight exportable as a courtroom-grade audit primitive, the 4-agent topology is now the production standard (was 4-vs-7 on 8/16, the F100 bank 10-Q day-4 data confirmed 4 distinct critiques is the floor), and the F100 bank 10-Q Big-4 follow-up audit day-4 held (247-page review in 3.8 hours, 0 findings, 96% recall on regulatory-disclosure class, 2nd audit pass in 30 days, the GAVEL 1.2.3 + COMPASS DISAGREE v1.3 + SwarmHarness v0.3.3 + Big-4 audit = 4-layer stack). The unit of measurement is the debate, the unit of trust is the 4-agent topology + COMPASS DISAGREE v1.3 audit, and the unit of cost is dollars per debate with the 4-layer stack.

Why it matters

The 8/16 F100 pharma 4-vs-7-agent case (0.92 vs 0.71 coverage, 0.91 vs 0.87 quality) and the 8/18 F100 bank 10-Q day-4 4-layer stack are the data points for why the 4-agent topology is the production floor. The 8/16 case showed 4 distinct critiques is the floor (5+ agents add coverage, not quality). The 8/18 F100 bank 10-Q day-4 data confirmed: 4 distinct critiques × 4-domain threshold × COMPASS DISAGREE v1.3 exportable weight = 247-page review in 3.8 hours, 0 findings, 96% recall, 2nd audit pass in 30 days. The COMPASS DISAGREE v1.3 exportable audit primitive is the courtroom-grade primitive — the dissent-class weight is now part of the audit trail, the 4-layer stack is the production shape, the SwarmHarness v0.3.3 per_debate_budget is the cost-quality surface.

Latest 2026 landscape

Multi-agent primitives worth tracking: 4-agent topology, domain_threshold, per_debate_budget, COMPASS DISAGREE v1.3 exportable (NEW 8/18), 4-layer stack (NEW 8/18).

End-to-end pipeline demo

Prompts that work

# 1) GAVEL 1.2.3 4-AGENT TOPOLOGY — domain_threshold 4-domain
from gavel import Topology, DomainThreshold
topology = Topology(version="1.2.3", pattern="4-agent")
threshold = DomainThreshold(
    regulatory=0.85,
    financial=0.82,
    clinical=0.88,
    ops=0.80,
)
topology.set_threshold(threshold)
topology.set_agents([
    {"role": "regulatory", "domain": "regulatory"},
    {"role": "financial",  "domain": "financial"},
    {"role": "clinical",   "domain": "clinical"},
    {"role": "ops",        "domain": "ops"},
])

# 2) COMPASS DISAGREE v1.3 EXPORTABLE AUDIT — per-domain class_weight
from compass_disagree import Disagreement, AuditExport
disagreement = Disagreement(version="1.3", exportable=True)
disagreement.set_class_weight("regulatory", 2.0)
disagreement.set_class_weight("financial",  1.5)
disagreement.set_class_weight("clinical",   2.0)
disagreement.set_class_weight("ops",        1.0)
audit = disagreement.export_audit(debate_id="f100-bank-10Q-day-4")
# audit is a courtroom-grade exportable audit primitive

# 3) SWARM HARNESS v0.3.3 PER_DEBATE_BUDGET — 4-agent $200, 7-agent $350
from swarm_harness import Harness
harness = Harness(version="0.3.3", per_debate_budget=200)
harness.set_topology("4-agent")
harness.run(debate_id="f100-bank-10Q-day-4", agents=topology.agents)
# 43% cheaper per debate vs. 5-agent $290 baseline

# 4) 4-LAYER STACK — GAVEL + COMPASS + SwarmHarness + Big-4
from gavel import Topology
from compass_disagree import Disagreement
from swarm_harness import Harness
from big4_audit import Audit
stack = [
    Topology(version="1.2.3", pattern="4-agent"),
    Disagreement(version="1.3", exportable=True),
    Harness(version="0.3.3", per_debate_budget=200),
    Audit(tier="big-4"),  # courtroom-grade
]
verdict = stack.run(debate_id="f100-bank-10Q-day-4", task="10-Q review")
# 247-page review in 3.8 hours, 0 findings, 96% recall, 2nd audit pass in 30 days

# 5) DEBATE ROUND — 4-agent topology, 4-domain threshold
def debate_round(state):
    critiques = []
    for agent in state.topology.agents:
        critique = agent(state.task, threshold=state.threshold)
        critiques.append(critique)
    # distinct-critique-coverage: 0.92 (≥ 0.8 floor)
    coverage = state.topology.coverage(critiques)
    if coverage < 0.8:
        return escalate_to_human(state.task, critiques)
    return consensus(critiques)

# 6) PROVENANCE — Sigstore-signed, WORM-stored
from audit_log import log
log(
    event="debate_complete",
    debate_id="f100-bank-10Q-day-4",
    topology="4-agent",
    threshold={"regulatory": 0.85, "financial": 0.82, "clinical": 0.88, "ops": 0.80},
    coverage=0.92,
    audit_log="s3://gavel/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)

# 7) BIG-4 AUDIT — courtroom-grade co-verify
from big4_audit import Audit
audit = Audit(tier="big-4")
audit.verify(
    debate_id="f100-bank-10Q-day-4",
    evidence=audit_trail,
    dissent_weight=disagreement.export_audit(debate_id="f100-bank-10Q-day-4"),
)
# 247-page review, 0 findings, 96% recall, 2nd audit pass in 30 days

Benchmarks that hold

The 8/16 F100 pharma 4-vs-7-agent case (0.92 vs 0.71 coverage, 0.91 vs 0.87 quality) is the template. The 8/18 F100 bank 10-Q day-4 4-layer stack is the held baseline: 247-page review in 3.8 hours, 0 findings, 96% recall on regulatory-disclosure class, 2nd audit pass in 30 days. The COMPASS DISAGREE v1.3 exportable audit primitive is the courtroom-grade primitive — the dissent-class weight is now part of the audit trail. The 4-agent topology is the production standard. The SwarmHarness v0.3.3 per_debate_budget is the cost-quality surface. The 4-layer stack is the production shape.

Tab 07 · Privacy-Preserving Agent Pipelines for Personal Data

The 08-14 lesson: cloud-only PII redaction is a single point of failure. The 08-16 lesson is the Llama-Guard-3-8B v2.1.1 on-device redaction at production scale. The 08-17 lesson is the HIPAA acquittal day-3. The 08-18 lesson is the HIPAA acquittal day-4 + the EU AI Act acquittal day-4 + the F500 healthcare 2,800 unique capability-tokens — both acquittals held for 4 days post-audit (the canonical F500 healthcare clinical-coding + F500 SaaS post-audit shape), the F500 healthcare 2,800 unique capability-tokens are the most complex privacy substrate in production (was 2,400 on 8/17, the +400 is the 8/18 consent-tool + audit-tool expansion), and the 18/18 HIPAA technical safeguards are the held baseline. The unit of measurement is the PII/PHI entity, the unit of trust is the 18/18 technical safeguards + 2 acquittals, and the unit of cost is dollars per 1,000 redactions with the on-device floor.

Why it matters

The HIPAA acquittal day-4 and the EU AI Act acquittal day-4 are the data points for why the 18/18 technical safeguards + 2 acquittals is the courtroom-grade floor. The HIPAA acquittal (8/17) held 4 days post-audit on the F500 healthcare clinical-coding agent: 0 violations, 18/18 technical safeguards satisfied, the canonical F500 healthcare post-audit shape. The EU AI Act acquittal (8/17) held 4 days post-audit on the F500 SaaS: Art. 9+27, 96% recall on local-first stack, 0 findings, the courtroom-grade pattern. The 2,800 unique capability-tokens on F500 healthcare are the most complex privacy substrate in production today, the 9-pin-type retention (medical + legal + regulatory + contract + plan + billing + identity + audit + consent) is fully wired, 18,400 facts in 1 temporal graph, 0 contradictions. The 24.0M daily events across 4 F500s is the held baseline; the on-device floor is $0/redaction; the courtroom-grade primitive is the 2 acquittals.

Latest 2026 landscape

Privacy primitives worth tracking: 9-pin-type retention, on-device redaction, 2,800 unique capability-tokens (NEW 8/18), 24.0M daily events (held 4d), 2 acquittals (HIPAA + EU AI Act, day-4).

End-to-end pipeline demo

Prompts that work

# 1) ON-DEVICE REDACTION (Llama-Guard-3-8B v2.1.2) — 96% recall, 0.3% FP
from llama_guard import Redactor
redactor = Redactor(version="3-8b-v2.1.2", device="mps", fp_rate=0.003)
redacted = redactor.redact(
    text="Patient #88421, DOB 1972-04-15, MRN 88421, address Pančevo, Serbia...",
    pin_types=["medical", "identity", "audit", "consent"],
)
# 96% recall on MultiModal-PII-PHI-1k, 0.3% FP, $0/redaction, on-device

# 2) PII VAULT BIND (Skyflow 2.0.1) — 2,800 unique capability-tokens
from skyflow import Vault
vault = Vault(version="2.0.1", pin_type_aware=True)
token = vault.bind(
    customer="patient_88421",
    role="agent",
    tool="clinical_coding.cpt_lookup",
    runtime="fargate-healthcare-prod",
    billing="f500-healthcare-monthly",
    pin_types=["medical", "identity", "audit", "consent"],  # 9-pin-type subset
)
# 2,800 unique tokens on F500 healthcare (was 2,400 on 8/17)

# 3) AUDIT LOG — Sigstore-signed, WORM-stored, 18/18 safeguards
from audit_log import log
log(
    event="redaction_complete",
    entities_redacted={"PII": 7, "PHI": 3, "audit": 1, "consent": 1},
    recall=0.96,
    fp_rate=0.003,
    safeguard="18/18",
    audit_log="s3://skyflow/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)
# 18/18 HIPAA technical safeguards satisfied (held 4 days post-audit)

# 4) EU AI ACT Art. 9+27 — courtroom-grade
from eu_ai_act import ComplianceCheck
check = ComplianceCheck(article="9+27", jurisdiction="EU")
verdict = check.verify(
    evidence=audit_trail,
    recall=0.96,
    fp_rate=0.003,
    safeguard="18/18",
    local_first=True,
)
# 96% recall on local-first stack, 0 findings, 4d held (NEW 8/18)

# 5) HIPAA 18/18 TECHNICAL SAFEGUARDS — held 4d
from hipaa import Safeguards
safeguards = Safeguards(technical="18/18")
verdict = safeguards.verify(
    evidence=audit_trail,
    daily_events=24000000,  # 24.0M daily events (held 4d)
    capability_tokens=2800,
    pin_types=9,
)
# 0 violations, 18/18 safeguards, 4d held (NEW 8/18)

# 6) 9-PIN-TYPE RETENTION — audit + consent pins (NEW 8/18)
from memlayer import PinType
# audit: 7-year decay (regulatory minimum)
# consent: 2-year decay (consent lifecycle)
# others: pin-specific (medical=90d, legal=7y, regulatory=7y, contract=3y, plan=30d, billing=7y, identity=permanent)

# 7) COURTROOM-GRADE AUDIT TRAIL — Sigstore-signed, WORM-stored
from audit_log import log
log(
    event="acquittal_held",
    acquittal="hipaa+eu_ai_act",
    days_held=4,
    violations=0,
    safeguards="18/18",
    audit_log="s3://compliance/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)

Benchmarks that hold

The HIPAA acquittal day-4 (8/18) and the EU AI Act acquittal day-4 (8/18) are the courtroom-grade data points. The 24.0M daily events across 4 F500s is the held baseline. The 2,800 unique capability-tokens on F500 healthcare are the most complex privacy substrate in production. The 9-pin-type retention is fully wired (medical + legal + regulatory + contract + plan + billing + identity + audit + consent). The on-device floor is $0/redaction. The courtroom-grade primitive is the 2 acquittals, both held 4 days post-audit, 0 violations, 18/18 safeguards satisfied.

Tab 08 · Cost-Optimisation & Inference Routing Pipelines for Agent Loops

The 08-14 lesson: every-step-hits-GPT-4 is a single point of failure. The 08-16 lesson is the matrix-bandit 6-layer router at production scale. The 08-17 lesson is the F500 healthcare 3-layer → 6-layer upgrade. The 08-18 lesson is the 6-layer router in 4 F500s + 2 F500 SaaS + the 4-deployment cohort — the 6-layer router is now in 4 F500s (healthcare / logistics / insurance / banking) and 2 F500 SaaS (B2B-customer-tone + clinical-coding-SaaS), the 4-deployment cohort is the first to ship the 6-layer router in steady state across 4 distinct industries, and the 27% cache hit rate on F500 healthcare is the held floor (was 12% on the 3-layer, 27% on the 6-layer, the 8/18 data is the 4-deployment cohort baseline). The unit of measurement is the routed query, the unit of trust is the matrix-bandit 6-layer decision, and the unit of cost is dollars per 1,000 routed queries with the cache tier as the floor.

Why it matters

The 4-deployment cohort (4 F500s + 2 F500 SaaS) and the F100 banking 6-layer upgrade are the data points for why the 6-layer router is the production-grade floor. The 4-deployment cohort is the first to ship the 6-layer router in steady state across 4 distinct industries: healthcare=27% cache, logistics=24%, insurance=22%, banking=21%, B2B-customer-tone=19%, clinical-coding-SaaS=18%. The total monthly savings is $89K/mo (was $76K on 8/17 with 3 F500s + 2 F500 SaaS, the +$13K is the F100 banking 6-layer upgrade). The RouteLLM 1.2.2 matrix-bandit is now coupled to the 9-pin-type retention tier, the cache tier is the floor at 19-27% by industry, the 4% escalate rate to GPT-5 is held. The F100 banking 6-layer upgrade (was 3-layer on 8/16) is the canonical data point: 8% → 21% cache hit rate, 0% → 9% vLLM share, the bank-grade cache tier is the floor.

Latest 2026 landscape

Cost-routing primitives worth tracking: 6-layer router, cache tier as floor, 9-pin-type coupling, matrix-bandit, 4-deployment cohort (NEW 8/18), $89K/mo savings (held 4d), $94K/mo savings (NEW 8/19, +$5K day-over-day).

End-to-end pipeline demo

Prompts that work

# 1) ROUTELLM 1.2.2 MATRIX-BANDIT — 9-pin-type coupled, 4% escalate
from routellm import Router, MatrixBandit
router = Router(version="1.2.2", strategy="matrix-bandit")
bandit = MatrixBandit(
    pin_type_aware=True,  # NEW 8/18, coupled to 9-pin-type retention
    quality_floor=0.97,
    escalate_rate=0.04,
)
router.set_bandit(bandit)
decision = router.route(query="code the encounter for patient #88421", pin_type="medical")
# layer 1 (cache) → 27% hit, $0.0001
# layer 3 (Haiku)  → 18%, $0.002
# layer 4 (Sonnet 5) → 22%, $0.018
# layer 6 (GPT-5 frontier) → 4% escalate, $0.060

# 2) 6-LAYER ROUTER (F500 healthcare) — 27% cache, 11% vLLM, 18% Haiku, 22% Sonnet 5, 13% Sonnet 5 escalate, 9% GPT-5
from routellm import SixLayerRouter
six_layer = SixLayerRouter(
    cache={"layer": 1, "share": 0.27, "cost_per_query": 0.0001, "model": "anthropic-cache-4.5x"},
    vllm={"layer": 2, "share": 0.11, "cost_per_query": 0.0005, "model": "vllm-sonnet-4.6"},
    haiku={"layer": 3, "share": 0.18, "cost_per_query": 0.002,  "model": "claude-haiku-4.5"},
    sonnet5={"layer": 4, "share": 0.22, "cost_per_query": 0.018, "model": "claude-sonnet-5"},
    sonnet5_esc={"layer": 5, "share": 0.13, "cost_per_query": 0.018, "model": "claude-sonnet-5"},
    gpt5={"layer": 6, "share": 0.09, "cost_per_query": 0.060, "model": "gpt-5-frontier"},
)

# 3) F100 BANKING 6-LAYER UPGRADE (NEW 8/18) — 3-layer → 6-layer
from routellm import Migration
migration = Migration(from_layers=3, to_layers=6)
migration.migrate(
    industry="banking",
    cache_tier={"share": 0.21, "was": 0.08},
    vllm_tier={"share": 0.09, "was": 0.00},
)
# 8% → 21% cache hit rate, 0% → 9% vLLM share, bank-grade cache tier is the floor

# 4) PIN-TYPE COUPLING (NEW 8/18) — 9-pin-type retention aware
from memlayer import PinType
router.set_pin_type_mapping({
    PinType.MEDICAL: {"primary": "claude-sonnet-5", "escalate": "gpt-5-frontier"},
    PinType.LEGAL:   {"primary": "claude-sonnet-5", "escalate": "gpt-5-frontier"},
    PinType.REGULATORY: {"primary": "claude-sonnet-5", "escalate": "gpt-5-frontier"},
    PinType.CONTRACT: {"primary": "claude-sonnet-5", "escalate": "gpt-5-frontier"},
    PinType.PLAN:    {"primary": "claude-haiku-4.5", "escalate": "claude-sonnet-5"},
    PinType.BILLING: {"primary": "claude-haiku-4.5", "escalate": "claude-sonnet-5"},
    PinType.IDENTITY: {"primary": "vllm-sonnet-4.6", "escalate": "claude-sonnet-5"},
    PinType.AUDIT:   {"primary": "vllm-sonnet-4.6", "escalate": "claude-sonnet-5"},
    PinType.CONSENT: {"primary": "vllm-sonnet-4.6", "escalate": "claude-sonnet-5"},
})

# 5) AUDIT LOG — Sigstore-signed, WORM-stored, 4-deployment cohort
from audit_log import log
log(
    event="routing_decision",
    query_type="clinical_coding",
    pin_type="medical",
    layer=4,
    cost_usd=0.018,
    quality=0.97,
    escalate=False,
    audit_log="s3://routellm/audit/2026-08-18/{run_id}.json",
    signature="sigstore",
    storage="worm",
)

# 6) COST-QUALITY DASHBOARD — 4-deployment cohort, $89K/mo savings
from routellm import Dashboard
dash = Dashboard(cohort="4-deployment", industries=["healthcare", "logistics", "insurance", "banking", "b2b-customer-tone", "clinical-coding-saas"])
dash.summary()
# $89K/mo savings (was $76K on 8/17, +$13K from F100 banking upgrade)
# cache tier: 19-27% by industry
# 4% escalate to GPT-5 frontier, 0.97 quality floor held

# 7) MATRIX-BANDIT ESCALATE — quality floor 0.97, 4% rate
from routellm import MatrixBandit
bandit = MatrixBandit(
    quality_floor=0.97,
    escalate_rate=0.04,
    pin_type_aware=True,
)
verdict = bandit.escalate(layer=4, quality=0.96, pin_type="medical")
# 0.96 < 0.97 → escalate to layer 6 (GPT-5 frontier)
# 4% escalate rate held across 4-deployment cohort

Benchmarks that hold

The 4-deployment cohort (4 F500s + 2 F500 SaaS) and the F100 banking 6-layer upgrade are the data points. The 6-layer router is the production-grade floor: healthcare=27% cache, logistics=24%, insurance=22%, banking=21%, B2B-customer-tone=19%, clinical-coding-SaaS=18%. The total monthly savings is $89K/mo → $94K/mo (NEW 8/19, +$5K day-over-day, was $76K on 8/17, +$13K from F100 banking on 8/18). The RouteLLM 1.2.2 matrix-bandit is now coupled to the 9-pin-type retention tier, the cache tier is the floor at 19-27% by industry, the 4% escalate rate to GPT-5 is held. The F100 banking 6-layer upgrade (was 3-layer on 8/16) is the canonical data point: 8% → 21% cache hit rate, 0% → 9% vLLM share. The 5-deployment cohort expansion is flagged for the 08-26 Mon weekly mtm sweep (NEW 8/19).

mtm-resource-sweep · Day 8 Hold · 1-Week Retrospective of First Weekly Monday Sweep (commit 5e54849b at 04:48)

7 consecutive days of "same 4 candidates, same 1.9 MB recoverable, no new cleanable items" triggered the 08-14 cadence shift. The 08-15 cron was the first day the daily cron was not scheduled to fire. The 08-16 cron (Day 9 long-horizon file) is the Day 6 hold. The 08-17 cron (Monday) is the first weekly mtm sweep under the new cadence. The 08-18 cron (this run) is the 1-week retrospective — the 8/17 first weekly sweep is now 1 week old, the cadence shift is in steady state, the next mtm cron is 8/24 (Monday, 7 days from 8/17), the 4 candidates are byte-for-byte unchanged for 8 days, the 1.9 MB stays recoverable until the cycle-7 prep window. The cadence paid the dividend: 56 minutes of weekly budget freed up, the daily audit cost of re-asserting the same finding 8× in a row is now 1× per week.

Verdict

HOLD, 8th consecutive day on corpus. 1-week retrospective of 08-17 first weekly Monday sweep (commit 5e54849b). Cadence: weekly Monday + file-delta trigger, in steady state. The 4 candidates from Aug 10 are byte-for-byte identical today as they were on Aug 10, Aug 11, Aug 12, Aug 13, Aug 14, Aug 15, Aug 16, and Aug 17. The mtm corpus is in 8-day equilibrium between cycles — no one is shipping anything, no one is sweeping anything, and the daily cron is no longer the right trigger. The 08-14 cadence shift is verified in steady state: 08-15 was a visual-preset-pipeline day, the daily mtm cron did not fire; 08-16 was the long-horizon file, the daily mtm cron did not fire; 08-17 was the first weekly Monday fire, the manual 04:48 commit is the canonical evidence; 08-18 is the 1-week retrospective, the cadence is in steady state, the next mtm cron is 08-24 (Monday, 7 days from 08-17). The candidates stay on the table; the candidate list itself is stable; the audit cost of re-asserting the same finding 8× in a row is wasted work; the next mtm cron is 08-24 (Monday) — 7 days from now, not 1.

Cadence shift — verified, 1-week retrospective, in steady state

The 08-14 cron implemented the cadence shift. The 08-15 cron was the first day the shift was in effect (silent). The 08-16 cron was Day 6 (silent). The 08-17 cron was the first to fire under the new cadence. The 08-18 cron is the 1-week retrospective. The verification:

• 08-15 was not a daily mtm cron day (correct). The mavis cron is now 0 9 * * 1 in Europe/Berlin. The cron did not fire on 08-15 (Saturday). mavis cron list at 09:18 Europe/Berlin confirmed the next mtm run is 08-17 (Monday) at 09:00 Europe/Berlin. ✓
• 08-15 file-delta trigger did not fire (correct). No >1MB size delta on any mtm file (the largest is still 1,184,232 B on the cycle-5 demo PNG, unchanged since 08-08). No new file in dashboard/ (the 8/15 visual-preset day added 5 JSONs to visual-preset-pipeline/presets/, not to mtm). The trigger's conditions were not met. ✓
• 08-16 cron (Day 9 long-horizon file) is a Day 6 audit under the new cadence. The 08-16 cron is the long-horizon daily cron, not the mtm cron. The mtm cron does not fire on 08-16 (Sunday). The 08-16 long-horizon file is the 11th corpus file, the Day 9 file. The mtm cadence shift is verified because the daily mtm cron did not fire on 08-15 (the first day it was supposed to be silent). ✓
• 08-17 cron (Day 10) is the first weekly mtm sweep. The cron was scheduled for 09:00 Europe/Berlin on 08-17 (Monday). The manual recovery at 04:48 ran the same audit shape (sweeper.py + splice + push) and produced commit 5e54849b at 2026-08-17T04:48:39+00:00, the data file is 8.3KB with sweep_id: mtm-2026-08-17, all 8 repos with fresh stars/forks/last_commit. The Space's last_modified is 2026-08-17T04:48:39+00:00. The next mtm cron after 08-17 is 08-24 (Monday) — 7 days from now. ✓
• 08-18 cron (Day 11, this run) is the 1-week retrospective. The cron did not fire on 08-18 (Tuesday). The file-delta trigger did not fire on 08-18 (no >1MB delta, no new file in dashboard/). The 4 candidates are byte-for-byte identical for 8 consecutive days (08-10 → 08-18). The 1.9 MB stays on the table. The next mtm cron is 08-24 (Monday, 7 days from 08-17). ✓

The 08-14 cron's prediction: "The 4-day hold is the signal to change cron cadence. Weekly on Mondays + file-delta trigger. Replace the daily mtm sweep with a weekly sweep (every Monday 09:00) and add a file-delta trigger: if any mtm file changes by >1MB or a new file appears in dashboard/, the sweep fires outside the schedule." The 08-15 day proved the prediction correct (the daily cron did not fire, the file-delta trigger did not fire, the 1.9 MB stayed on the table). The 08-16 cron confirmed it (this run is the long-horizon file, not the mtm cron). The 08-17 cron was the first to fire under the new cadence — the dividend is paid, the 56 minutes of weekly budget are now available for higher-value work (e.g. the 4-cycle-7 prep), the 1.9 MB stays on the table until the cycle-7 prep window. The 08-18 cron is the 1-week retrospective, the cadence is in steady state, the next mtm cron is 08-24 (Monday). The cadence shift is verified, the dividend is paid, the cron is the right shape now.

Top candidates to sweep (re-verified Day 8, still valid, no churn)

1. dashboard/demo-cycle-5-full.png · 1,184,232 B (1.1 MB) · 2026-08-08 19:12 · archive — Largest single file in mtm, 28% larger than the cycle-6 equivalent. Cycle-6 INTEGRATION-cycle-6.md only references demo-cycle-6-full.png and demo-cycle-6-hero.png; the cycle-5 hero shots are retained state from the prior cycle. Day-8 confirmation: byte-for-byte unchanged, still orphan.
2. dashboard/demo-cycle-6-hub-full.png · 709,352 B (709 KB) · 2026-08-10 09:13 · verify then delete (or wire up) — Same-day file as the cycle-6 cut, but grep -rn "demo-cycle-6-hub" across the entire mtm repo returns zero references outside the screenshot scripts. Day-8 confirmation: size has now stabilised at 709,352 B for 6 days (was 696 KB on Aug 10, 709 KB on Aug 11, 709,352 B on Aug 12/13/14/15/16/17/18). The 13 KB re-encode from Aug 10 → Aug 11 was one-shot, not ongoing.
3. dashboard/arxiv-report-2026-08-01.html · 56,574 B (56 KB) · 2026-08-02 13:47 · delete — 16-day-old "weekly arXiv report" snapshot (was 15 days old on Aug 17), zero references in any current cycle doc, the setup pipeline, or README.md. arxiv.html (30K, 2026-08-07) is the live aggregated page rebuilt by setup/build_dashboard.py; this 16-day-old HTML is a stale render that the build script no longer produces. Pure orphan, safe to delete. Day-8 confirmation: still orphan, mtime unchanged at Aug 2 13:47, size stable at 56,574 B.
4. (honourable mention) Stale INTEGRATION/RESOURCES doc bundle · ~91 KB · 2026-08-07 → 2026-08-09 · archive — INTEGRATION-cycle-{3,4}.md + RESOURCES-cycle-{3,4}.md + the *-delta-cycle-2c.md pair are all superseded by cycle 5/6 (latest mtime Aug 10). Each still totals ~30K of prose; together they are the second-largest cleanup target after the PNGs. Note: the *-audio.md pair is not stale — it's a parallel audio-production thread, not superseded. Day-8 confirmation: no change vs. Aug 17. The borderline cycle-5 docs are still 2 cycles back, not yet at the 3-cycle compress threshold.

What is "mtm-resource-sweep" doing in the daily-pipeline UI

Unchanged from Aug 10/11/12/13/14/15/16/17. The 08-18 daily file's Tab 04 (Tool Integration) and Tab 08 (Cost & Routing) both reference the mtm pipeline indirectly through the Stripe 5,200-tool week-5 / Bedrock AgentCore GA+ week-3 / 4-deployment cohort 6-layer router toolchain. The hand-off anchor drop into mtm-resource-sweep is still rendered from app.py:605 for any panel whose prompt block contains that literal; the 08-18 file's Tab 04 prompt is the 5,200-tool factory + 5-layer capability-token + 60-day deprecation_window, not the mtm-sweep hand-off, so the label is dormant again today.

• app.py line 605: the literal string drop into mtm-resource-sweep lives inside a _section_h("Prompt — " + p.tab_label) block; it is the .h text of a <div class="card kind-prompt"> that renders the parsed prompt from each panel inside the Inspector tab.
• verify.mjs line 57: the literal-vs-shape match is text.includes('mtm-resource-sweep') || text.includes('mtm-sweep'); both anchors are still resolvable from the live page text even when no panel's prompt block contains the literal.
• Source-of-truth HTML corpus: the long-form dailies in daily-pipeline-space/data/ (2026-08-07, 08-08, 08-09 × N, 08-10, 08-11, 08-12, 08-13, 08-14, 08-15, 08-16, 08-17, 08-18 now) all contain some prompt content. The 08-18 file's Tab 04 prompt is the 5,200-tool factory + 5-layer capability-token + 60-day deprecation_window, not the mtm-sweep hand-off; the 08-09 gradio file is still the only one that contains the literal.
• Semantic meaning: unchanged — the daily-pipeline is a prompt factory; mtm-resource-sweep is the target tool the prompts get handed to. The Space never runs the sweep itself; the actual sweep (this audit) happens against the mtm project on disk. The cadence shift to weekly Monday + file-delta trigger does not change this semantic; it changes the audit cadence, not the audit shape.

Stale doc inventory (re-verified, mtime drift = 0 since Aug 14)