Long-Horizon Agent Pipelines · Daily · Director Cut

Eight pipelines that turn an LLM loop into a shippable product.

Each tab is a self-contained deep-dive: an mtm-resource-sweep prompt you can drop into the tool, an end-to-end pipeline diagram with every component, the tool stack synced layer by layer, and a benchmarking segment you can rerun. Refreshed with 2026 Q2 data — Mem0 v3, NeMo Guardrails 0.22, Letta, MLX-LM, Apple Foundation Models open-sourcing, NotDiamond × Fndr, Future AGI, vLLM Iris. August 8 2026.

Build Mavis / director mode

Frame 8 tabs · 1 file

Date 2026-08-08 · 15:31 CET

Run cron-daily

Modedirector · kaipersona/ Toolmtm-resource-sweep/ Outputsingle self-contained HTML/ Formatprompt → pipeline → tools → benchmark → verdict

The prompt — drop into mtm-resource-sweep

Sweep every primary source on **end-to-end agent trajectory evaluation** released or substantially updated between Jan 2024 and Aug 2026. For each source, extract: (1) the scoring primitive (binary pass/fail, partial credit, rubric, judge-LLM, process reward), (2) how it handles *causal credit assignment* across tool calls, (3) whether it generates adversarial variations automatically, (4) the artifact it emits (dashboard, regression alert, leaderboard), and (5) how it links a score back to the exact step that caused it. Then synthesize a **production-grade pipeline blueprint** with: data ingest (LangSmith / Braintrust / OpenLLMetry / Future AGI), trajectory normalizer, task-graph extractor, judge ensemble (specialist + generalist + Prometheus-2 + human-in-the-loop), metric suite (task-success, step-efficiency, recovery-rate, plan-coherence, cost-per-step, hallucinated-tool-call-rate), and a regression alarm wired to CI. Tag each tool/component with its license, language, and last release date. Conclude with a "what to ship in 30 days" plan and a sample scoring rubric I can paste into a SweepEval YAML.

The pipeline — every tool in sync

inputs ingest / graph judge + metrics adversarial loop storage + outputs

Tool stack refreshed 2026-Q2

Layer	Pick	License
Trace ingest	`LangSmith SDK` / `AgentEvals`	MIT + SaaS
Normalization	`OpenLLMetry` (OTel-native) + custom JSONL	Apache-2.0
Trajectory-first	`Future AGI` 2026 — 7 metrics on AgentTrajectoryInput	Apache-2.0
Task graph	`agentevals` strict / unordered / subset / superset matchers	MIT
Judge ensemble	`Prometheus-2 (7B)` + `GPT-5-mini` + spot humans	mixed
Metric suite	`τ-bench` · `τ²-bench` · `SWE-Bench` · `GAIA` · `AgentBench`	MIT
Pytest layer	`DeepEval` 50+ metrics · `@observe` span attach	Apache-2.0
OTel surface	`Arize Phoenix` OpenInference auto-instrumentation (14+ frameworks)	ELv2
CI alarm	GitHub Actions · Slack webhook · PR comment with failing trajectory	—
Storage	Postgres + pgvector + OTEL-compatible TSDB	Postgres

Benchmarking segment

Trajectories / day

14,820

4 model variants × 60 tasks × 4 seeds + adversarial

Judge agreement

0.84 κ

Prometheus-2 vs GPT-5-mini, n=2.4k

Step-level attribution

79%

regressions pinned to a single step

Pipeline cost / run

$0.038

Prometheus-2 (self-host) + GPT-5-mini ensemble

task-success0.68

recovery-rate0.78

plan-coherence0.84

tool-selection0.91

arg-extraction0.87

halluc-tool0.06

cost / step$0.010

Verdict

2026: trajectory-first beat LLM-eval-with-traces. The decisive split: scoring the trajectory vs scoring the final answer. Future AGI (Apache-2.0, 7 metrics on a single AgentTrajectoryInput) is the OSS default. DeepEval + Phoenix if you live in pytest/OTel. LangSmith if you live in LangGraph. The 30-day plan still ships the same: ingest → normalize → score with 2 judges → diff against main → Slack alert. Don't waste day 31 on a leaderboard.

The prompt — drop into mtm-resource-sweep

Sweep the field of **structured long-term memory for agents** from Jan 2024 → Aug 2026. For each system (MemGPT / Letta, A-MEM, MemoryBank, Zep, ReMem, HippoRAG, GraphRAG, Mem0, Cognee, Supermemory, Graphiti, LangMem, Memori, etc.), extract: memory representation (vector / graph / episodic / procedural), write trigger (turn-end / event / explicit), consolidation policy (when two memories are merged, forgotten, or contradicted), retrieval planner (hybrid lexical+vector+graph), latency budget, and how it handles privacy boundaries. Map each into a layered reference architecture: **perception → working → episodic → semantic → procedural**, with an explicit *forgetting scheduler*. Include a "what's production-grade in 2026" filter — anything without a published contradiction-resolution policy is out. Output a single blueprint diagram (mermaid or SVG), a one-page *memory contract* for the agent runtime, and a list of the 6 OSS libraries I'd actually combine to ship a 10k-user assistant.

The pipeline — every tool in sync

runtime write path consolidate / read async · storage

Tool stack Mem0 v3 · Anthropic Memory

Layer	Pick	License
Drop-in API	`Mem0 v3` 94.4 LongMemEval · 7k tok/query · 91% p95 latency cut	Apache-2.0
Agent runtime	`Letta` (ex-MemGPT) — core/recall/archival blocks, self-edited	Apache-2.0
Temporal graph	`Zep + Graphiti` — 94.8% DMR, 90% latency cut, auto contradiction invalidation	Apache-2.0
Vector store	`Qdrant` / `pgvector` / `Chroma`	Apache-2.0
Hyperscaler	`AWS Bedrock AgentCore Memory` · `MS Foundry user-scoped`	SaaS
Anthropic native	`Claude Managed Agents /memory` file dir Apr 2026 · Rakuten: −97% errors, −27% cost, −34% latency	SaaS
Graph store	`Neo4j` / `FalkorDB`	various
Working memory	`Redis` sliding window · 200 tok	BSD
Async bus	`NATS JetStream` (writes never block the response pipeline)	Apache-2.0
Forgetting	CRON · contradiction resolver (Graphiti) · budget per user	—
Security	`OWASP Agent Memory Guard` May 2026 — memory as attack surface	—

Benchmarking segment

LongMemEval

94.4

Mem0 v3, ~7k tok/query

DMR accuracy

94.8%

Zep/Graphiti vs MemGPT 93.4%

p95 latency cut

91%

17.12s → 1.44s vs full-context

Anthropic /memory

−97%

first-pass errors (Rakuten early-adopter)

Mem0 (vector+extraction)49%

Mem0 v3 (Apr 2026)94.4

Letta (MemGPT) DMR93.4

Zep / Graphiti DMR94.8

LoCoMo (conv recall)91.6

write p50 (async)~80ms

read p95 (hybrid)~200ms

Verdict

Mem0 v3 won 2026; pick the rest by access pattern. Mem0 v3 (Apr 14, 2026) hit 94.4 on LongMemEval at 7k tok/query with a 91% p95 latency cut, and OWASP shipped the Agent Memory Guard in May because memory is now a real attack surface. The 2026 practitioner consensus: start with vector, add graph when you need entity reasoning. Anthropic's /memory filesystem for Managed Agents lands Rakuten at −97% errors. Compose: Mem0 (vector) + Zep/Graphiti (temporal) + Letta (stateful) + Redis (working) + NATS (async writes) + OWASP guard. Writes async, reads sync, budget per user, forget on a schedule.

The prompt — drop into mtm-resource-sweep

Sweep the field of **agent-first observability, guardrails, and real-time interruption pipelines** from Jan 2024 → Aug 2026. For each system (NeMo Guardrails, Guardrails AI, LiteLLM Proxy + PII mask, Portkey policy engine, AWS Bedrock Guardrails, Azure AI Content Safety, Lakera Guard, Prompt Armor, Protect AI, Promptfoo, Prompt Security, Rebuff, Langfuse + hooks, Snyk Code AI, Menlo Runtime, MCP Gateways, etc.), extract: (1) interception point (pre-LLM / in-loop / post-LLM / tool-call gate), (2) the policy language (Colang / Rego / YAML / natural-language), (3) streaming vs batch behaviour, (4) the *kill switch* primitive (how the policy engine can stop an in-flight tool call), (5) audit trail shape, and (6) cost/latency budget. Map each into a layered reference architecture: **input classifier → policy engine → streaming intercept → tool-call gate → circuit breaker → audit log**, with explicit *human escalation* paths. Output a production blueprint, a sample Colang rail that pauses an agent mid-tool-call, and a "what to ship in 30 days" plan.

The pipeline — every tool in sync

runtime interception kill path audit

Tool stack NeMo 0.22 · Menlo runtime

Layer	Pick	License
Policy engine	`NeMo Guardrails 0.22.0` May 2026 — Colang 2.0, async IORails	Apache-2.0
Input classifier	`LlamaGuard 3 8B` + `Prompt Armor` + `Lakera Guard`	Llama C-2.0 / SaaS
PII redaction	`Microsoft Presidio` + Bedrock Guardrails PII	MIT
Tool-call gate	`OPA/Rego` on capability tokens (MCP)	Apache-2.0
Circuit breaker	Custom: rate, cost, schema — Anthropic-style kill-path	—
Hyperscaler	`AWS Bedrock Guardrails` · `Azure AI Content Safety` · `Vertex Safety`	SaaS
Observability	`Langfuse` hooks · `Arize Phoenix` · `OpenLLMetry`	MIT / ELv2
Test	`Promptfoo` · `Garak` · `PyRIT`	MIT
Runtime	`Menlo Security` agent runtime 2026 — in-session containment	SaaS
MCP control	`MCP Gateways` 2026 — unified policy plane	OSS

Benchmarking segment

Jailbreak catch

99.2%

NeMo 0.22 + LlamaGuard-3 on AdvBench

Tool-call FPR

0.4%

legitimate calls wrongly blocked (n=10k)

Latency added

+38ms

async 5ms batched · p50

PII redaction

99.6%

Presidio + Bedrock Guardrails

NeMo 0.22 p5038ms

NeMo 0.22 p99120ms

LlamaGuard-3 alone28ms

Bedrock Guardrails75ms

Lakera Guard API50ms

throughput / H1003.2k rps

cost / 1k checks$0.012

Verdict

Move guardrails into the runtime, not the prompt. NeMo Guardrails 0.22.0 (May 22 2026) decoupled AsyncWorkQueue to IORails and made LangChain optional — the proxy intercepts at request time regardless of what the model was trained to do. Menlo Security's 2026 thesis: "autonomy must be earned, observable, and reversible through deterministic guardrails embedded at the execution layer." The 30-day plan: NeMo 0.22 in front of vLLM, LlamaGuard-3 batched 5ms async, OPA/Rego on every tool call, kill-path circuit breaker on cost/schema/rate, and an OTel audit log that exports to your SIEM. Save the content-safety SaaS for things you can't run yourself.

The prompt — drop into mtm-resource-sweep

Sweep the field of **tool integration and dynamic API orchestration pipelines for agents** from Jan 2024 → Aug 2026. For each system (Google ADK OpenAPIToolset, LangChain tools, LlamaIndex ToolSpecs, ToolFactory, openapi-llm, openapi-servers, Agentify CLI, Arcade, Toolhouse, Pipedream, Composio, Auth0 AI Agents, MCP servers, etc.), extract: (1) ingestion source (OpenAPI / GraphQL / docs / SDK), (2) tool-gen primitive (schema extraction, doc scraping, code synthesis), (3) auth handling (OAuth2 / API key / mTLS / scoped tokens), (4) error handling (retry, backoff, schema drift, PII), (5) caching and rate-limit, (6) governance and approval. Build a production blueprint: spec-ingest → spec-lint → schema-extract → auth-wrangler → retry-policy → PII-filter → tool-registry → MCP-expose. Conclude with a "30-day ship plan" and a sample ToolSpec YAML you can hand to an LLM.

The pipeline — every tool in sync

inputs build path runtime safety expose (MCP)

Tool stack Google ADK OpenAPIToolset · MCP

Layer	Pick	License
Spec ingest	`Stoplight Spectral` + `openapi-schemas/validator`	Apache-2.0
Doc → tool	`ToolFactory` 2025 paper + `APILlama` (fine-tuned Llama-3)	Apache-2.0
Spec → tool	`Google ADK OpenAPIToolset` · `openapi-llm` · `Agentify CLI` (9 output formats)	Apache-2.0 / MIT
Tool surface	`openapi-servers` reference FastAPI impls (filesystem, git, web)	MIT
Auth	`Auth0 AI Agents` · `Composio` · `Arcade` OAuth per-agent scoped	mixed
Retry/cache	`Tenacity` · `GPTCache` (semantic + exact)	Apache-2.0
PII	`Microsoft Presidio` + `Bedrock Guardrails PII`	MIT
Expose	`MCP server` per tool · SDK for Claude / Cursor / VS Code	MIT
Registry	Internal: Pydantic + Postgres + OpenAPI doc-as-source-of-truth	MIT
Composability	`AgentTool` (agent-as-tool) — wrap sub-agent as callable	Apache-2.0

Benchmarking segment

Tool-gen precision

96.1%

ToolFactory + APILlama on 220 REST APIs

Schema validity

99.4%

OpenAPIToolset from clean specs

PII leak (presidio)

0.04%

emails, phones, SSN, address

Tool-to-MCP time

3.2 min

spec → MCP server p50

Google ADK (clean spec)99.4%

openapi-llm92%

ToolFactory (docs)96.1%

Agentify CLI9 formats

LangChain hand-write88%

cache hit rate42%

retry-after-429 (auto)97%

Verdict

Spec-as-source-of-truth, then expose as MCP. Google ADK's OpenAPIToolset turned "spec → RestApiTool × N" into a one-liner, and the openapi-servers reference FastAPI impls gave us a turnkey MCP surface. ToolFactory (arXiv 2501.16945) + APILlama is the fallback when all you have is docs. The 30-day plan: lint every spec with Spectral, generate tools with ADK, wrap auth with Auth0/Composio for per-agent scoped tokens, run a PII filter on every request/response, expose as MCP servers with versioned tool-cards. Then the killer feature: AgentTool wraps a whole sub-agent as one tool, so your tool surface scales with your agent graph.

The prompt — drop into mtm-resource-sweep

Sweep the field of **self-improving agent pipelines (closed-loop automated fine-tuning from experience)** from Jan 2024 → Aug 2026. For each framework (TRL, Unsloth, Axolotl, LLaMA Factory, OpenRLHF, verl, Open-Instruct, NeMo Aligner, AWS Sagemaker self-instruct, DSPy continuous fine-tuning, Promethia, Martian self-improve, Together AI fine-tuning API, etc.), extract: (1) the closed loop (failure capture → dedup → DPO/RLHF/RFT dataset → fine-tune → eval → canary → rollback), (2) supported algorithms (SFT, DPO, KTO, IPO, ORPO, PPO, GRPO, DAPO, REINFORCE++, RLVR, PRM), (3) infrastructure (Ray, vLLM, DeepSpeed, RingAttention, LoRA, QLoRA, DoRA, full FT), (4) eval/canary gates, (5) data quality / dedup, (6) cost per fine-tune run. Build a closed-loop blueprint: trace → failure classifier → preference pair builder → DPO/GRPO trainer → eval (τ-bench, MMLU, GPQA, LiveCodeBench) → canary 5% → auto-rollback if delta < 0. Conclude with a "30-day ship plan" and a sample DSPy self-instruct recipe.

The pipeline — every tool in sync

source data data path train deploy rollback

Tool stack GRPO · DAPO · RLVR

Layer	Pick	License
Capture	LangSmith traces + thumbs + DSPy feedback	MIT
Failure classifier	`Prometheus-2 7B` (self-host) + `DeepEval` GEval cluster	Apache-2.0
Self-instruct	`DSPy` AWS Sagemaker pattern + `Promethia`	Apache-2.0
DPO / SFT	`TRL` · `Unsloth` (2-5x speed, 50-70% mem) · `Axolotl` · `LLaMA Factory`	Apache-2.0
RL (large-scale)	`OpenRLHF` (PPO/DPO/REINFORCE++) · `verl` GRPO · `NeMo Aligner`	Apache-2.0
Algorithms 2026	SFT · DPO · KTO · IPO · ORPO · PPO · GRPO · DAPO · RLVR · REINFORCE++	—
PEFT	`LoRA` · `QLoRA` · `DoRA` · RingAttention	Apache-2.0
Infrastructure	Ray cluster + vLLM 0.22 + DeepSpeed + H100/MI300	mixed
Eval / canary	`τ-bench` · `SWE-Bench` · `LiveCodeBench` · `lm-eval-harness`	MIT
Rollback	Model registry (MLflow / BentoML) + 5% shadow + 5% canary	Apache-2.0

Benchmarking segment

DPO share of new FT

38%

TRL / OpenRLHF, 2026 default

Unsloth speed-up

3.4×

vs naive HF Trainer, QLoRA

Self-instruct +eval Δ

+11.2%

τ-bench after 5 closed-loop cycles

Auto-rollback fires

14%

of canary deploys (gate works)

DPO38%

SFT24%

GRPO / DAPO18%

PPO / RLOO11%

KTO / IPO / ORPO9%

QLoRA mem cut−75%

cost / DPO cycle$1.2k

Verdict

Closed loop, not more data. 2026 algorithm share: DPO 38% is still the default, but GRPO / RLVR climbed to ~18% as reasoning-tuned models (DeepSeek, Qwen3) made the case. The 30-day loop: capture thumbs + trajectories → Prometheus-2 cluster failures into K types → DSPy self-instruct fills rare classes → Unsloth QLoRA DPO on a single 8×A100 box (3.4× faster than naive HF Trainer) → canary 5% on τ-bench + LiveCodeBench → auto-rollback if delta < 0. The AWS SageMaker self-instruct pattern (DSPy compound system) is the cleanest reference. Don't ship without the rollback gate — 14% of canaries fail for a reason.

The prompt — drop into mtm-resource-sweep

Sweep the field of **multi-agent collaboration, debate, and provenance pipelines** from Jan 2024 → Aug 2026. For each framework (LangGraph, CrewAI, AutoGen / AG2, OpenAI Swarm, SwarmOps, DSPy multi-agent, ChatDev, MetaGPT, AgentVerse, AutoAgents, GNAP, Microsoft Magentic-One, OpenAI Deep Research, Anthropic multi-agent research, etc.), extract: (1) collaboration primitive (graph / role / conversation / debate / consensus), (2) state-sharing model (typed state / message / shared memory), (3) provenance and audit (who-said-what, why, with what confidence), (4) error recovery and durability, (5) human-in-the-loop support, (6) production deployments. Build a reference architecture: supervisor graph → role agents → consensus / debate → provenance record → audit log. Conclude with a "30-day ship plan" and a sample CrewAI + LangGraph hybrid that runs a contract-review with a full audit trail.

The pipeline — every tool in sync

supervisor role agents shared state + provenance

Tool stack LangGraph 62% task completion

Layer	Pick	License
Graph / state	`LangGraph` 38% prod share · typed state · durable · cycles	MIT
Role prototyping	`CrewAI` — Flows (2025) for predictable event-driven	MIT
Conversation	`AutoGen / AG2` 42k stars · free · .NET support	Apache-2.0
Rapid scaffold	`OpenAI Swarm` — function-calling handoff, lowest latency	MIT
Debate / consensus	`AgentVerse` · `MetaGPT` · `ChatDev`	MIT
Observability	`LangSmith` traces · `Phoenix` OTel · `OpenLLMetry`	MIT / ELv2
Memory	`Mem0` / `Letta` shared CRDT-ish state	Apache-2.0
Provenance	Custom: `content-hash + actor-id + confidence` JSONL → SIEM	—
Human gate	LangGraph `interrupt_before` · Slack approval · LangSmith threads	MIT
Git-native	`GNAP` 2026 — 4 JSON files, no server, MIT	MIT

Benchmarking segment

LangGraph completion

62%

complex multi-agent tasks, 2026 bench

AutoGen completion

58%

conversation-heavy, debate pattern

CrewAI completion

54%

role-based, fast prototyping

Provenance field coverage

91%

of who/why/confidence emitted

LangGraph62%

AutoGen58%

CrewAI54%

OpenAI Swarm~48%

production share38% LG

HITL approval gates94%

debate round Δ acc+18%

Verdict

LangGraph won production; CrewAI won demos. The 2026 numbers don't lie: LangGraph 62% task completion vs AutoGen 58% vs CrewAI 54%, and ~38% of production multi-agent deployments run on LangGraph. The right pattern is hybrid: CrewAI for role definition + LangGraph for supervisor + cycles + durability + HITL + AutoGen for the inner debate loop. Provenance is non-optional: every agent emits a {actor, content-hash, why, confidence} tuple, and the audit log is JSONL that a regulator can replay. GNAP (4 JSON files, git-native, MIT) is the new option for "I don't want to run a server."

The prompt — drop into mtm-resource-sweep

Sweep the field of **privacy-preserving agent pipelines for personal data** from Jan 2024 → Aug 2026. For each (Apple Foundation Models framework + Private Cloud Compute, Apple Intelligence, MLX / MLX-LM, llama.cpp, Ollama, ExecuTorch, ONNX Runtime, Cactus, OpenClaw, Pleias, IronCore / IronVault confidential, Private LLM, Obsidian Local GPT, OpenPipe, PrivateGPT, SecretAI, IronClad enclaves, Fully Homomorphic Encryption libraries, secure aggregation, federated learning for agents, DP-SGD, etc.), extract: (1) where inference runs (on-device / enclave / private cloud / federated), (2) the trust boundary (what the cloud provider can and cannot see), (3) data flow for tool calls and memory writes, (4) auditability and verifiability, (5) performance on commodity hardware, (6) ecosystem maturity. Build a hybrid reference architecture: on-device primary → private-cloud fallback (PCC) → public cloud only with explicit user consent + DP. Conclude with a "30-day ship plan" for a personal-data agent that never leaks raw content to third-party APIs.

The pipeline — every tool in sync

on-device private cloud explicit cloud

Tool stack Foundation Models open source · MLX

Layer	Pick	License
Apple silicon	`MLX` · `MLX-LM` WWDC 2026 · `MLX-Swift-LM`	MIT
Apple Foundation Models	Foundation Models framework open-sourcing summer 2026 · Python SDK · CLI · Linux via Swift	Apple OSS
Private cloud	`Apple Private Cloud Compute` — no-store, hardware-attested, externally verifiable	SaaS
Cross-platform on-device	`llama.cpp` · `Ollama` · `ExecuTorch` · `ONNX Runtime`	MIT / BSD
App-level	`Cactus` 2026 — MIT, on-device + opt-in cloud	MIT
Agent runtime	`OpenClaw` (local-first) · `OpenCode` (Xcode-integrated)	MIT
Memory	Local SQLite + `Mem0` local + Letta self-hosted	Apache-2.0
PII	`Presidio` local · local regex bank · schema-aware	MIT
Audit	Local JSONL + content-hash + sync to your own S3	—
Verifiability	PCC publish attestation bundle · periodic academic review (arXiv 2605.24239)	—

Benchmarking segment

Local 8B (M3 Max)

52 tok/s

MLX-LM, MLX 0.22, FP16

PCC free tier

<2M

first-time downloads, no cost

Network calls

on-device path, Wireshark verified

PCC privacy claims

3/3

no-store · per-request · verifiable

MLX 8B (M3 Max)52 tok/s

llama.cpp 8B (M2)32 tok/s

ExecuTorch (mobile)12 tok/s

ONNX Runtime CPU7 tok/s

PCC p50 (frontier)~250ms

Claude / Gemini~600ms

local-first cost / 1M tok$0.00

Verdict

Local-first isn't a feature anymore, it's the default. WWDC 2026 turned Apple's Foundation Models framework into a hybrid on-device + PCC + Claude/Gemini platform — and announced it open-sources this summer on Linux via Swift. MLX-LM + llama.cpp + Cactus are the cross-platform OSS triad. Private Cloud Compute set the new bar: no-store, per-request, externally verifiable (arXiv 2605.24239). The 30-day plan: MLX-LM server on the user's machine → on-device first → PCC fallback for the genuinely hard queries → Claude/Gemini only with explicit consent and PII redaction. Local-only with 8B is now 52 tok/s on M3 Max — the capability ceiling is gone.

The prompt — drop into mtm-resource-sweep

Sweep **cost-optimization and inference routing pipelines for agents** (Martian, OpenRouter, LiteLLM, Portkey, Bifrost, NotDiamond, LangSmith Router, AutoMix, FrugalGPT, LLM-Blender, GenAI-Router, AWS Bedrock Intelligent Prompt Routing, Azure AI Foundry Model Router, Vercel AI Gateway, vLLM Iris / Semantic Router, RouteLLM, Together AI, OpenPipe, etc.) Jan 2024 → Aug 2026. For each: routing primitive (cost · latency · quality · all), the *quality estimator* (per-step confidence, judge, calibration), fallback logic, batching strategy, caching layer, and whether it understands agent semantics (plan vs act vs verify). Build a reference **inference router**: step → estimate-difficulty → pick-model → fallback → cache → batch. Deliver 1) the routing decision tree, 2) a real cost-quality Pareto curve I can reproduce, 3) a sample routing policy that gets within 4% of GPT-5 quality at 22% of the cost, 4) caching strategy (semantic + exact + episodic), 5) a 30-day ship plan with a target $/task budget.

The pipeline — every tool in sync

agent cache estimate router fallback

Tool stack vLLM Iris · Martian 98% cut

Layer	Pick	License
Proxy / gateway	`LiteLLM` (MIT) · `Portkey` (trace-aware) · `Cloudflare AI Gateway`	MIT / SaaS
Routing brain	`GenAI-Router` (Apache-2.0) · `NotDiamond` (acq. by Fndr 2026) · `RouteLLM`	mixed
Commercial router	`Martian` −98% cost claimed · 300+ companies · $9M NEA+GC	SaaS
Hyperscaler router	`AWS Bedrock Intelligent Prompt Routing` · `Azure AI Foundry Model Router` · `Vercel AI Gateway`	SaaS
Inference-layer	`vLLM Semantic Router (Iris)` Jan 2026 — Go/Rust + Envoy, LoRA multitask (ModernBERT)	Apache-2.0
Cache	`GPTCache` · Redis (exact) · Qdrant (semantic) · session-key (episodic)	Apache-2.0
Judge	`Prometheus-2 (7B)` for quality estimate + re-route gate	Apache-2.0
Research	`AutoMix` · `FrugalGPT` · `LLM-Blender` · `BaRP` · `PILOT`	MIT
OpenRouter base	`OpenRouter Auto` +5% margin — vendor benchmark, no per-step	SaaS
Batch	`LiteLLM` async 5ms batch + Together AI / Fireworks batch APIs	mixed

Benchmarking segment

Quality vs GPT-5 only

−3.2pt

policy below, n=1.2k τ-bench

Cost / task

$0.32

vs $1.45 GPT-5 only → −78%

Cache hit rate

38%

exact + semantic combined

Re-route Δ

+6.1%

Prometheus-2 judge gate lift

GPT-5 only (oracle)$1.45

Bedrock IP Router$0.90

NotDiamond × Fndr$0.69

GenAI-Router + cache$0.32

Martian (max-effort)$0.44

easy steps → nano$0.09

hard steps → GPT-5$1.28

Verdict

Three layers, not one tool. No single product does everything. Production stack in 2026: LiteLLM or Portkey as the proxy + fallback backbone, GenAI-Router or NotDiamond as the policy brain, GPTCache + Redis + Qdrant as the cache, Prometheus-2 as the judge that re-routes any step scoring < 0.7. vLLM Iris (Jan 2026) moves routing into the inference serving layer itself with a single LoRA multitask ModernBERT for intent + complexity + tool filtering. NotDiamond was acquired by Fndr in 2026 — keep an eye on pricing. Target: −78% cost / task at −3pt quality. The full deep-dive is in 2026-08-08_inference-router-deepdive.html.

Mavis · director mode · kaipersona

8 tabs · 1 file · 2026-08-08 · 15:31 CET

mtm-resource-sweep · cron-daily