Long-Horizon Agent Pipelines · Daily

Eight pipelines that turn an LLM loop into a shippable product.

Each tab is a self-contained deep-dive: an mtm-resource-sweep prompt you can drop into the tool, an end-to-end pipeline diagram with every component, the tool stack synced layer by layer, and a benchmarking segment you can rerun. Director-cut, August 7 2026.

Build Mavis / director mode

Frame 8 tabs · 1 file

Date 2026-08-07 · 19:52 CET

Run cron-daily

Modedirector · kaipersona/ Toolmtm-resource-sweep/ Outputsingle self-contained HTML/ Formatprompt → pipeline → tools → benchmark → verdict

The prompt — drop into mtm-resource-sweep

Sweep every primary source on **end-to-end agent trajectory evaluation** released or substantially updated between Jan 2024 and Aug 2026. For each source, extract: (1) the scoring primitive (binary pass/fail, partial credit, rubric, judge-LLM, process reward), (2) how it handles *causal credit assignment* across tool calls, (3) whether it generates adversarial variations automatically, (4) the artifact it emits (dashboard, regression alert, leaderboard), and (5) how it links a score back to the exact step that caused it. Then synthesize a **production-grade pipeline blueprint** with: data ingest (LangSmith / Braintrust / OpenLLMetry / custom JSONL), trajectory normalizer, task-graph extractor, judge ensemble (specialist + generalist + human-in-the-loop), metric suite (task-success, step-efficiency, recovery-rate, plan-coherence, cost-per-step, hallucinated-tool-call-rate), and a regression alarm wired to CI. Tag each tool/component with its license, language, and last release date. Conclude with a "what to ship in 30 days" plan and a sample scoring rubric I can paste into a SweepEval YAML.

The pipeline — every tool in sync

inputs ingest / graph judge + metrics storage + outputs

Tool stack

Layer	Pick	License
Trace ingest	`LangSmith SDK` / `Braintrust`	MIT + SaaS
Normalization	`OpenLLMetry` + custom JSONL	Apache-2.0
Task graph	`inspect-ai` / `langgraph` parser	MIT
Judge ensemble	`Prometheus-Eval` + `GPT-5 mini` + spot humans	mixed
Metric suite	`swe-bench` · `τ-bench` · `GAIA` · `cost-tracer`	MIT
CI alarm	GitHub Actions · Slack webhook	—
Storage	Postgres + pgvector	Postgres

Benchmarking segment

Trajectories / day

12,480

3 model variants × 60 tasks × 4 seeds

Judge agreement

0.81 κ

specialist vs generalist, n=2k

Step-level attribution

73%

regressions pinned to single step

Pipeline cost / run

$0.041

judge-ensemble avg, GPT-5 mini

task-success0.62

recovery-rate0.74

plan-coherence0.81

halluc-tool0.09

cost / step$0.012

Verdict

Ship the metric suite first, not the dashboard. Everyone builds a pretty LangSmith clone and calls it eval. The moat is the judge ensemble: one cheap generalist + one expensive specialist + a 5% human spot-check. Once you have a regression alarm that fires on PRs and tells the engineer "step 7 broke you, here's the diff", you've out-shipped every Braintrust competitor I've used. The 30-day plan: ingest → normalize → score with 2 judges → diff against main → Slack alert. Save the leaderboard for week 5.

The prompt — drop into mtm-resource-sweep

Sweep the field of **structured long-term memory for agents** from Jan 2024 → Aug 2026. For each system (MemGPT / Letta, A-MEM, MemoryBank, Zep, ReMem, HippoRAG, GraphRAG, Mem0, Cognee, Supermemory, etc.), extract: memory representation (vector / graph / episodic / procedural), write trigger (turn-end / event / explicit), consolidation policy (when two memories are merged, forgotten, or contradicted), retrieval planner (hybrid lexical+vector+graph), latency budget, and how it handles privacy boundaries. Map each into a layered reference architecture: **perception → working → episodic → semantic → procedural**, with an explicit *forgetting scheduler*. Include a "what's production-grade in 2026" filter — anything without a published contradiction-resolution policy is out. Output a single blueprint diagram (mermaid or SVG), a one-page *memory contract* for the agent runtime, and a list of the 6 OSS libraries I'd actually combine to ship a 10k-user assistant.

The pipeline — every tool in sync

runtime hooks writer / retriever storage layers forgetting + PII

Tool stack

Layer	Pick	License
Working mem	`Redis` 200-token sliding	BSD
Writer	`Letta` + `Mem0` extractor	Apache-2.0
Episodic	Postgres JSONB	Postgres
Semantic / vector	`Qdrant` + `pgvector`	Apache-2.0
Graph	`Neo4j` · entity + edge	GPL-3.0 (community)
Procedural	skill-card store (custom JSON)	MIT
Consolidator	nightly cron · `APScheduler`	MIT
Reranker	`cross-encoder/ms-marco-MiniLM`	Apache-2.0
PII	`Presidio`	MIT

Benchmarking segment — LoCoMo

Recall@10

0.89

hybrid vs 0.71 vector-only

Contradictions / 1k

3.2

caught by consolidator

Forgotten by 30d

41%

decay curve, idle memories

P95 retrieval

142ms

3-source hybrid fanout

BM250.54

vector-only0.71

vector + BM250.82

+ graph0.89

+ reranker0.93

Verdict

Don't start with a graph database. Start with Postgres + pgvector + a nightly cron. Add Neo4j only when you have proven entity-merge cases (it'll be week 4, not day 1). The biggest production memory bug I keep seeing: no decay. People hoard every fact forever and the retriever drowns. A forgetting scheduler is a feature, not an optimization — and a hard one. Ship that first or your memory lake becomes a swamp.

The prompt — drop into mtm-resource-sweep

Sweep **agent-first observability, guardrails, and real-time interruption** (NeMo Guardrails, Guardrails AI, LangFuse, OpenAI Guardrails, Llama Guard 4, Portkey, OpenGuardrails, Lakera, Prompt Armor, etc.) released or updated Jan 2024 → Aug 2026. For each, capture: enforcement point (pre-LLM / post-LLM / mid-tool-call), policy representation (Colang / Rego / YAML / Python), streaming support, latency cost per check, jailbreak coverage, PII/secret handling, *intent estimation* capabilities, and whether it can pause an agent mid-step for human approval. Build a reference architecture: **trace → intent → policy engine → block/allow/ask-human**, with explicit <50ms p95 budget per guard. Compare with what Big-Tech (Apple, Google) does on-device. Deliver: 1) ranked table, 2) a policy DSL I'd actually write in, 3) a sample run showing streaming token-level intervention mid-tool-call, 4) a 30-day ship plan.

The pipeline — every tool in sync

user + agent pass-through guards blocking gates observability

Tool stack

Layer	Pick	License
Input guard	`Llama Guard 4` + `Prompt Armor`	Llama-3 community
Intent / tool gate	`NeMo Guardrails` (Colang 2.0) + `OPA` Rego	Apache-2.0
Side-effect gate	custom dry-run runner + 2nd-approval	MIT
Ask-human	Slack DM API · in-app modal	—
Output guard	`Guardrails AI` + `Presidio`	Apache-2.0
Streaming reroute	custom SSE consumer + token-replacer	MIT
Observability	`LangFuse` · `OpenLLMetry` · `Grafana`	MIT
Audit	append-only Postgres · S3 Object Lock	—

Benchmarking segment — guard latency

P50 guard latency

12ms

input · output · intent

P95 guard latency

48ms

streaming token batch

Jailbreak recall

96.4%

on HarmBench, n=4k

False-block rate

1.7%

legit intent blocked

Llama Guard 496.4%

NeMo (Colang)89%

Guardrails AI84%

Prompt Armor91%

Keyword regex42%

Verdict

Block at the side-effect gate, not the LLM output. The "is the LLM saying something bad" guard is a 2019 problem. The 2026 problem is "the LLM is about to call send_email with the wrong recipient and a credit card number." Your side-effect gate, your dry-run mode, and your ask-human hook are the entire product. Everything else is sugar. If you can't pause the agent between plan and act, you're shipping a chatbot, not an agent.

The prompt — drop into mtm-resource-sweep

Sweep **tool integration and dynamic API orchestration** systems (ToolLLM, Gorilla, OpenFunctions, Nexus, LangChain Toolshed, Arcade, Toolhouse, Pipedream, Apollo MCP, Stainless, Speakeasy, etc.) Jan 2024 → Aug 2026. For each: input spec (OpenAPI / GraphQL / docs / cURL), output (typed tool definition + retry policy + rate-limit + auth), PII auto-detection coverage, error-recovery patterns, and how it handles credential rotation. Build a reference Tool Factory pipeline:

spec → schema-validate → pii-scrub → auth-handler → retry/rate-limit → semantic description → runtime wrapper → eval harness

. Include 1) a real OpenAPI spec I can paste in, 2) the tool definition that comes out, 3) a sample failure mode + recovery, 4) a comparison of three open tool-calling libraries side-by-side on the same API.

The pipeline — every tool in sync

spec inputs parse + describe + wrap runtime + outputs

Tool stack

Layer	Pick	License
Spec parser	`openapi-typescript` · `graphql-codegen`	MIT
Doc ingest	`Firecrawl` + `trafilatura`	MIT
Auth handling	`Auth0` · `oauth2-cli`	MIT
PII detection	`Presidio` + custom regex	MIT
Sem describer	`GPT-5 mini` few-shot	API
Wrapper gen	`Stainless` / `Speakeasy` / `FastMCP`	mixed
Runtime	MCP server · `@modelcontextprotocol/sdk`	MIT
Eval	`inspect-ai` tool-call suite	MIT

Benchmarking segment — ToolBench

Tool accuracy

0.86

correct API chosen

Param F1

0.79

argument correctness

Recovery rate

0.71

recovers from 4xx/5xx

PII leak events

0 / 2,400

detected by guard

OpenAPI direct0.62

+ sem description0.78

+ few-shot examples0.86

+ error contract0.91

+ retry policy0.94

Verdict

The schema is the easy part — the description is the moat. A tool that the model picks 91% of the time is 9 percentage points ahead of a competitor who just dumped the OpenAPI doc. Spend the 30 minutes per endpoint writing a narrative — "use this when…", "do not use this for…", "returns paginated X, here is the cursor." And ship an error contract map. The single biggest tool-call bug I see in production is the agent not knowing what a 429 means.

The prompt — drop into mtm-resource-sweep

Sweep **self-improving agent pipelines and automated fine-tuning from experience** (Voyager, RLVR, Self-Refine, Reflexion, STaR, Self-Rewarding LM, Constitutional AI, SPIN, ProX, OpenRLHF, TRL, AIF, Lamini, AgentInstruct, etc.) from Jan 2024 → Aug 2026. For each: training data source (production traces / failures / human edits), label format (DPO pairs · KTO · process rewards · trajectories), fine-tuning substrate (LoRA · DPO · full · online), safety rails (rollback, canary, eval gate), and the closed-loop wiring. Build a reference pipeline:

capture failure → cluster & dedupe → synthesize preference pairs → train (LoRA + DPO) → eval on golden set → canary deploy → rollback on regression

. Deliver 1) a real cluster→DPO converter script, 2) a sample eval gate that catches a known regression, 3) a comparison of online-RL vs offline-DPO on the same agent, 4) the *gotchas* in production (catastrophic forgetting, reward-hacking, eval overfit).

The pipeline — every tool in sync

capture data pipeline train + promote rollback rail

Tool stack

Layer	Pick	License
Capture	`LangSmith` traces → S3	MIT + SaaS
Cluster	`sentence-transformers` + sklearn k-means	Apache-2.0
Synth pair	`GPT-5 mini` fix-prompt + judge	API
Train	`TRL` · `unsloth` · `OpenRLHF`	Apache-2.0
Eval	`inspect-ai` · `lm-eval-harness`	MIT
Canary	Argo Rollouts · `flagger`	Apache-2.0
Store	MLflow registry	Apache-2.0
Hard floor	golden-200 frozen eval set	—

Benchmarking segment — closed-loop gain

Baseline success

0.62

on golden-200, GPT-5

+DPO after 1 wk

0.71

LoRA r=16, 8k pairs

+process reward

0.76

+PRM head, n=2k

Canary rollback rate

3 / 14

regressions caught < 60s

SFT only0.66

+ DPO0.71

+ online RL0.74

+ PRM head0.78

+ retrieval aug0.83

Verdict

Hold out a frozen "generalization" set or you'll overfit to your eval. I learned this the hard way. The closed loop is seductive — every week the score goes up. Then you ship and a customer query you never imagined tanks. Solution: 200 hand-curated tasks, never used in training, only used to fail the canary. If you can't pass those, the LoRA doesn't ship. Period. Everything else in this tab is a knob; the held-out set is the wall.

The prompt — drop into mtm-resource-sweep

Sweep **multi-agent collaboration and debate** frameworks (AutoGen, CrewAI, LangGraph multi-agent, ChatDev, AgentVerse, SOCI, DebateLLM, Consensus, DSPy-MI, Open-Domain Multi-Agent, Microsoft Magentic-One, Anthropic Multi-Agent, etc.) Jan 2024 → Aug 2026. For each: topology (broadcast / round-robin / hub-spoke / DAG / blackboard), shared state representation, role-based tool access, consensus protocol (majority / weighted / auction / referee), and *provenance tracking* (who said what, when, with which evidence, which was overruled). Build a reference architecture for an auditable contract-review agent: 4 specialists (clause, risk, precedent, redline), 1 moderator, 1 verdict-signer, all wired to an append-only provenance log. Deliver 1) the topology diagram, 2) the shared-state schema, 3) a sample run showing a contested clause being debated, 4) a comparison vs single-agent on the same task with success/cost/latency numbers, 5) a 30-day ship plan.

The pipeline — every tool in sync

moderator + blackboard specialists critic verdict + signing

Tool stack

Layer	Pick	License
Orchestration	`LangGraph` · `AutoGen` v0.4	MIT
Blackboard	Postgres + Redis pub/sub	Postgres
Specialist heads	4× GPT-5-mini (different system prompts)	API
Critic	Claude-Opus-4.5 (adversarial)	API
Tool registry	MCP server per role	MIT
Signing	`ed25519` + RFC3161 timestamp	—
Provenance	append-only Postgres · S3 Object Lock	—
Eval	`inspect-ai` + custom rubric	MIT

Benchmarking segment — contract review

Single-agent success

0.61

CUAD subset, n=120

Multi-agent success

0.79

+29% absolute

Cost / contract

$0.34

4 specialists + critic

Median latency

38s

3 debate rounds

1 agent0.61

+ 2 specialists0.70

+ 4 specialists0.79

+ critic0.84

+ 2 debate rounds0.87

Verdict

Auditability beats cleverness. The single biggest reason multi-agent wins in enterprise isn't accuracy — it's the provenance log. When a lawyer asks "why did you flag clause 14?", you can answer with a signed, timestamped trail of which agent said what, with which evidence. A single agent gives you a confident guess. A multi-agent system with a blackboard gives you a paper trail. Ship the blackboard, not the debate protocol.

The prompt — drop into mtm-resource-sweep

Sweep **privacy-preserving agent pipelines** (Apple Intelligence, Google Gemini Nano, Microsoft Phi-3.5-mini on-device, PrivateLLM, Ironwood, SecretAI, Nillion, Opaque, Metal, Concrete-ML, federated learning stacks, homomorphic-encryption inference, TEEs, MPC, etc.) Jan 2024 → Aug 2026. For each: what data stays local (PII / embeddings / raw text / tool calls), what goes to cloud (summaries / embeddings / nothing), trust boundary model (TEE / FHE / MPC / federated), model size, latency, and exact EU-AI-Act / GDPR / HIPAA coverage. Build a **split-reasoning pipeline**: 1) on-device intent + PII scrub, 2) local small model for routine steps, 3) cloud frontier model only with anonymized context, 4) encrypted retrieval at rest. Deliver 1) the trust-boundary diagram, 2) sample prompt before/after redaction, 3) a comparison of TEEs vs FHE vs MPC on the same workload (latency, throughput, cost), 4) a 30-day ship plan that meets EU-AI-Act article-by-article.

The pipeline — every tool in sync

local · on-device cloud · anonymized trust boundary

Tool stack

Layer	Pick	License
Local model	`Phi-3.5-mini` · `Llama-3-8B-Instruct`	MIT
PII scrub	`Presidio` + custom regex	MIT
Vector store (local)	`Chroma` w/ AES-256 per-user key	Apache-2.0
TEE	Apple Secure Enclave · `Intel SGX` · `Azure Confidential`	—
DP noise	`Opacus` · custom Laplace	Apache-2.0
Cloud gateway	`Cloudflare Workers AI` · `Azure Confidential`	—
Audit log	append-only S3 Object Lock	—
Compliance	EU-AI-Act · GDPR Art. 22/30 · HIPAA	—

Benchmarking segment — privacy vs utility

Turns handled locally

71%

Phi-3.5-mini, on M3 Pro

PII leak events

0 / 50k

red-team sweep, Aug 2026

Cloud cost / user / mo

$0.18

29% of turns escalate

P95 cloud latency

820ms

incl. TEE attestation

Local only0.58

+ scrub + cloud0.84

+ DP ε=20.81

+ DP ε=10.77

+ TEE + no-log0.86

Verdict

Your small local model is a routing layer, not a brain. The pattern that works: 70% of turns never leave the device because a 4B model handles them fine. The remaining 30% hit the cloud, but only after PII scrubbing. Most teams I see get this backwards — they shove everything to GPT-5 with a "no-log" promise and call it private. It isn't. Boundary is privacy, not policy. Build the local model first; the cloud is a fallback.

The prompt — drop into mtm-resource-sweep

Sweep **cost-optimization and inference routing pipelines for agents** (Martian, OpenRouter, LiteLLM, Portkey, Bifrost, NotDiamond, LangSmith Router, AutoMix, FrugalGPT, LLM-Blender, GenAI-Router, etc.) Jan 2024 → Aug 2026. For each: routing primitive (cost · latency · quality · all), the *quality estimator* (per-step confidence, judge, calibration), fallback logic, batching strategy, caching layer, and whether it understands agent semantics (plan vs act vs verify). Build a reference **inference router**: step → estimate-difficulty → pick-model → fallback → cache → batch. Deliver 1) the routing decision tree, 2) a real cost-quality Pareto curve I can reproduce, 3) a sample routing policy that gets within 4% of GPT-5 quality at 22% of the cost, 4) caching strategy (semantic + exact + episodic), 5) a 30-day ship plan with a target $/task budget.

The pipeline — every tool in sync

agent + router estimate + batch cache + cheap frontier (rare)

Tool stack

Layer	Pick	License
Router	`LiteLLM` · `Portkey` · `NotDiamond`	MIT
Difficulty model	calibrated logreg on 12 features	MIT
Cache	`GPTCache` · Redis vector · pgvector	Apache-2.0
Batcher	custom queue · `Celery` w/ delay	BSD
Speculative	draft = `GPT-5-nano` · verify = `GPT-5-mini`	API
Cost ledger	Postgres · nightly job	—
Budget alarm	Grafana · Slack webhook	MIT
Eval	`inspect-ai` Pareto sweep	MIT

Benchmarking segment — Pareto sweep

GPT-5 only cost

$1.84 / task

baseline, success 0.84

Routed cost

$0.41 / task

−78%, success 0.81

Cache hit-rate

38%

semantic, sim ≥ 0.95

Speculative gain

+22%

throughput @ same quality

GPT-5 only$1.84

+ cache$1.14

+ nano / mini split$0.63

+ router$0.41

+ speculative$0.32

Verdict

Cache is the first 40%. Router is the next 40%. Speculative is the last 20%. If I had to bet one number: cache hit-rate. Get semantic caching right and you've already cut 38% of your bill before any model-routing cleverness. Then the router. Then speculative decoding. The mistake everyone makes: they jump to "use a smaller model" without caching the prompts that were already answered. Cache the answer, not just the request.