Eight pipelines that turn an LLM from a demo into a system you can actually ship.

Tab 01 · End-to-End Agent Evaluation & Trajectory Benchmarking

Final-answer scoring marks a failing agent green. The answer was right. The path was wrong — 20 steps, two policy violations, an exfiltration. Trajectory eval closes that gap.

Why it matters

A correct answer reached in 20 steps with two policy-violating intermediate calls is a failing trajectory. Production agents need four layers: unit, trajectory, outcome, online. Anything less accumulates technical debt in the form of undetected regressions.

Latest 2026 landscape

Benchmarks worth tracking: tau-bench, tau2-bench, SWE-Bench, AgentBench, LongMemEval, LoCoMo.

End-to-end pipeline demo

Prompts that work

# 1) Strict trajectory match (reference) — bloqueia qualquer desvio de ordem/argumento
from agentevals import create_trajectory_match_evaluator
strict = create_trajectory_match_evaluator(mode="strict")
strict(inputs=case.input, outputs=agent_run, reference_trajectory=case.golden)

# 2) Tool-selection LLM judge (reference-free) — quando não dá pra manter golden
judge_prompt = """
You are grading an agent's tool selection.
For each tool call, decide:
  (a) Was the right tool picked for the user's goal?
  (b) Are the arguments schema-valid AND semantically correct?
  (c) Did the agent recover from a failed tool call, or did it loop?
Return JSON: { "tool_selection": 0-1, "args": 0-1, "recovery": 0-1, "reasoning": "..." }
Trajectory: {trajectory}
User goal: {goal}
"""

# 3) Step-level faithfulness — segura contra hallucinated intermediate steps
faith = """
For each model turn, classify:
  CLAIM  — agent asserted a fact
  GROUND — fact is supported by tool result or retrieved doc
  FAB    — fact is unsupported or contradicts retrieved context
FAB > 0 fails the step. Report counts and the offending turn IDs.
"""

Benchmarks that hold

AgentAssay (arXiv 2603.02601): 78–100% cost reduction at equivalent statistical guarantees via SPRT; 86% behavioral detection power where binary pass/fail has 0%. Total: $227 across 5 models, 3 scenarios, 7,605 trials.

Tab 02 · Structured, Long-Term Memory with Automatic Consolidation

A context window is short-term and re-paid every turn. Memory is persistent, has its own storage tier, evolution rules, and a forgetting policy. The two are complements, not substitutes.

Why it matters

LongMemEval showed 1M-token models drop 30%+ accuracy when forced to recall across sessions without a memory layer. The dream is hybrid: structured graph for relations, vector store for surface recall, the agent itself paging between them — and a sleep-time process that consolidates the way the hippocampus does.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) Single-pass ADD-only extraction (Mem0 v3 idiom) — one LLM call, no UPDATE/DELETE
extract_prompt = """
Read this conversation turn. Emit only NEW facts about the user
that are not already in the existing memory store. Skip generic chit-chat.
Skip things the user already corrected this turn.
Each fact: { "subject": str, "predicate": str, "object": str, "confidence": 0-1, "ts": ISO8601 }
Existing memory (for dedup awareness): {existing}
Conversation: {turn}
"""

# 2) Conflict resolution — when new info contradicts old, stamp the old invalid, keep history
conflict_prompt = """
New fact: {new_fact}
Candidate conflicting edges: {candidates}
Decide:
  REPLACE — new supersedes (mark old invalid_at=now, add new with valid_at=now)
  COEXIST — both can be true (e.g., user has two roles over time)
  REJECT — new is hallucinated; ignore
Return JSON with decision, reason, and updated edges.
"""

# 3) Sleep-time reflection (Anthropic Dreaming / Letta sleep-time agent idiom)
reflect = """
Recent episodic trace (last 24h): {episodes}
Long-term store: {store}
Produce 0–5 NEW high-order insights that:
  (a) cross multiple episodes,
  (b) are not yet in the store,
  (c) are at a higher abstraction than the events themselves.
Do not summarize. Do not repeat. Output JSON array of insights.
"""

# 4) Recency-decay reranking (production default)
final_score = 0.7 * relevance + 0.3 * exp(-lambda * days_since)
#  lambda ≈ ln(2) / 14d → 14-day half-life, tune from production retrieval

Benchmarks that hold

Production target: HOT tier <2K tokens (current session) · WARM 2K–8K (prefs+recent decisions) · COLD unlimited vector-indexed. Recency default: 14-day half-life.

Tab 03 · Agent Observability, Guardrails, and Real-Time Interruption

Tracing tells you what happened. Eval tells you if it was good. Guardrails stop the bad from reaching the user — at sub-200ms, with graduated interventions, ideally before the tool call.

Why it matters

Agents autonomously call tools, send email, write rows. Enterprises won't deploy them without runtime enforcement. A keyword filter is too blunt. A post-hoc audit is too late. The new floor: streaming interception, intent modeling, and policy-as-code with hardware-rooted attestation behind it.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) NeMo Colang — jailbreak + content safety rail
# config.yml
rails:
  input:
    flows:
      - content safety check input $model=llama-3.1-nemoguard-8b-content-safety
      - self check jailbreak
  output:
    flows:
      - content safety check output $model=llama-3.1-nemoguard-8b-content-safety
      - self check hallucination
  execution:
    flows:
      - check tool call $allowed_tools=['search','create_ticket']

# 2) GAAT Governance Telemetry Schema (OPA-compatible)
package agent.guardrails

deny[msg] {
  input.tool_call.name == "send_email"
  not input.user.consent.email_outbound
  msg := sprintf("tool %v requires email_outbound consent", [input.tool_call.name])
}

warn[msg] {
  input.tool_call.arguments.pii_detected == true
  msg := "PII in tool arguments — redact before send"
}

# 3) Graduated intervention (L0–L4)
intervene(severity, action) = {"L0":"allow","L1":"alert","L2":"flag","L3":"redirect","L4":"quarantine"}[severity]

Benchmarks that hold

Deterministic checks <50ms · LLM-judge 1–5s on cloud Turing models · budget the synchronous chain <300ms end-to-end. Defense in depth beats any single layer.

Tab 04 · Tool Integration & Dynamic API Orchestration

Function calling is a runtime convention inside one LLM API. MCP is a transport-level protocol that lives outside it. In 2026 you ship one MCP server and every compatible client can use it. That's the unlock.

Why it matters

Connecting agents to real APIs is still messy. The 2026-07-28 MCP spec — stateless core, MRTR, async tasks, formal extension system — pushed the protocol from a developer-facing tool-calling convention toward enterprise-grade production infrastructure. Auto-generation from OpenAPI collapsed the onboarding tax.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) Tool description generation — LLM writes the JSON-schema description a model will see
describe_tool = """
Given this OpenAPI operation, produce an MCP tool definition:
  - name: snake_case, action_object
  - description: 1–2 sentences stating WHEN to call (not what it returns)
  - inputSchema: JSON Schema, required fields marked
  - outputSchema: shape of the success response
  - examples: 2 input/output pairs
OpenAPI op: {op}
"""

# 2) Argument schema validation — never trust the LLM
from jsonschema import validate, ValidationError
try:
    validate(instance=tool_call.arguments, schema=tool.inputSchema)
except ValidationError as e:
    # refuse + emit structured error so agent can retry with corrected args
    return {"error": "schema_violation", "path": e.absolute_path, "msg": e.message}

# 3) Progressive discovery — defer tool defs into context
search_tools_prompt = """
Available MCP servers: {server_list}
For this user request, which 1–2 servers are needed? Reply with [server_name].
Do not list tools — defs load on demand.
"""

# 4) Tool-result PII redaction (the under-appreciated surface)
policy = (GuardrailPolicyBuilder()
    .RedactPii()             # order 20 — runs on input, output, AND tool results
    .GuardToolResults()      # wires the tool-result interception lane
    .Build())

Benchmarks that hold

PII has 4 entry points — training data, user inputs, retrieved context, model outputs. The fifth is tool-call arguments, which execute before any output guardrail sees them. Cover all five or you have a hole.

Tab 05 · Self-Improving Agent Pipelines (Auto Fine-Tuning from Experience)

Human labels are expensive. Static prompts plateau. The agent should learn from production — failure → preference pair → DPO → canary → ship or roll back, all without a human in the loop.

Why it matters

Production agents accumulate failure modes faster than anyone can label them. The 2026 stack: SFT for instruction following, DPO/SimPO for general preferences, GRPO/DAPO with verifiable rewards for reasoning. The closed loop: capture failures → synthesize preference pairs → fine-tune a smaller model → canary it → beat the old one on cost and quality, or revert.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) Best-of-N task prompt — rejection sampling for DPO pairs
bos_prompt = """
You are attempting: {task}.
Run the test suite when done. Return:
  status: "pass" | "fail"
  diff:   unified diff of your changes
  notes:  one-paragraph rationale
Do not explain before attempting. Just attempt.
"""

# 2) DPO preference pair generator — deterministic judge (no LLM cost)
def to_pair(trace, test_runner):
    won  = next((t for t in trace.attempts if test_runner(t) == "pass"), None)
    lost = next((t for t in trace.attempts if test_runner(t) == "fail"), None)
    if won and lost:
        return {"prompt": trace.prompt, "chosen": won.code, "rejected": lost.code}
    return None

# 3) Automatic curriculum (Voyager-style — what skill to learn next?)
curriculum = """
Current skill library (embeddings): {skill_descs}
Recent failures (last 24h): {failures}
Open unexplored territory in: {domain}
Propose the next 3 skills to acquire. Each skill: name, description (1 sentence), why now.
Do not duplicate. Do not propose skills already in the library.
"""

# 4) Self-verifier (Voyager iterative prompting)
verify = """
The skill output was: {output}
Environment state: {state}
Did the skill achieve its stated goal? Reply:
  VERIFIED — clear evidence in state
  REJECTED — no evidence, here's why
  AMBIGUOUS — partial; what to check next
"""

Benchmarks that hold

Voyager: removing the skill library plateaued late runs; removing self-verification dropped performance 73%. Skill library transfer lifted AutoGPT zero-shot from 0/3 to 1–2/3.

Tab 06 · Multi-Agent Collaboration & Debate with Provenance

Three models agreed. It was still wrong. An ICML 2025 study found that when two models both err, they converge on the same wrong answer 60% of the time. The fix isn't more votes — it's structured debate with provenance.

Why it matters

Cross-model agreement is correlated evidence, not confirmation. Models share training data, architectures, post-training recipes. The 2026 audit: agreement-to-correctness Spearman ρ of 0.20–0.59 — positive but weak. The most agreement-prone model had the least trustworthy agreement signal. Real reconciliation is a designed process, not a vote count.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) Independent briefs (kill shared-input bias — different words, different framing)
def brief(question, framing):
    return f"""You are a {framing}. Answer ONLY this question, in 200 words.
Do not mention other reviewers. Do not hedge. State your position and the top 2 pieces
of evidence you would cite. Question: {question}"""

# 2) Forced refutation (kill agreeableness)
refute = """The other reviewers currently lean toward: "{emerging_answer}".
Make the STRONGEST possible case that this answer is WRONG.
Cite specific evidence. Do NOT play devil's advocate — only argue if you genuinely believe it.
If you cannot find a real flaw, say "I cannot refute this on present evidence."
"""

# 3) GAVEL Evidence Contract (atomic subclaim ↔ evidence unit)
claim = {
  "id": "C1",
  "text": "The acquisition closed on 2025-04-12",
  "evidence": [
    {"type": "doc", "id": "sec-filing-2025-Q2.pdf#p14", "span": "...closed April 12, 2025..."},
    {"type": "doc", "id": "press-release-2025-04-12.html#body", "span": "...today announced the closing..."}
  ]
}

# 4) Minority Sentinel fingerprint (LightGBM input)
fingerprint = {
  "dynamics":  [agreement_velocity, persistence, refutation_strength]*3 + [...],   # 10 dims
  "voting":    [majority_size, dissent_persistence, refuter_count, vote_flip],     # 4 dims
  "semantic":  [novel_evidence_introduced, claim_grounding, citation_density, ...]  # 8 dims
}

Benchmarks that hold

Self-consistency (one model, vote) still works for checkable answers (+17.9pp on GSM8K). But cross-model consensus on open-ended judgment is unproven — that's a different claim with weaker evidence.

Tab 07 · Privacy-Preserving Local-First Agents

The most useful agents need intimate data. The most intimate data is the one you can't send to a third party. Local-first isn't a preference in 2026 — it's a regulatory and trust floor.

Why it matters

Cloud agents leak. Always. Even with the best contract. The 2026 answer: on-device processing by default, confidential compute for the heavy lifts, no raw data ever leaving the user's machine or its attested enclave. Federighi's line: privacy is non-negotiable, verifiable by outside experts.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) ADAPT dynamic privacy-preserving delegation (local decides what to ship)
delegate_prompt = """
Local user context: {context}
Remote task: {task}
Decide:
  LOCAL_ONLY      — task can be answered from local context, do not call remote
  PRIVACY_SAFE    — sanitize context, then call remote with the sanitized prompt
  FULL_CONTEXT    — task needs full context; call remote but only over attested PCC
For each decision emit: { "mode": ..., "sanitized_prompt": ..., "why": "..." }
"""

# 2) iClaw scaffolding for the Apple Foundation Model (small-model constraints)
instructions = InstructionsBuilder() \\
    .limit_tool_choice(max_tools=2-3)             # AFM gets distracted past that
    .set_maximum_response_tokens(256)             # keep responses predictable
    .temperature(0.3)                              # diverse but constrained
    .prewarm()                                     # reduce latency, cache prefix
    .build()

# 3) Multi-Agent Firewall local PII redaction (T10 config, best F1)
firewall_config = {
  "interceptors": ["browser", "mitm_proxy"],
  "agents": [
    {"role":"deterministic_pii", "model":"regex+ner", "lane":"sync"},
    {"role":"semantic_pii",      "model":"llm-judge", "lane":"async"},
    {"role":"code_leakage",      "model":"ast_diff",   "lane":"sync"},
  ],
  "policy": "block_on_high_confidence, redact_on_medium"
}

# 4) CryptoKit + Secure Enclave for credentials
let key = SecKeyCreateRandomKey(...)             # never leaves enclave
let att = SecKeyCreateAttestation(key, ...)      # prove enclave-bound

Benchmarks that hold

Hybrid is the production standard: local for data-sensitive + high-frequency + pre-processing; cloud frontier for complex reasoning, long context, MCP-compatible local-model endpoints. Pure-local is not yet recommended for most production requirements in 2026.

Tab 08 · Cost-Optimisation & Inference Routing for Agent Loops

Frontier models cost 20–100× more than their small-model cousins. The cheapest way to cut an AI bill without cutting capability is to stop paying frontier prices for easy requests.

Why it matters

40–70% of production queries can be served by the cheap model with no detectable quality loss. 20–30% require frontier. The middle 10–20% benefits from cascade verification. The 2026 stack: semantic cache → difficulty router → cheap → escalate on confidence → cost attribution per step → drift monitor. Done right, you keep 95% of frontier quality at 15–30% of the bill.

Latest 2026 landscape

End-to-end pipeline demo

Prompts that work

# 1) Difficulty classifier (the cheap-LLM-as-router)
classify = """
Read this user query and rate its difficulty on 3 axes (0=trivial, 10=hard):
  reasoning      — multi-step logic required?
  tool_count     — how many distinct tools will the agent need?
  context_depth  — needs deep retrieval / long doc reasoning?
Reply JSON: {"reasoning": int, "tool_count": int, "context_depth": int, "tier": "nano|mid|frontier"}
Query: {query}
"""

# 2) Conformal threshold setter (distribution-free, no calibration drift)
def conformal_threshold(scores, alpha=0.1):
    # 90% of cheap-model outputs that score above this are actually correct
    return np.quantile(scores, 1 - alpha)

# 3) Cascade trigger (cheaper than re-querying)
trigger = cheap_response if confidence(conformal, cheap.score) > tau else escalate(cheap, mid)

# 4) SLO tier → model pool mapping (production default)
TIERS = {
  "realtime":  {"latency_ms": 500,  "cost_per_req": 0.001,  "pool": ["haiku-4.5", "gpt-5.4-nano"]},
  "standard":  {"latency_ms": 3000, "cost_per_req": 0.01,   "pool": ["sonnet-4", "gpt-5"]},
  "premium":   {"latency_ms": 30000,"cost_per_req": 0.10,   "pool": ["opus-4", "o3"]},
  "batch":     {"latency_ms": None, "cost_per_req": 0.0001, "pool": ["local-llama", "deepseek-r1"]}
}

Benchmarks that hold

Tier price gap in 2026: 100×. Claude Opus 4.6 $5/$25 vs GPT-5.4 nano $0.05/$0.40 per 1M. Route 70% cheap / 20% mid / 10% frontier → $15/M → $2.10/M blended.

Benchmarking · What to build first, second, and not at all

Cross-cutting comparison: maturity, leverage, time-to-value, lock-in, and the order you should ship them.

Maturity × leverage matrix

The shipping order, ranked by ROI

1. Tab 8 — Cost & Routing. Three days of work, recovers 40–70% of the bill. The money funds everything else. Start with semantic cache + static rules, then RouteLLM.
2. Tab 1 — Agent Eval. You cannot safely ship anything else without it. Two weeks. Start with DeepEval in CI on golden-50.
3. Tab 3 — Observability + Guardrails. Phoenix + Langfuse for the trace story, NeMo + Lakera for the runtime story. Two to three weeks. Required for enterprise.
4. Tab 4 — Tool Orchestration. Stand up MCP server per API, wire AutoMCP. Pays off the moment you have a second tool.
5. Tab 2 — Long-Term Memory. Add Mem0 + Zep hybrid the moment your agent has cross-session requirements. Two to four weeks.
6. Tab 5 — Self-Improving Agents. Pilot Online DPO only after eval infrastructure catches the regressions. Then iterate.
7. Tab 7 — Privacy / Local-First. Apple-native if iOS/macOS; AgenTEE on Arm for cross-platform confidential compute. Required for health, legal, finance.
8. Tab 6 — Multi-Agent Debate. Niche. Add GAVEL + Minority Sentinel only when stakes justify the cost — legal review, medical, financial analysis.

Concrete dollar math (illustrative)

The benchmark that matters most

It's not the one in the paper. It's the one in your production: CPSO — cost per successful outcome. Not tokens. Not traces. Successful outcomes — generated, sent, accepted by the business without rework. That single number tells you whether every other optimization is real or theater.