Cost-Optimisation & Inference Routing for Agent Loops

# policy.yaml — inference router, 2026-08-08
# Target: −78% cost vs GPT-5-only, quality ≥ 0.81 on golden-200
router:
  name: "cost-quality-pareto-v3"
  estimator: "difficulty_logreg_v2"      # trained on 38k labelled steps
  cache_first: true
  batch_window_ms: 50
  speculative: true
  judge_sample_rate: 0.05                  # 5% of steps scored by Prometheus
  budget_per_task_usd: 0.45                # hard ceiling, alarm at 80%

models:
  - id: "gpt-5-nano"  cost_in: 0.05  cost_out: 0.40  tier: "easy"
  - id: "gpt-5-mini"  cost_in: 0.25  cost_out: 2.00  tier: "med"
  - id: "gpt-5"       cost_in: 2.50  cost_out: 10.00 tier: "hard"

cache:
  exact:       enabled: true  backend: "redis"           ttl: 86400
  semantic:    enabled: true  backend: "qdrant"  sim: 0.95  ttl: 3600
  episodic:    enabled: true  backend: "redis+sessionid"  ttl: 300

rules:                                                  # evaluated top-to-bottom, first match wins
  - when: "cache.exact.match == true"             route: "cache.hit"
  - when: "cache.semantic.sim >= 0.95"          route: "cache.hit"
  - when: "step.type == 'verify' and tokens < 200"  route: "gpt-5-nano"
  - when: "difficulty <= 0.35"                     route: "gpt-5-nano"
  - when: "difficulty <= 0.70"                     route: "gpt-5-mini"
  - when: "step.type == 'plan'"                     route: "gpt-5-mini"   # plan always verified by a second pass
  - when: "difficulty > 0.70 or self_eval.confidence < 0.6"  route: "gpt-5"
  - fallback: true                                route: "gpt-5-mini"

fallback:
  strikes: 3                                       # 3 low-confidence responses in a row → escalate one tier
  on_regression:  "rollback_to_previous_pareto_point"
  on_cache_poison: "purge+alert"

telemetry:
  log:  "postgresql://ledger/tasks"
  dash: "grafana://agent-cost/inference"
  alarm: "slack://oncall/agent-platform"

Cache hit decision code (Python, 18 lines)

async def cache_lookup(prompt, ctx, session_id, step_type):
    # 1. exact
    exact_key = sha256(json.dumps({"p":prompt, "m":ctx.model, "s":ctx.system}, sort_keys=True))
    hit = await redis.get(exact_key)
    if hit: return CacheHit("exact", hit)

    # 2. semantic — reject sensitive tags
    if any(tag in prompt.lower() for tag in ["today", "now", "price", "weather"]): return None
    emb = embed(prompt)
    near = await qdrant.search(emb, top_k=1, score_threshold=0.95)
    if near: return CacheHit("semantic", near[0].payload["response"])

    # 3. episodic (only for tool-call re-attempts)
    if step_type == "act":
        e_key = sha256(f"{session_id}:{ctx.last_tool}:{ctx.last_args}")
        hit = await redis.get(e_key, ex=300)
        if hit: return CacheHit("episodic", hit)

    return None

Observation: one frontier call at step 8 is 51% of the bill. The whole task lands at $0.013 — well under the $0.45 ceiling. 9 of 12 steps ran on nano or cache. A frontier-only run would have cost $0.043 (3.2×) at the same quality. Lesson: it's not the 71% nano share that wins — it's the 5% frontier share that you don't have to pay for, because the difficulty estimator correctly identifies the synthesis step.