The 5-layer voice-AI stack: listen, think, speak, clone, ship. Each layer is one or two tools, and the whole pipeline ships in a single weekend. No call center. No IVR hell. Just a voice that sounds like you.
Every voice-AI app is a sequence of five jobs. Listen (turn audio into text), think (run an LLM on the text), speak (turn the answer into audio), clone (make it sound like you), ship (wire it to a phone).
Whisper is fast, multilingual, and runs at the edge via MLX on Apple Silicon. Below 200ms latency for a 5-second utterance, which is fast enough for a phone conversation.
A voice conversation needs sub-second response time. Claude Haiku and GPT-4o-mini both fit. The smart model runs offline for context, the small model runs the live loop.
ElevenLabs sets the bar in 2026 — natural inflection, emotional range, multilingual. OpenAI TTS is the budget option for non-critical paths (status updates, scheduled readouts).
ElevenLabs Voice Design and Resemble AI both clone from 30 seconds of clean audio. The result is "sounds like you, but more articulate and never tired." Use it sparingly — voice clones that pass 100% of the time also feel uncanny.
Vapi gives you a phone number + the orchestration layer in one product. LiveKit is the lower-level WebRTC for custom UIs. Twilio is the carrier. Pick the one that matches the build cost you can stomach.
The 2024-25 wave of "AI call center" products proved you can ship a voice agent. They also proved that 90% of them sound like robots and solve problems nobody has. The win is at the personal scale: a voice that knows you, talks like you, runs on your number.
The voice agent is just an LLM with a voice attached. The system prompt is where you encode the persona, the context, the guardrails. The template that works.
# Voice agent system prompt · ~400 words · for Claude Haiku 4.5 ROLE You are [your name]'s personal voice agent. You answer the phone when [your name] can't. You speak in [your name]'s voice, with [your name]'s cadence and [your name]'s vocabulary. You are not a generic assistant. WHO YOU TALK TO - Clients calling about active projects - Vendors calling about quotes / invoices / scheduling - Friends calling to make plans - Strangers calling the wrong number (be brief, redirect) CONTEXT YOU HAVE - Today's calendar (next 8 hours) - Last 5 emails sent to this caller (if email matched) - 3 most recent project notes (if relevant) - [Your name]'s preferences (no calls before 9am, no calls after 8pm, Friday afternoons are deep-work blocks) VOICE RULES - Short answers. 1-3 sentences max per turn. This is a phone, not email. - No filler ("Sure!", "Of course!", "Great question!"). Start with the substance. - If you don't know, say so and offer to take a message. - Never invent facts about [your name]'s schedule, projects, or opinions. - If asked something outside your context, take a message. GUARDRAILS - Never make commitments on [your name]'s behalf (no scheduling, no agreeing to deliverables, no price quotes). - Never share contact info, addresses, or financial details. - Never engage with harassment — politely end the call and flag. - If the caller is in crisis, route to the human immediately. WHEN TO HAND OFF - Caller asks for [your name] by name → "They're unavailable right now, want me to take a message or have them call back?" - Caller is emotional / upset → "[Your name] is going to call you back personally. When works for you?" - Caller asks something the context doesn't cover → take a detailed message, including name, callback number, and a 1-sentence summary. TONE Warm. Direct. Not performative. The way a smart friend would answer your phone if you were in the shower.
The worked example: a real build, real tools, real timestamps. Saturday morning to Sunday afternoon, one voice agent on a Vapi number.
Personal voice agent. Answers the phone when I'm unavailable. Cloned voice, calendar context, system prompt above. Vapi number, deployed to a single Twilio DID. Budget: $0 in build cost, $20/mo in runtime.
ElevenLabs clone (20 min) → Vapi assistant config (90 min) → Twilio number port (10 min) → calendar + email context (3 hours) → 8 test calls (45 min) → deploy (15 min). Total: ~6 hours across the weekend.
# Real build, timestamped Saturday 9:00 — ElevenLabs voice clone Record 90 seconds of clean audio (mic: SM7B, room: closet) ElevenLabs → Instant Voice Clone → wait 60s Test: "What's the weather in Zagreb tomorrow?" in the cloned voice PASS — sounds like me, but more articulate Saturday 10:00 — Vapi assistant config vapi.ai → New Assistant → paste system prompt Model: claude-haiku-4-5 (lowest latency tier) Voice: my ElevenLabs clone ID First message: "Hey, this is Kai's line. He's heads-down right now — want me to take a message, or can I help with something?" Max duration: 4 min (longer than that, take a message) End-call function: log to /tmp/calls/$(date +%s).json Saturday 11:30 — Twilio number Twilio → Buy a number ($1/mo, local DID) Vapi → Phone Numbers → Import Twilio → paste SID + token Vapi → Route inbound → my assistant Test call from my cell: PASS, agent answers in my voice Saturday 14:00 — Calendar + email context Vapi → Custom Tools → tool_calendar_next_8h (Google Calendar API) Vapi → Custom Tools → tool_recent_emails_to (Gmail API, 5 most recent) Update system prompt: "When asked about schedule, call tool_calendar_next_8h" Test: "Am I free at 3pm Tuesday?" → agent calls tool, answers "Yes, free." PASS Sunday 10:00 — Test matrix (8 calls) Call 1: "Is Kai free Thursday?" → correctly checks calendar → PASS Call 2: "What's the address for the meeting?" → "I'll have him text you" → PASS Call 3: "Can you quote me a price for X?" → "Kai will email you back" → PASS Call 4: hostile caller → polite end, flag → PASS Call 5: caller asks for "Kajica" by name → "He's unavailable, want a msg?" → PASS Call 6: silent caller (3s silence) → "Are you there? — I'll take a msg." → PASS Call 7: wrong number → "This is Kai's line, who were you trying to reach?" → PASS Call 8: friend calling to make plans → "I'm not in Kai's calendar, want a msg?" → PASS Sunday 14:00 — Deploy + monitoring Flip Twilio routing: personal cell → Vapi during focus blocks Daily cron: email me the previous day's call log at 8am Weekly cron: review missed calls, update system prompt if needed Build cost: $0. Runtime: $19/mo (Vapi) + $1/mo (Twilio) = $20/mo. Coverage: 6 focus blocks per week × 4 hours = 24 hours of autonomous phone handling. Reclaimed: ~24 hours × 4 (interruption cost) = ~96 hours of deep work per month.
The five jobs are universal. The 12 tools are 2026. Swap tools as the field moves; the layer model holds.
Nine stacks. One philosophy. Zero babysitting.