Build an AI voice assistant
Voice agents chain speech-to-text, a language model and text-to-speech in real time. Latency is the main challenge, so use a framework built for streaming audio.
Recommended stack
Model
- OpenAI GPT-6 Luna: ~$0.10 in / $0.50 out per 1M tokens (verified 2026-10-14)
- OpenAI speech-to-text API: Usage-based (not verified - check site)
- ElevenLabs: Free tier + paid plans (not verified - check site)
Framework
- Pipecat: Free, open-source voice-agent framework
Hosting
No-code alternative
- Voiceflow: Free tier + paid (not verified - check site)
Rough monthly cost
$10–50/mo (speech minutes add up fast)ApproximateDifficulty
AdvancedSome links may be affiliate links. Disclosure
Why this stack
- Pipecat is an open-source framework for real-time voice and multimodal agents.
- OpenAI's speech-to-text and ElevenLabs' voices cover the audio ends; a fast model keeps turn latency low.
- Fly.io or Railway host long-running audio processes; serverless functions usually can't.
Step-by-step
- Run the Pipecat quickstart locally with your microphone.
- Swap in your own system prompt and a fast model.
- Measure end-to-end latency and keep replies short.
- Only then connect phone or Discord voice.
Common pitfalls
- Speech minutes are the main cost; set usage caps.
- Tell callers they are talking to an AI and follow local recording-consent laws.
- Handle interruptions (barge-in) or it will feel robotic.