We’re so excited for our friends at OpenAI, who launched their Realtime API today. This tutorial is built using this exciting API. The API opens up Speech to Speech (S2S) capabilities for their GPT-4o multimodal model, which supports direct audio input and output – avoiding translating back and forth from text with a speech-to-text (SST) or text-to-speech (TTS) step.
Chapters:
0:42 Prerequisites
1:52 Code starts here
3:01 Configuring the application
3:53 Handling the incoming call
4:38 Setting up our OpenAI WebSocket
6:35 Handling Twilio Media Stream events
7:43 Starting the application
8:13 Tunneling with ngrok
8:41 Configuring our Twilio phone number
9:18 Testing the application
This video will help you build an AI assistant using Twilio Voice and OpenAI’s Realtime API. Here’s what you’ll need to build it:
* A Twilio account – you can get one at https://twilio.com/try-twilio?utm_source=youtube&utm_medium=video&utm_term=twiliodevs&utm_campaign=twiliodevs-youtube&utm_content=openai-realtime-node
* A Twilio phone number – learn more about phone numbers here: https://www.twilio.com/docs/phone-numbers?utm_source=youtube&utm_medium=video&utm_term=twiliodevs&utm_campaign=twiliodevs-youtube&utm_content=openai-realtime-node
* An OpenAI API Key with Realtime API access (https://platform.openai.com/docs/overview)
Blog post: https://www.twilio.com/en-us/blog/voice-ai-assistant-openai-realtime-api-node?utm_source=youtube&utm_medium=video&utm_term=twiliodevs&utm_campaign=twiliodevs-youtube&utm_content=openai-realtime-node
Github repo: https://github.com/twilio-samples/speech-assistant-openai-realtime-api-node
Resources:
Twilio Media Streams: https://www.twilio.com/docs/voice/media-streams?utm_source=youtube&utm_medium=video&utm_term=twiliodevs&utm_campaign=twiliodevs-youtube&utm_content=openai-realtime-node
OpenAI GPT-4o: https://openai.com/index/hello-gpt-4o/
Ngrok: https://ngrok.com
CodeExchange example: https://www.twilio.com/code-exchange/ai-voice-assistant-openai-realtime-api?utm_source=youtube&utm_medium=video&utm_term=twiliodevs&utm_campaign=twiliodevs-youtube&utm_content=openai-realtime-node
source