In this coding challenge, I build a conversational voice chatbot entirely in the browser with p5.js. I combine three pieces: speech-to-text with OpenAI's Whisper model, text-to-speech with Kokoro TTS, and a "brain" for the bot. I also explore the transformers.js pipeline API and the Web Audio API. For the bot's brain, I start with a simple ELIZA-style therapist, then incorporate a RiveScript number-guessing game, and finally a local LLM. Code: https://thecodingtrain.com/challenges/188-voice-chatbot

🚀 Watch this video ad-free on Nebula https://nebula.tv/videos/codingtrain-coding-challenge-188-voice-chatbot

p5.js Web Editor Sketches:
🕹️ LLM Chatbot: https://editor.p5js.org/codingtrain/sketches/RHhT9I4Nm
🕹️ Number Guessing Bot: https://editor.p5js.org/codingtrain/sketches/AJw7zMN9q
🕹️ Therapy Bot: https://editor.p5js.org/codingtrain/sketches/37LFEPUVV
🕹️ Model Loading Bars: https://editor.p5js.org/codingtrain/sketches/E9Ob3x8eJ
🕹️ Waveform of Recording: https://editor.p5js.org/codingtrain/sketches/cck49wDub
🕹️ Real Time Waveform: https://editor.p5js.org/codingtrain/sketches/aaRIT-x6a

🎥 Previous: https://youtu.be/g3-PXyF8U70?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH
🎥 All: https://www.youtube.com/playlist?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH

References:
📓 p5.2 Reference: https://beta.p5js.org
📓 Introducing Whisper: https://cdn.openai.com/papers/whisper.pdf
📓 Model Cards for Model Reporting: https://arxiv.org/abs/1810.03993
📓 Open Neural Network Exchange: https://onnx.ai
📓 Onnx-community Whisper-tiny.en model: https://huggingface.co/onnx-community/whisper-tiny.en
📓 Xenova: https://github.com/xenova
📓 Transformers.js: https://huggingface.co/docs/transformers.js/installation
📓 Announcing the new p5.sound.js library!: https://medium.com/processing-foundation/announcing-the-new-p5-sound-js-library-42efc154bed0
📓 getUserMedia() documentation: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia
📓 MediaRecorder() documentation: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder
📓 Kokoro Repo: https://github.com/hexgrad/kokoro
📓 KokoroTTS Model: https://huggingface.co/hexgrad/Kokoro-82M
📓 ELIZA: https://en.wikipedia.org/wiki/ELIZA
📓 Rivescript: https://www.rivescript.com
📓 SmolLM3: https://huggingface.co/HuggingFaceTB/SmolLM3-3B
📓 Running models on WebGPU: https://huggingface.co/docs/transformers.js/guides/webgpu
📓 Using quantized models (dtypes): https://huggingface.co/docs/transformers.js/v3.8.1/guides/dtypes

Videos:
🚂 https://youtu.be/0Ad5Frf8NBM
🚂 https://youtu.be/KR61bXsPlLU

Live Stream Archives:
🔴 https://www.youtube.com/watch?v=KRDJAHArqaw

Related Coding Challenges:
🚂 https://youtu.be/eGFJ8vugIWA
🚂 https://youtu.be/8Z9FRiW2Jlc
🚂 https://youtu.be/iFTgphKCP9U

Timestamps:
0:00:00 Hello!
0:00:35 Mapping out the pieces: speech-to-text, text-to-speech, and the brain
0:01:07 Thoughts on AI and creative exploration
0:02:44 Choosing the tools: Whisper and Kokoro TTS
0:04:06 Building a push-to-talk UI in p5.js
0:04:51 Finding models on Hugging Face with Transformers.js
0:05:36 About the Whisper model and model cards
0:06:55 Loading the Whisper pipeline in p5.js
0:09:04 Accessing the microphone with getUserMedia
0:10:44 Capturing audio with MediaRecorder
0:12:05 Processing audio chunks into a waveform
0:15:55 Speech-to-text working!
0:16:36 Building the chatbot brain (ELIZA-style therapist)
0:18:50 Setting up Kokoro TTS for text-to-speech
0:21:07 Playing synthesized audio with AudioBufferSource
0:23:41 Text-to-speech working!
0:25:32 Handling playback events
0:26:56 Swapping in a RiveScript number-guessing brain
0:31:22 Adding a language model (SmolLM2) as the brain
0:38:33 Final demo: the random number chatbot
0:39:03 Goodbye!

Editing by Mathieu Blanchette
Animations by Jason Heglund
Music from Epidemic Sound

🚂 Website: https://thecodingtrain.com/
👾 Share Your Creation! https://thecodingtrain.com/guides/passenger-showcase-guide
🚩 Suggest Topics: https://github.com/CodingTrain/Suggestion-Box
💡 GitHub: https://github.com/CodingTrain
💬 Discord: https://thecodingtrain.com/discord
💖 Membership: http://youtube.com/thecodingtrain/join
🛒 Store: https://standard.tv/codingtrain
🖋️ Twitter: https://twitter.com/thecodingtrain
📸 Instagram: https://www.instagram.com/the.coding.train/

🎥 https://www.youtube.com/playlist?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH
🎥 https://www.youtube.com/playlist?list=PLRqwX-V7Uu6Zy51Q-x9tMWIv9cueOFTFA

🔗 p5.js: https://p5js.org
🔗 p5.js Web Editor: https://editor.p5js.org/
🔗 Processing: https://processing.org

📄 Code of Conduct: https://github.com/CodingTrain/Code-of-Conduct

This description was auto-generated. If you see a problem, please open an issue: https://github.com/CodingTrain/thecodingtrain.com/issues/new

#texttospeech #speechtotext #chatbot #rivescript #llms #agents #ai #transformersjs #webaudioapi #javascript #p5js

The Coding Train

A step-by-step coding challenge building a voice chatbot inside a p5.js sketch using three components: speech-to-text (OpenAI Whisper via Transformers.js), a chatbot brain (ranging from simple Eliza-style pattern matching to RiveScript to a small LLM), and text-to-speech (Kokoro TTS). The tutorial covers Web Audio API for microphone capture, MediaRecorder for push-to-talk, audio buffer processing, WebGPU acceleration, and integrating HuggingFace models client-side in the browser. Multiple brain implementations are demonstrated, culminating in using SmallLM2 with a system prompt and conversation history.

Coding Challenge 188: Voice Chatbot