Nathaniel Okenwa

Performant Architecture for a Fast Gen AI User Experience

Stop blaming slow models for your AI app's latency. Your architecture is the real problem.

Performant Architecture for a Fast Gen AI User Experience
#1about 2 minutes

Building a real-time translator inspired by sci-fi

The Babel fish from "Hitchhiker's Guide to the Galaxy" serves as the inspiration for a real-time audio translation project.

#2about 4 minutes

Analyzing the latency of a basic AI architecture

A demonstration of the initial 2019 architecture using GCloud reveals a significant latency of over ten seconds for a simple translation.

#3about 2 minutes

Reducing latency by upgrading the AI service stack

Switching to modern, specialized APIs like Deepgram and 11 Labs significantly cuts the total processing time from twelve to five seconds.

#4about 2 minutes

Implementing streaming to reduce response wait times

Adopting a streaming approach provides a major performance boost, but a naive implementation results in chaotic and low-quality audio output.

#5about 2 minutes

Using chunking to balance streaming speed and quality

Chunking data based on sentence punctuation controls the streaming waterfall, improving the quality of generated audio without sacrificing speed.

#6about 6 minutes

Eliminating network latency with local and edge models

Running a smaller, local AI model like Whisper on the edge eliminates cross-continental network latency and provides near-instantaneous results.

#7about 3 minutes

Using caching to serve pre-generated AI responses

Implementing caching, from simple request matching to semantic search with vector databases, avoids redundant generation and speeds up common queries.

#8about 2 minutes

Optimizing prompts and user experience for speed

Fine-tuning performance involves optimizing prompts to generate fewer tokens and improving perceived speed with clear loading states for the user.

#9about 2 minutes

Summary of key performance optimization techniques

A final recap covers the essential strategies for building fast Gen AI experiences, including streaming, edge computing, caching, and prompt optimization.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Using Chrome's built-in AI for on-device features
06:44 MIN

Using Chrome's built-in AI for on-device features

Devs vs. Marketers, COBOL and Copilot, Make Live Coding Easy and more - The Best of LIVE 2025 - Part 3

Using AI to overcome challenges in systems programming
02:49 MIN

Using AI to overcome challenges in systems programming

AI in the Open and in Browsers - Tarek Ziadé

Why corporate AI adoption lags behind the hype
03:28 MIN

Why corporate AI adoption lags behind the hype

What 2025 Taught Us: A Year-End Special with Hung Lee

Increasing the value of talk recordings post-event
04:57 MIN

Increasing the value of talk recordings post-event

Cat Herding with Lions and Tigers - Christian Heilmann

Exploring the role and ethics of AI in gaming
14:06 MIN

Exploring the role and ethics of AI in gaming

Devs vs. Marketers, COBOL and Copilot, Make Live Coding Easy and more - The Best of LIVE 2025 - Part 3

How AI is changing the freelance developer experience
09:10 MIN

How AI is changing the freelance developer experience

WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More

Integrating AI into Firefox while respecting user privacy
08:40 MIN

Integrating AI into Firefox while respecting user privacy

AI in the Open and in Browsers - Tarek Ziadé

The shift to on-device AI models in smartphones
00:48 MIN

The shift to on-device AI models in smartphones

Fake or News: Coding on a Phone, Emotional Support Toasters, ChatGPT Weddings and more - Anselm Hannemann

Featured Partners

Related Videos

See all videos
Generate AI in the Browser with Chrome AI - Raymond Camden36:41

Generate AI in the Browser with Chrome AI - Raymond Camden

Raymond Camden

Prompt API & WebNN: The AI Revolution Right in Your Browser30:36

Prompt API & WebNN: The AI Revolution Right in Your Browser

Christian Liebel

How AI Models Get Smarter35:16

How AI Models Get Smarter

Ankit Patel

Privacy-first in-browser Generative AI web apps: offline-ready, future-proof, standards-based30:13

Privacy-first in-browser Generative AI web apps: offline-ready, future-proof, standards-based

Maxim Salnikov

Chatbots are going to destroy infrastructures and your cloud bills21:21

Chatbots are going to destroy infrastructures and your cloud bills

Stanislas Girard

Generative AI power on the web: making web apps smarter with WebGPU and WebNN32:27

Generative AI power on the web: making web apps smarter with WebGPU and WebNN

Christian Liebel

Make it simple, using generative AI to accelerate learning26:54

Make it simple, using generative AI to accelerate learning

Duan Lightfoot

Supercharge your cloud-native applications with Generative AI32:13

Supercharge your cloud-native applications with Generative AI

Cedric Clyburn

Related Articles

View all articles
DC
Daniel Cranney
Dev Digest 198: 30 years of JS, In-Browser AI, How Attackers Abuse GenAI
Inside last week’s Dev Digest 198 . 🎂 30 years of JavaScript ⏰ How long is a JavaScript second 💻 Clean code in Angular 🤦‍♂️ AI makes different mistakes than humans 👨‍💻 In-browser and offline AI 🟠 Undocumented Hacker News features 🐋 DeepSeek censored...
Dev Digest 198: 30 years of JS, In-Browser AI, How Attackers Abuse GenAI

From learning to earning

Jobs that call for the skills explored in this talk.