Design a Meeting Summarization System
A meeting summarization system captures a meeting, creates a transcript that identifies who said what, and turns the conversation into useful notes. These usually include a short summary, key decisions, and action items with owners.
Some products join calls as a bot, some capture audio on the user's device, and others process uploaded recordings. Many also provide live captions during the meeting and deliver a finished summary a few minutes after it ends.
Popular Examples: Zoom AI Companion, Microsoft Teams Copilot, Google Meet's "Take notes for me," Otter.ai, Fireflies, and Granola
Behind these features are two different AI workloads. Speech recognition processes every second of audio, so it needs substantial compute capacity. Summarization places more emphasis on accuracy: assigning a task to the wrong person or reporting a decision that was never made can quickly undermine trust. Both workloads also handle sensitive information from private conversations.
In this chapter, we'll design a meeting summarization system end to end, from capturing audio and generating live notes to searching past meetings and evaluating summary quality. We'll start with the requirements and architecture, then look more closely at speaker attribution, live and post-meeting processing, and long recordings.