When you call an LLM API without streaming, the API waits until the entire response is generated, then sends it back in one shot. For short answers, this is fine. But when a model generates a 500-word response, the user stares at a blank screen for several seconds with no feedback. Streaming fixes this by sending tokens to the client as they are generated, so the response appears incrementally instead of all at once.

The second problem is even more fundamental. LLM APIs are stateless. Every API call is independent. The model has no memory of what you asked it two seconds ago. If you want a back-and-forth conversation, you need to build that yourself by sending the full conversation history with every request.

In this chapter, you will learn:

By the end, you will build a streaming chatbot that remembers conversations and gracefully handles context window limits.