Most AI applications spend more time waiting on other systems than running Python code. They wait for model APIs, vector databases, file stores, queues, webhooks, and internal services.

Async Python helps with that kind of waiting. It lets one Python process keep many network operations in progress at the same time, without creating one thread per request.

The important limitation is simple: async and await do not make one model call faster. A five-second model call is still a five-second model call. Async helps when you have several independent I/O-bound operations and can overlap the waiting time.

In this chapter, you will learn the async patterns that show up often in AI systems: async HTTP calls, concurrent model requests, structured concurrency, streaming responses, timeouts, cancellation, and concurrency limits.

Think of an application that needs three independent API responses. In regular synchronous code, it often does this:

If each request takes five seconds, the program waits about fifteen seconds total.

With async code, the program can send all three requests, then wait while the network and remote services do their work. The requests still take about five seconds each, but the waiting happens at the same time.

This is the main value of async in AI engineering. It does not remove latency. It helps you avoid waiting serially for independent operations.

Python's built-in async library is asyncio. You do not need much syntax to get started, but the details matter.

The Event Loop

The event loop is the scheduler for async code.

It runs a coroutine until that coroutine reaches an await. If the coroutine is waiting on I/O, such as an HTTP request, the event loop can run another coroutine in the meantime. When the I/O is ready, the original coroutine continues.

Most application code does not manage the event loop directly. You write async functions and use asyncio.run() at the edge of the program.

async def and await

Two keywords do most of the work.

Here is a small example that simulates API calls with asyncio.sleep:

This code is async, but it is still sequential. Each call is awaited before the next one starts, so the total time is about 0.2 + 0.3 + 0.1 seconds.

This is a common point of confusion: async syntax alone does not create concurrency. To overlap the calls, you must schedule them together. We will do that with asyncio.gather() shortly.

One important detail: await asyncio.sleep(delay) is non-blocking. It gives control back to the event loop. By contrast, time.sleep(delay) blocks the whole thread. Do not use time.sleep() inside async code.

asyncio.run()

asyncio.run() starts an async program from regular Python code. It creates an event loop, runs the top-level coroutine, and cleans up afterward.

asyncio.run(main())

This is the standard entry point for scripts and command-line programs. In notebooks, web frameworks, and async test runners, an event loop may already be running. In those environments, you normally await the coroutine directly or use the framework's async hooks instead of calling asyncio.run() again.

asyncio.sleep() is useful for examples. Real AI applications usually wait on HTTP calls.

The popular requests library is synchronous. If you call it inside async code, it blocks the event loop. For async HTTP, use a library such as httpx or aiohttp. This chapter uses httpx because its API is close to requests and it has a clean async client.

This example shows the main habits to build early:

This is where async starts to help.

asyncio.gather() takes several awaitables, runs them concurrently, and returns their results in the same order you passed them in.

Without gather, these calls would take about 0.9 seconds in sequence. With gather, they finish in about 0.4 seconds because the slowest call determines the total time.

Use this pattern when the work is independent. If request B depends on the result of request A, keep it sequential.

asyncio.gather() is still common, especially when you want ordered results. In Python 3.11 and newer, asyncio.TaskGroup is often a better fit when several tasks belong to one operation.

A task group gives child tasks a clear lifetime. Tasks start inside the async with block, and the block does not finish until the tasks finish. If one task fails, the task group cancels the remaining tasks and raises an exception group.

That behavior is useful when partial work should stop after the parent operation has failed.

Use gather(return_exceptions=True) when partial failure is acceptable and you want to inspect every result. Use TaskGroup when the tasks belong to the same unit of work and one failure should stop the rest.

Error Handling with return_exceptions

By default, if any coroutine inside gather() raises an exception, that exception is raised to the caller. You do not get a normal list of results.

That is sometimes correct. But if you are calling several providers or scoring many independent documents, one timeout may not make the whole batch unusable.

return_exceptions=True changes the behavior. Exceptions are returned as items in the result list, next to successful results:

Two calls succeed, and the failure is captured without losing the successful results. This is useful when partial answers are acceptable and you have a clear plan for handling failures.

Async programs need clear stop conditions. Without timeouts, one slow dependency can keep a request, job, or worker alive far longer than intended.

For individual operations, asyncio.timeout() is a simple built-in guard in Python 3.11 and newer:

Cancellation is the other side of the same idea. If a user closes the page, a request times out, or a parent task fails, your coroutine may be cancelled.

Most of the time, you should let cancellation propagate. Use try and finally only for cleanup:

Do not catch asyncio.CancelledError unless you plan to re-raise it after cleanup. In modern Python, CancelledError inherits from BaseException, so a normal except Exception block will not catch it. That is intentional: cancellation should usually stop the coroutine.

One production problem appears as soon as async starts working well: you can send too much work at once.

Model providers and vector databases usually limit requests per minute, tokens per minute, concurrent requests, or some combination of those. Your own service also has limits: memory, CPU, connection pools, database connections, and queue capacity.

If you start 100 requests at once with gather(), you may get HTTP 429 errors, increase tail latency, or overload your own process. asyncio.Semaphore is the simplest way to cap in-flight work.

A semaphore has a counter. Each task acquires one slot before it starts the protected work. When all slots are taken, the next task waits until a slot is released.

With a limit of 3, only three tasks enter the protected section at the same time. When one finishes, another task can start.

A semaphore limits concurrency. It does not, by itself, enforce requests per minute or tokens per minute. For strict provider limits, combine a concurrency limit with retry backoff, provider rate-limit headers, and a real rate limiter.

Here is a small client wrapper that applies the same idea to LLM API calls:

This class keeps the HTTP client and semaphore together. Every call through it respects the same concurrency limit, and the async context manager closes the connection pool when the batch is done.

Tasks wait behind the semaphore. Only three pass through at a time in this diagram. When one task finishes, the next waiting task gets a slot.

asyncio.gather() waits for all tasks to finish, then returns all results at once. Sometimes that is not ideal.

If tasks have different latencies, you may want to process each result as soon as it is ready. asyncio.as_completed() gives you results in completion order, not submission order:

This is useful for progress updates, partial UI results, and "fastest successful response wins" designs. If you use the fastest-response pattern, remember to cancel work you no longer need and close any streaming responses cleanly.

Many LLM APIs support streaming. Instead of waiting for the full response, the application receives small chunks as the model produces output.

In Python, async iteration fits streaming well. You can process each chunk as it arrives while still allowing the event loop to handle other work.

Here is a raw HTTP streaming example using the OpenAI Responses API:

You can also write your own async generators. An async generator is an async def function that uses yield. The caller consumes it with async for.

Here is a simple simulated stream:

Async generators are useful for streaming UIs, live logs, agent traces, progress events, and any pipeline where partial output is valuable.

Async is not a general performance switch.

Use async when your program waits on I/O: HTTP APIs, databases, queues, sockets, object storage, and streaming responses.

Do not expect async to speed up CPU-heavy work such as parsing huge files, resizing images, running local model inference, or doing large numerical computations. For CPU-bound work, use better algorithms, vectorized libraries, multiprocessing, worker queues, or a dedicated service.

Also be careful with blocking libraries. If a library does not support async and performs network or disk I/O, calling it inside an async function can block the event loop. In that case, use an async-compatible library, run the blocking call in a thread with asyncio.to_thread(), or move it out of the request path.

References