An online store produces many kinds of events: customers place orders, payment services capture payments, and warehouses dispatch shipments. Applications need a clear way to publish and find the events they care about. Mixing everything into one stream makes that harder as the system grows.

Kafka organizes these streams into topics. A topic gives records a shared name and a set of storage settings, while the applications using it agree on what those records mean. In this chapter, we’ll explore how topics work, how to choose useful boundaries and names, and how to create and inspect one.

A topic is a named stream of records within a Kafka cluster. Producers write records to a topic, and consumers read records from it. For example, an order service can publish an OrderPlaced event to orders.placed whenever the store accepts an order.

The topic name identifies the stream. The individual record describes a particular event:

Here, orders.placed holds events for many orders. The key identifies one order, and the value carries the event data. We don’t need a separate topic for ord-1042 and another for ord-1043.

A topic can exist before it contains any records. It also continues to exist when producers and consumers disconnect. Applications connect to the stream when they need it; their lifetimes do not determine the topic’s lifetime.

Multiple producer instances can write to the same topic. For example, three instances of the order service can all publish to orders.placed. They should follow the same event contract: an agreement about what an event means, which fields it contains, and the format of those fields.

Kafka stores keys and values as bytes. Creating a topic does not define a JSON schema or make Kafka verify that every record describes a placed order. The applications and any schema tooling they use are responsible for that agreement.

Suppose both notifications and reporting need every placed-order event. The order service can publish each event once to orders.placed, and the two applications can read it independently using separate consumer groups.

The diagram shows two producer instances sharing one topic. Each consumer box represents a separate application with its own group. The outgoing arrows represent records brokers return in response to consumer fetch requests.

Notifications reading an event does not remove it or prevent reporting from reading it. Each group tracks its own progress through the same stored history. If both applications instead used the same regular consumer group, they would share the reading work rather than each independently reading the entire stream.

This makes the topic a useful boundary between applications. The order service does not need a list of all its consumers, and a new reporting application can start reading retained events without changing the producer.

The independence has limits. Consumers still depend on the event contract and on the data remaining available. A producer that changes an amount from paise to rupees without coordinating the change can break reporting even though Kafka successfully stores and delivers the record.

Publishing successfully also does not mean every consumer has completed its work. A topic stores records; it does not track whether a consumer delivered an email or updated a report.

A topic is the name applications use for a stream, but Kafka stores its records in one or more partitions. Each partition is an ordered log. Kafka appends new records to its end. An offset identifies a record’s position within that partition.

The following example shows a topic with two partitions. Assume the producer routed the records as shown; the diagram does not imply a particular key-routing algorithm.

Both partitions have an offset 0, but those offsets identify different records. There is no single sequence of offsets for the whole topic. Kafka preserves the log order within each partition; it does not establish one total order across the two partitions.

Partition logs live on brokers, the servers that store and serve Kafka records. Kafka can distribute a topic’s partitions across brokers. Each partition can also have replicas, which are copies of its log. The replication factor is the number of copies of each partition, including its leader.

Partitions divide the records; replicas copy them. A topic with two partitions and replication factor three has two distinct logs and six partition replicas. Consumers do not receive three business events merely because Kafka stores three copies of a partition.

For topic design, a topic spans partition logs and can span brokers. Applications use its name while Kafka manages where its partitions live.

A useful topic groups records with related meanings and compatible storage and access needs. Before creating one, consider who owns the events, who needs to read them, and whether they should share retention and access rules.

For the online store, these are plausible boundaries:

Separating these streams lets a fulfillment application subscribe to placed orders without reading every shipment update. It also allows the team to retain payment events differently or grant access to a narrower set of applications.

These boundaries are design choices, not a Kafka requirement to create one topic per event type. A topic named orders.events could contain OrderPlaced, OrderCancelled, and OrderCompleted events if consumers need the order lifecycle together and the records can share policies. That design requires an explicit way to identify each event type and consumers that understand the agreed formats. Combining them in one topic still does not create ordering across partitions.

At the other extreme, a topic named all.events can become difficult to use when unrelated teams publish unrelated data into it. Consumers may need to read and discard most records, and a single retention or access policy may not fit everyone.

Creating a topic for every order introduces a different problem: a growing number of topics and partitions to administer, often without a useful difference in policy. Keep the order identifier in the record unless there is a specific operational reason to isolate its stream.

Nor does every new consumer need a new topic. If a fraud-detection application needs the same placed-order events as reporting, it can use its own consumer group on orders.placed. A new topic becomes useful when the data itself needs a separate contract or policy, such as a derived stream containing only suspicious orders.

A good topic name helps someone understand its contents without inspecting sample records. Prefer stable business concepts such as orders.placed over names tied to a current implementation, such as service-a-output.

A team might adopt a convention such as commerce.orders.placed, where the first part identifies the business domain. Lowercase words and consistent separators make names easier to scan. These are team conventions, not special Kafka behavior.

In particular, dots do not create a topic hierarchy. orders, orders.placed, and orders.cancelled are separate topic names. Publishing to orders.placed does not also publish to orders, and subscribing to the literal name orders does not include the other two topics. Clients can explicitly subscribe to multiple topics or use supported pattern subscriptions, but that is a separate choice.

Kafka also does not inspect eventType and select a topic automatically. The producer must choose the destination. A record whose value says OrderCancelled can still reach orders.placed if the application sends it there by mistake.

Alongside the name, document the owning team, event meaning, data format, intended readers, and retention expectations. Ownership matters because consumers need someone to contact when an event is unclear or its contract needs to change.

Access controls must enforce who can read and write. A name such as private.payments communicates intent but does not, by itself, restrict access.

A topic’s cleanup policy controls how Kafka removes stored history. With cleanup.policy=delete, Kafka removes old log segments according to configured time or size limits. A segment is a file containing part of a partition’s log.

retention.ms sets the time limit in milliseconds. retention.bytes sets a size limit per partition, not for the entire topic. If you configure both limits, Kafka can remove old data when either limit is reached. Cleanup works on segments in the background, so a time limit is not an exact deletion deadline for each record.

To see the consequence for consumers, imagine reporting has stopped while notifications continues. The diagram shows two possible outcomes when reporting returns:

Cleanup does not wait for every consumer group to finish. A consumer’s saved position cannot protect unread records from removal. Once needed history is gone, recovery requires another source or an application-specific way to handle the gap.

Choose retention with both outages and catch-up time in mind. If reporting can be offline for a day and needs several more hours to recover, a one-day history leaves too little room. The right duration depends on recovery requirements, event volume, and storage capacity.

Kafka also supports cleanup.policy=compact, which removes older values for the same key while preserving the latest state, subject to deletion markers. Compaction runs in the background and does not immediately replace an existing record. It suits streams of changing state; it is not a substitute for preserving every event in a history. You can also combine the delete and compact policies.

Let’s make these choices concrete with a local topic. The commands below use Kafka 4.3 tools inside a running container named kafka-local, with a single combined broker and KRaft controller reachable at localhost:9092 inside the container. Run the commands from a Bash-compatible host terminal; you do not need to install Kafka on the host.

Configure and start the local container before running these commands. Plaintext connections and replication factor one are for learning. A single replica provides no backup copy if the broker loses its data.

Create a dedicated topic for inspecting its configuration:

This creates two partitions and explicitly chooses a three-day time limit: 3 × 24 × 60 × 60 × 1000 = 259200000 milliseconds. retention.bytes=-1 disables the size-based retention limit for this example; it does not provide unlimited disk capacity. These values illustrate the settings rather than recommend production sizing.

If the topic already exists, choose another name and use it consistently in the remaining commands. Rerunning --create does not update an existing topic’s configuration.

List the topics to confirm the name is present:

Then inspect its layout:

The output should report PartitionCount: 2 and ReplicationFactor: 1, followed by entries for partitions 0 and 1. Each should have an assigned leader. Broker IDs and the generated topic ID depend on the environment. Both partitions can live on the same broker; two partitions do not require two brokers.

Finally, inspect the topic’s explicit configuration overrides:

Look for cleanup.policy=delete, retention.ms=259200000, and retention.bytes=-1. Settings without a topic override use the applicable broker defaults. An omitted setting in this override listing does not mean the topic has no value for it.

An empty topic is enough for this inspection. Creating it establishes its metadata and partition layout; it does not generate sample events.

Explicit Creation

Kafka can automatically create missing topics when broker settings, client requests, and permissions allow it. The broker setting auto.create.topics.enable controls server support for this behavior.

Automatic creation is convenient for experiments, but it can turn a misspelled destination into a new topic with unintended defaults. For example, a producer might write to orders.place while consumers wait on orders.placed.

In production, create topics explicitly so the team can review their names, partition counts, replication, and cleanup settings before applications use them. If you disable automatic creation, an authorized administrator or provisioning tool must create each topic first.

A Kafka topic gives a stream of records a name and shared storage policies. Producers choose where to write, while separate consumer groups can read the same retained history independently. The applications agree on the meaning and format of the records.

Choose topic boundaries around related data and compatible operational needs. Use clear names, assign ownership, and inspect the settings rather than assuming defaults. Partitions hold the actual logs, and cleanup determines how much history remains available for consumers and replay.