Running Kafka on your own machine gives you a place to practice. You can create topics, send and read records, and see what happens when you stop and restart the server. Along the way, you’ll learn how to check that Kafka is ready to handle requests.

In this chapter, we’ll start a single Kafka node with Docker, check that it works, and create a topic. We’ll also cover how to stop, restart, and reset the setup. We’ll use KRaft, with the broker and controller running in the same process.

We’ll use the official JVM-based Docker image, apache/kafka:4.3.1. Using a specific version keeps the setup and bundled tools consistent with the examples.

You need:

All Kafka commands run inside the container, so you don’t need to install Java or Kafka on your machine. You won’t need Docker Compose either.

Check Docker first:

The output should include both client and server information. If Docker cannot connect to its daemon, start Docker Desktop or your Docker Engine service and try again. The docker command needs a running engine to work.

This setup has one broker and one controller in the same process, with no backup node to take over if it fails. Connections use plaintext with no authentication. Use sample data and keep access limited to your own machine.

By default, the image uses port 9092 for broker connections and port 9093 for controller connections inside the container. Each is a listener: an address and port where Kafka accepts a particular type of connection.

We’ll make the broker port available on your machine at 127.0.0.1:9092. The address 127.0.0.1 is the IPv4 loopback address, which limits access to your own machine. We won’t publish the controller port to the host.

The diagram shows two ways to reach the broker: from your host machine through Docker’s port mapping, or from a tool inside the container.

The commands below use docker exec to run Kafka’s tools inside the running container. For these tools, localhost means the container itself. A client on your host can also use localhost:9092 through the port mapping, as long as localhost resolves to 127.0.0.1.

Inside another container, localhost points to that other container. Connecting an application from there needs a different network setup.

Download the image:

Start a container named kafka-local. The -d flag keeps it running in the background:

The -p value has three parts: host-address:host-port:container-port. Here, it maps 127.0.0.1:9092 on your machine to port 9092 in the container. Using the loopback address limits access to your machine, but it does not add authentication or encryption.

Docker prints a container ID when the container starts. Kafka may still need time to finish starting.

The image’s startup scripts handle the configuration and KRaft storage setup. You don’t need a separate controller container, a manually generated cluster ID, or a storage-formatting command.

Use the command as shown for your first run. Adding broker settings through environment variables can change how the image builds its configuration, so don’t assume all defaults will stay in place when you add an override.

Inspect the container and its recent logs:

The container should stay running. If it exits, read the logs to find the cause before trying again. Look for problems such as invalid settings or a port that is already in use.

Ask Kafka to list its topics to check that the broker is ready:

On a fresh instance, the command may succeed without printing anything because there are no topics yet. To check its exit status, run this immediately afterward in the same terminal:

An exit status of 0 means the command succeeded. If you get a nonzero status or connection warnings, Kafka may still be starting. Wait briefly and try listing the topics again. If the problem continues, check the container logs.

You can also check the KRaft metadata quorum, which manages the cluster’s metadata:

With the default settings, the node ID is 1, and that node’s controller should be the leader. The output shows the cluster ID, leader details, and metadata replication status. Some values, such as the metadata log position, will change as Kafka runs.

These checks reach Kafka from inside the container. To inspect the port mapping for host clients, run:

The output should be 127.0.0.1:9092. This confirms Docker has the port mapping. To check that a host client can reach Kafka, that client still needs to make a successful Kafka request.

Create a topic named orders.placed with one partition and one replica:

One partition keeps the topic easy to inspect. We also need a replication factor of 1 because there is only one broker, and copies of the same partition must live on different brokers. This gives us one copy of the data, with no backup replica.

The --if-not-exists option lets you rerun the command without an error if the topic already exists. It won’t change an existing topic’s settings, so check the result:

For a new topic you create with the default setup, expect these values:

In the last three rows, 1 refers to broker 1. Those values are broker IDs, not counts. The output may also include a topic ID and other fields.

You now have a working broker and a topic with a leader. In the next chapter, we’ll publish and read records to see how producers and consumers use them.

A Kafka client first connects to a bootstrap address to ask for cluster metadata. It then uses the broker addresses in the response. An advertised listener is an address the broker gives clients for these connections.

This image advertises localhost:9092 by default. That works for the tools inside kafka-local and for host clients using our port mapping. Clients in other containers or on other machines need a different advertised address they can reach.

Suppose you change the Docker mapping to use host port 19092. A client may connect successfully to localhost:19092 at first, but Kafka still tells it to use localhost:9092. The client then tries that address and may fail. If you change the port mapping, you also need to update Kafka’s listener configuration to match.

The diagram shows why the first connection succeeds and the second one fails.

If something goes wrong, use this table to narrow down the cause:

If you suspect Kafka ran out of memory, check the container’s status:

If OOMKilled is true, the system killed the process because it ran out of available memory. Check the container’s memory limits and the resources available to Docker. The resources needed for this small exercise won’t tell you how much a production cluster needs.

Stopping the container keeps its data available for the next start. Removing it discards that data. This setup does not use a named data volume to keep data across container replacements.

The diagram shows what happens when you stop, restart, or remove the container:

To stop Kafka, give it time to shut down:

Docker allows up to 30 seconds for a clean shutdown before forcing the process to stop. This gives Kafka time to shut down, but does not guarantee that Kafka safely stores all data.

Resume the same container with:

After restarting, repeat the readiness check and describe orders.placed. The topic should still exist. Kafka’s retention policy still applies, so keeping the container does not keep every record forever.

By default, this image stores Kafka’s data logs in /tmp/kraft-combined-logs inside the container. This path belongs to the container’s filesystem. A normal stop and start keeps it intact; removing the container deletes it. To preserve these logs with a volume, the volume must cover the directory Kafka actually uses.

For a fresh start, run the commands below. They delete the container and its learning data. The -v option also removes any anonymous volumes that the container uses. The downloaded Kafka image stays available.

If the container is already stopped, run only docker rm -v kafka-local. Then use the original docker run command and create the topic again. These commands target this learning container, so there’s no need to clean up other containers or volumes.

Keeping the container makes it easy to continue practicing. Remember that it holds the only copy of your Kafka data; there is no backup node if you lose that data.

You’ve started Kafka with Docker, checked that the broker and controller respond, and created a topic. Kafka’s tools run inside the container, so you can practice without installing Java or Kafka on your machine.

Clients need broker addresses they can reach, which is why the port mapping and advertised listener must agree. Stop and restart the same container to continue where you left off, or remove it when you want a fresh start. Use this single-node setup for local learning.