An order service sends customer addresses and order totals to Kafka. Those records travel from the producer to a broker, between brokers during replication, and from brokers to consumers. Without transport encryption, someone who can observe those network connections may be able to read the data.

TLS protects traffic on each connection and lets a client verify the server it is talking to. Getting it right in Kafka requires more than installing a certificate on one broker: clients discover other brokers, internal servers open their own connections, and certificates eventually need replacement.

This chapter explains how TLS works, how Kafka uses certificates and truststores, and how to configure and verify protected connections. Configuration examples target Apache Kafka 4.3 with KRaft and the Apache Kafka Java clients.

Transport Layer Security, or TLS, protects bytes as they travel between two endpoints. It provides confidentiality against network observers, detects unauthorized changes to protected traffic, and authenticates the server when the client validates its certificate correctly. TLS can also authenticate the client through a client certificate.

For Kafka, the protected traffic includes record keys, values, headers, and Kafka requests and responses. Serialization and compression do not provide this protection. A compressed order event is still readable by someone who can decode its format.

TLS protects each connection separately. The broker decrypts incoming traffic so it can process the Kafka request. When a follower fetches records or a consumer reads them, those transfers use other connections that need their own TLS configuration.

The diagram follows one order through three protected connections. The arrows show data movement, including responses to fetch requests.

The disk write is outside TLS's scope. Storage encryption needs a separate mechanism, such as encrypted volumes. The broker process and the receiving application can also see the decrypted data. If a field must remain hidden from the broker, the application needs to encrypt that field before publishing it and manage the corresponding keys.

TLS also leaves some network information visible, including endpoint addresses, connection timing, and approximate traffic volume. Its purpose is to protect the contents and integrity of the connection, not conceal all evidence of communication.

Encryption is useful only if the client establishes the connection with the intended server. Otherwise, an attacker could accept an encrypted connection while pretending to be a broker.

A certificate binds a public key to an identity and includes information such as its issuer, validity period, and permitted uses. The server holds the corresponding private key and proves possession of it during the handshake. You can share the certificate; the private key must remain secret.

A certificate authority, or CA, signs certificates. A client accepts a server certificate only if its validation rules establish a valid chain to a trusted authority and the certificate is appropriate for that connection. A typical chain runs from a broker certificate through an intermediate CA to a trusted root CA.

For example, the store might trust a company CA that issues certificates for its Kafka brokers. The broker presents its own certificate and the intermediate certificates needed to build the chain. The client already has the trusted root. The client does not normally contact the CA to ask it to approve every connection.

Hostname Verification

Trusting the issuer does not establish which server the client reached. A company CA may issue certificates for hundreds of services.

The client also compares the hostname it connected to with the certificate's Subject Alternative Name, or SAN, entries. For a connection to broker-1.kafka.example.com, the certificate should contain a matching DNS SAN. A connection using an IP address needs a matching IP SAN; a DNS entry containing the same digits is not equivalent.

Suppose Broker 2 presents a valid certificate for broker-2.kafka.example.com, but the client is trying to reach broker-1.kafka.example.com. The client may trust the issuing CA, yet hostname verification should reject the mismatch.

Keep hostname verification enabled. Correct the certificate or the connection address when they disagree. Disabling the check removes part of the protection against reaching an unintended server.

Keystores and Truststores

Java-based Kafka configurations commonly use two kinds of stores:

For server authentication, the broker needs its private key and certificate chain, while the client needs the trust material it uses to validate the broker. With mutual TLS, the client also has a keystore, and the broker has trust material for validating clients.

The examples use PKCS12 files, a supported format for both roles. A filename ending in .p12 does not configure the format by itself, so the examples set the store type explicitly.

Keep private keys accessible only to the processes that need them. Truststores contain public certificates, but their integrity matters: someone who changes the trusted authorities can change which identities the process accepts.

Before sending Kafka application traffic, the endpoints perform a handshake. They agree on compatible cryptographic settings, authenticate the required identities, and derive keys for protecting the connection.

Certificates help establish identity. The endpoints then use symmetric traffic keys to encrypt and authenticate the bulk data, rather than encrypting every order with the certificate's public key.

The sequence below shows the main responsibilities in a full certificate-based handshake. Exact protocol messages and their ordering depend on the TLS version.

If required validation fails, the connection cannot proceed to normal Kafka requests. An untrusted issuer, an expired certificate, or incompatible TLS parameters can therefore prevent a producer from sending any records.

Kafka clients reuse connections, so clients do not need a handshake for every record. Creating a producer for each order would cause unnecessary connection setup as well as other client overhead. A long-lived producer can send many batches over established connections.

Server Authentication and Mutual TLS

With server authentication alone, the client verifies the broker, but the broker does not obtain a client identity from a certificate. Kafka can combine this TLS connection with SASL authentication through SASL_SSL.

With mutual TLS, or mTLS, both sides authenticate using certificates. Kafka's SSL security protocol supports this arrangement when the listener requires client certificates. The configuration name remains SSL even though the connection uses modern TLS.

An accepted client certificate establishes an identity. Whether that identity may write to orders.placed remains an authorization decision. Issuing a certificate and granting topic access are separate administrative actions.

A Kafka client uses bootstrap.servers to make its initial contact. It then discovers broker addresses through metadata and connects to the brokers needed for partition access and group coordination.

Those discovered addresses come from the brokers' advertised listeners. Every address the client uses must be reachable and match the certificate that endpoint presents.

Suppose the order producer bootstraps through Broker 1, then discovers that Broker 2 leads the partition it needs. Both connections require independent validation, as the sequence below shows.

A successful bootstrap connection proves only that the first endpoint worked. If Broker 2 advertises an internal hostname that the producer cannot resolve, or presents a certificate for a different name, writes to its partitions will fail. The problem may appear only after a leadership change sends traffic to that broker.

Plan certificates from the actual addresses peers will use. Include required bootstrap aliases as well as broker-specific names where applicable. The bind address 0.0.0.0 means the server listens on all interfaces; it is not a client destination or a hostname to certify.

Internal paths need the same care. Brokers act as TLS clients when connecting to other brokers, and KRaft nodes open connections to controllers. Configure trust, identity, and hostname validation for those paths as well as application traffic.

If a proxy terminates TLS, the proxy becomes a decryption endpoint. Protecting traffic from the proxy to Kafka requires another TLS connection. A TCP proxy that passes TLS through has a different arrangement: the Kafka endpoint still presents the certificate. In either case, the deployment must support the broker addresses Kafka advertises.

The example below configures an application listener named CLIENT to require client certificates. It is a TLS configuration fragment for an existing KRaft deployment, not a complete cluster startup file.

Assume the deployment already defines its broker and controller roles, storage, quorum, and protected internal listeners. Add a CLIENT endpoint to the broker's existing listener lists, using port 9094 here:

Repeat the setup with each broker's own advertised hostname and certificate. Replace these example DNS names with names that resolve in your environment.

Before applying the settings, provision these files:

Use your certificate issuance process to create them. Keep the CA signing key outside brokers and application containers. If an internal node uses the same certificate for both client and server roles, its permitted usages must support both roles.

Broker Settings

These settings apply specifically to the CLIENT listener. Kafka uses the lowercase listener name in the property prefix.

The password values are placeholders. Have the deployment supply the real values through its secret-management process into a protected configuration; do not commit them to source control. Kafka does not automatically substitute these placeholder strings.

required makes a client certificate mandatory. requested allows a client to omit its certificate, so it does not enforce the same boundary. The truststore determines which client issuers this listener accepts.

The protocol list allows TLS 1.3 and TLS 1.2. The two peers still need compatible cryptographic settings their runtimes permit. If your environment requires TLS 1.3 exclusively, verify every client and internal peer before restricting the list.

Java Client Settings

The order service needs to trust the brokers and present its own identity. Save these connection settings in a protected file such as /etc/order-service/kafka-tls.properties, using real file paths and passwords from the deployment:

The https value selects hostname verification rules. Kafka still speaks its own protocol over TLS; it does not switch to HTTP.

These are connection properties that the Java producer, consumer, and Admin client share. A producer still needs its serializers and other application settings. Other client libraries may use different property names or file formats, so translate the intent using that library's configuration rather than copying Java properties blindly.

The broker must also authorize the principal it derives from the application certificate. Without the appropriate permissions, TLS can succeed while a Kafka operation fails.

Verify TLS independently before investigating topic permissions or application processing. This makes it easier to distinguish certificate problems from Kafka authorization failures.

With OpenSSL 3.x installed, you can inspect Broker 1 using the following command. It assumes you also have PEM files for the broker CA trust anchors, the application certificate, and its private key. These are separate from the PKCS12 files the Java client uses.

If the client certificate needs intermediate certificates, also supply them through -cert_chain with a PEM chain file. Protect the private key the diagnostic tool uses just as you protect the application's key.

-servername sends the intended server name; -verify_hostname checks it against the certificate. -verify_return_error makes server certificate validation failures abort the handshake. Inspect the negotiated protocol and verification result, and check for a server alert rejecting the client certificate.

This command does not send Kafka requests. Follow it with a Kafka operation using the Java configuration. With the Kafka 4.3 distribution and a supported Java runtime installed, run from the Kafka installation directory:

Assume orders.placed exists and this identity has permission to describe it. A successful response checks Kafka communication as well as the TLS setup. It does not prove write access or connectivity to every partition leader, so also verify the application's intended produce or consume operation.

When a connection fails, inspect both the client error and the receiving server's logs:

Check every advertised broker endpoint when failures affect only some partitions. Repeated retries cannot fix a certificate that names the wrong host.

Certificates expire, service names change, and issuing authorities sometimes need replacement. Treat certificate renewal as a normal deployment operation with monitoring and a tested procedure.

An established connection may keep working while fresh connections fail certificate validation. A service can therefore appear healthy until it restarts or reconnects to another broker. Test new connections when validating a renewal, rather than relying only on existing traffic.

Replacing a broker certificate under the same trusted CA is usually simpler than replacing the CA itself. A CA change needs an overlap period so peers can accept both old and new certificates while the deployment moves between them.

A practical sequence for a broker CA replacement is:

Apply the same reasoning in the opposite direction when rotating the CA that issues application certificates: brokers need the new trust before applications present the new identities.

Use a reload or restart procedure that the component supports for this configuration. Replacing a file on disk does not guarantee that every running process reloads it. Also, removing trust does not necessarily terminate connections that already authenticated; incident response may require explicitly closing affected connections.

When introducing TLS into an existing cluster, stage the change so peers can reach the new listener before they depend on it. Move clients and internal traffic deliberately, verify the new paths, then retire plaintext endpoints. Leaving a reachable plaintext listener indefinitely leaves traffic exposed for any clients that continue using it.

TLS adds cryptographic work and connection setup cost. The effect depends on the runtime, hardware, traffic volume, record sizes, batching, and connection churn. Measure with representative encrypted traffic instead of applying a fixed throughput penalty to a plaintext benchmark.

For the order service, reuse the producer and keep normal batching behavior. During a broker restart, watch connection creation and handshake failures alongside request latency. Many clients reconnecting together can create a different load pattern from steady traffic over established connections.

Transport protection also leaves Kafka's delivery behavior intact. If the producer sends an order and loses the connection before receiving the acknowledgment, the write outcome may be uncertain. TLS does not resolve that uncertainty or prevent duplicates that application retries cause. Producer reliability settings and application processing rules still determine recovery behavior.

This separation helps during an incident. Fix the transport failure, then assess any uncertain writes or interrupted processing using the application's normal recovery procedure.

TLS protects Kafka traffic on each configured connection. Certificates establish identities through trusted issuers and hostname checks, while mutual TLS also requires the connecting client to prove its identity.

A reliable setup covers every advertised broker and internal connection, keeps keys protected, and verifies both the TLS handshake and real Kafka operations. Certificate renewal and fresh-connection testing are part of operating that setup. Storage protection, authorization, and delivery guarantees remain separate responsibilities.