In this lab, you secure a single-broker cluster in stages. You add TLS, authenticate clients with certificates and SCRAM, enforce ACLs for ordinary principals, and throttle replay traffic. You test rejected and successful requests for TLS, authentication, and ACLs, then compare replay times with and without a quota. For each control, save the expected failure or throttling evidence and a successful request.

Use this fixed lab setup: the node exposes three broker listeners plus a separate controller listener: an internal PLAINTEXT listener on port 9092 that Docker never publishes to your host, an SSL listener named CLIENT on host port 9094, and a SASL_SSL listener named SECURE on host port 9095. Every certificate chains to one lab CA, the broker certificate names only localhost, the SCRAM users are order-service and reporting-service, and the only application topic is orders.placed with three partitions. Change one setting at a time and inspect the error to identify the failure stage.

Environment: a single-node variant of E2. One apache/kafka:4.3.1 combined broker and controller named kafka-1 runs under Docker Compose, with its full server.properties mounted at /mnt/shared/config and its keystores mounted under /etc/kafka/secrets. Key generation runs on your host with OpenSSL 3.x and keytool from a Java 21 JDK. Every Kafka tool runs inside the container through docker exec and reads client property files from the same /etc/kafka/secrets mount, so the host needs no Kafka download. Use Terminal A for long-running consumers and Terminal B for everything else.

Run every command from kafka-labs/lock-down-cluster/. Create the directories above before writing files. Start with a new lab container so orders.placed and order-reporting have no earlier data or offsets. Complete the cleanup of any earlier lab using the container name kafka-1 first.

Create docker-compose.yml. Only the two secured ports reach the host:

The ca/ directory stays on the host. The container receives only the broker/client material in secrets/; it never receives either CA signing key.

Create config/server.properties. The keystore and truststore settings are unprefixed so both secured listeners share them; the listener-prefixed forms from the TLS chapter override these per listener when you need that:

Look for which principal each listener produces; every later phase attaches evidence to one of these three paths:

Create secrets/client-ssl.properties. It trusts the lab CA and presents no certificate:

Create secrets/client-wrong-ca.properties as a copy whose truststore is /etc/kafka/secrets/wrong-ca.truststore.p12. Create secrets/client-mtls.properties as a copy of client-ssl.properties with four lines appended:

Create secrets/order-service.properties, which matches the SASL chapter's client example:

Create secrets/reporting-service.properties as a copy with username reporting-service, password reporting-secret-1, and one extra line, enable.auto.commit=false, so these console consumers do not save new offsets automatically. With a fresh order-reporting group and --from-beginning, each run replays retained history. Existing committed offsets would still take precedence.

Create events/batch-1.txt, one keyed record per line:

Create the lab CA and a second, unrelated CA that nothing should trust:

Write the extension files that give the broker its SAN and give the client its usage:

Create the broker key and request, then sign it with the lab CA:

Repeat both commands for the client with subject /CN=order-client, files client.key, client.csr, client.crt, and client-ext.cnf. Then verify the chain, inspect the SAN, and try the verification against the wrong CA:

Package the broker key and chain into a PKCS12 keystore, and do the same for the client with client.crt, client.key, name client, and output client.keystore.p12:

Build the two truststores, one per CA. Repeat with wrong-ca.crt, alias wrong-ca, and output wrong-ca.truststore.p12:

The broker process in the image runs as a non-root user, so make the mounted files readable:

Save every output as observations/01-certificates.txt.

What you should see: the first openssl verify prints secrets/broker.crt: OK and secrets/client.crt: OK. The SAN line contains DNS:localhost and nothing else. The verification against wrong-ca.crt fails with an error saying OpenSSL could not find the issuer certificate. Both CAs are valid; only one signed the broker certificate, which is the distinction Phase 2 turns into a client error.

Start the broker, wait until a topic listing over the internal listener succeeds, then create the topic:

Check the CLIENT listener from the host with the TLS chapter's diagnostic, then again with -CAfile ca/wrong-ca.crt:

Now send a Kafka request over TLS:

Break it twice. First, rerun with --command-config /etc/kafka/secrets/client-wrong-ca.properties. Second, rerun with the original config and --bootstrap-server 127.0.0.1:9094, an address the certificate does not name. Save everything as observations/02-tls.txt.

What you should see: openssl s_client prints a Verification: OK line and the negotiated protocol version; with the wrong CA it prints a verification error and, because of -verify_return_error, no established connection. The describe shows three partitions with Leader: 1. The wrong-truststore describe fails with SslAuthenticationException: SSL handshake failed and a cause containing PKIX path building failed. The 127.0.0.1 describe fails with the same exception and a cause containing No subject alternative names matching IP address 127.0.0.1 found. Copy both texts into your failure table. The broker did not change between the three attempts; the client's trust and the address it dialed did.

Append one line to config/server.properties, then restart the broker:

Run the Phase 2 describe with client-ssl.properties, which has no keystore, and then with client-mtls.properties. Save both as observations/03-mtls.txt.

What you should see: the first describe fails with SSL handshake failed and a cause naming a fatal alert, certificate_required under TLS 1.3 or bad_certificate under TLS 1.2. The second prints the same three-partition output as Phase 2. With Kafka's default SSL principal handling, the broker now knows this client as User:CN=order-client, the certificate's distinguished name. No authorizer is running yet, so that principal restricts nothing. Phase 5 changes that.

Provision both users over the internal listener, which needs no authorization yet:

Repeat for reporting-service with password reporting-secret-1. Publish batch 1 as order-service on SECURE:

In Terminal A, start the reporting consumer and leave it running:

Break authentication twice with kafka-topics.sh --describe on port 9095: once with a copy of reporting-service.properties whose password is wrong, and once with a copy whose sasl.mechanism is SCRAM-SHA-512. Then rotate the reporting password:

Publish one more record with the same producer command, feeding it ord-1045|{"eventId":"evt-7004","eventType":"OrderPlaced","orderId":"ord-1045"} on standard input, and watch Terminal A. Then run a describe on 9095 with the unchanged reporting-service.properties, edit its password to reporting-secret-2, and run it again. Save everything as observations/04-scram.txt and stop Terminal A with Ctrl-C.

What you should see: each --alter reports that it completed updating the config for the user. Terminal A prints the three batch-1 keys. The wrong password fails with SaslAuthenticationException and a message about invalid credentials with mechanism SCRAM-SHA-256. The SHA-512 attempt fails with UnsupportedSaslMechanismException and lists [SCRAM-SHA-256] as the enabled mechanisms. After the rotation, Terminal A still prints ord-1045 on the connection it authenticated before the change, while the broker rejects the fresh describe with the old secret and the edited file succeeds. That is the SASL chapter's rotation diagram reproduced: the existing session survives, and the next authentication needs the current secret.

Append the authorizer settings from the ACL chapter, plus the lab's superuser, and restart with the stop and start commands from Phase 3:

Publish ord-1046|{"eventId":"evt-7005","eventType":"OrderPlaced","orderId":"ord-1046"} with the Phase 4 producer command and record the error. Then grant the producer its single permission over the internal listener:

Publish ord-1046 again. Now run the Phase 4 consumer command with --max-messages 5 --timeout-ms 30000 appended and record the error. Add the topic grant, rerun, and record the error again:

Add the group grant by repeating the command with --group order-reporting in place of --topic orders.placed, and rerun the consumer. List the effective policy:

Repeat the listing with --group order-reporting. Finally, run the Phase 3 mTLS describe on port 9094 once more. Save everything as observations/05-acls.txt.

What you should see: the first publish fails with TopicAuthorizationException: Not authorized to access topics: [orders.placed], and the second succeeds silently. The consumer with no grants fails with an authorization error naming either the topic or the group; record whichever appears. With only the topic grant it fails with GroupAuthorizationException: Not authorized to access group: order-reporting. With both grants it prints five records and exits. The listings show exactly three allow entries: WRITE for User:order-service on the topic, READ for User:reporting-service on the topic, and READ for User:reporting-service on the group. The mTLS describe now fails, either with a topic authorization error or with a message that the topic does not exist, because the metadata listing hides topics the principal User:CN=order-client may not describe. Record the exact text.

Generate a backlog of 20,000 records of about 1 KiB each and publish it as order-service with the Phase 4 producer command, reading events/backlog.txt instead of batch-1.txt:

Record the file size, then give reporting-service a 1 MiB/s consumer quota:

Time a full replay. The console consumer does not print the fetch-throttle-time-avg metric from the quotas chapter, so elapsed time provides indirect evidence. Repeat the runs or collect throttle metrics to distinguish quota delay from warm-cache, startup, and processing effects:

Remove the quota with --delete-config consumer_byte_rate in place of the --add-config argument, describe the user again, and time the same replay a second time. Save everything as observations/06-quotas.txt.

What you should see: the describe shows consumer_byte_rate=1048576 next to the user's SCRAM entry, and after the deletion only the SCRAM entry remains. Both replays report that they processed a total of 20005 messages. Use backlog size divided by 1 MiB/s as a rough estimate, allowing for quota sampling and client startup. The unthrottled run may finish faster if the quota was the main bottleneck. Record both durations and investigate other limits. Reporting kept its Read permission throughout. The quota changed how fast the broker served it, not whether the broker allowed the read.

Stop any running consumer with Ctrl-C, then remove the broker and its container filesystem:

The listing should show no container. Delete ca/ca.key, ca/wrong-ca.key, secrets/broker.key, secrets/client.key, secrets/broker.keystore.p12, and secrets/client.keystore.p12 once your report is complete; the submission needs the redacted property files, not the keys. Keep the rest of kafka-labs/lock-down-cluster/.

You built one combined node with three broker listeners and a controller listener and tested TLS, authentication, and authorization with rejected and successful requests. TLS rejected the wrong trust anchor and the wrong address before the client sent any Kafka request, mutual TLS turned a certificate into the principal User:CN=order-client, and SCRAM turned a provisioned password into User:reporting-service while showing that a rotated secret only matters at the next authentication. None of those steps restricted an operation on its own.

The authorizer did. With allow.everyone.if.no.acl.found=false, the producer needed one Write grant and the consumer needed both a topic Read and a group Read, and the listings showed exactly those three entries. The quota then limited how fast an authorized reader could pull the backlog without touching its permissions. Encryption, identity, permission, and capacity are separate controls, and the evidence comes from the authentication and authorization checks, successful requests, and replay timing comparisons.