A reporting application may begin by reading orders.placed. Later, it needs cancellation events too, or separate order topics for each region. Choosing those inputs is a small API call with consequences for what the application processes, how consumers share work, and where reading begins.

We’ll look at explicit topic lists, pattern subscriptions, and changing a subscription while a consumer is running. We’ll also distinguish subscribing to topics from choosing partitions directly.

The examples use the Apache Kafka Java client 4.3.1 and ordinary consumer groups. Unless stated otherwise, they use group.protocol=classic. The Java snippets illustrate subscription choices inside an application that already owns a consumer and handles polling, processing, commits, and shutdown; they are not standalone programs.

A subscription describes the topics a consumer wants to read. An assignment is the set of topic partitions that the consumer currently owns through its group.

Suppose orders.placed has four partitions, and two consumers in the order-reporting group both subscribe to it. One possible assignment gives partitions 0 and 1 to Consumer A and partitions 2 and 3 to Consumer B. Both consumers subscribe to the whole topic, but each reads only its assigned share.

The diagram separates the common subscription from one possible distribution of work:

The assignment policy determines the actual distribution. Subscribing to a topic does not promise that this particular consumer will receive every record in it. A separate application that needs its own complete view should normally use a separate group.

Subscription also says nothing about record-level filtering. If an order event has key ord-1042, a source header, and a JSON field currency=INR, those fields do not affect whether the topic matches the subscription. Filtering by an order ID, header, or value requires application processing or a different topic design.

An explicit list is a good starting point when you know the application’s inputs. The names are visible in configuration or code, and adding another input requires a deliberate change.

For these examples, assume Java 21 and a KafkaConsumer<String, String> named consumer. Configure it with the following properties before constructing it, along with string deserializers for both key and value:

The local address assumes a running Kafka broker reachable from the application. Plaintext local connections are for learning; a secured cluster also needs its own connection and authentication settings. The named topics must exist, and the application must have permission to use them and its consumer group.

After importing java.util.List, a single-topic subscription is:

To read placed and cancelled order events, supply both names in the same call:

This selects two independent topic streams. It does not merge their offsets or establish an order between them. Even if both topics use ord-1042 as a key, the application cannot assume it will observe the placement before the cancellation. Ordering remains a property of each partition’s log.

Replacing the List

Each list-based subscription supplies the complete desired set. It does not add names to an earlier call.

The following is an incorrect attempt to subscribe to both topics:

After the second call, the requested topic set contains only orders.cancelled. Use one list containing both names to keep both inputs. To remove cancellations later, replace that list with one containing only orders.placed.

An explicit string is also a literal topic name. List.of("orders.*") does not mean “all order topics.” Pattern subscriptions use a different overload of subscribe().

A successful return from subscribe() means the client accepted the subscription request locally. It does not confirm that topic metadata is available, group assignment has completed, or the consumer can fetch records.

The application must continue calling poll(). During normal consumption, the client discovers the needed metadata and coordinates its assignment. One poll may return no records while that work is in progress.

For an explicit list, these calls let you inspect the requested topics and the current assignment:

Immediately after the first subscribe(), the requested topics can be present while the assignment is empty. An empty assignment can also be legitimate if other group members own all available partitions. Neither a nonempty subscription nor an empty poll result is a complete readiness check.

Do not add a throwaway poll() solely to wait for an assignment and then discard its return value. That call can deliver real records and advance the consumer position. Every poll result belongs in the application’s normal processing and offset-handling path.

If you misspell a topic name, disabling allow.auto.create.topics prevents this consumer from requesting its creation. It does not guarantee that subscribe() immediately reports the missing topic. Metadata and access problems can become visible later through logs or exceptions during consumer operations.

Check both the topic setup and the running consumer. Verify the intended topics and permissions before starting the application, then observe whether the running consumer obtains assignments and processes data as expected.

An explicit list becomes inconvenient when topics follow a controlled naming convention and new members of that family should automatically become inputs.

Suppose a reporting service reads regional topics named orders.placed.us, orders.placed.eu, and orders.placed.in. Each topic carries the same event contract. When another region launches, the service should include its orders without a new list in application configuration.

For the classic protocol, use java.util.regex.Pattern. On a fresh consumer, write the subscription as:

Import java.util.regex.Pattern along with the other Java imports. This example is an alternative to the explicit list, not another call to append to it.

The expression requires the literal prefix orders.placed. followed by exactly two lowercase letters. In a regular expression, a dot normally matches any character. The backslash makes it literal, and Java’s string syntax requires you to escape that backslash again. The anchors ^ and $ make the intended whole-name match clear.

The table shows the boundary of this subscription:

The two-letter suffix is this application’s naming convention, not validation of a real region code. The pattern would also match orders.placed.zz. Topic creation policies must enforce which names are meaningful.

Discovering New Topics

With this Java Pattern subscription under the classic protocol, the client periodically refreshes metadata and checks topic names. A newly created matching topic can lead to a changed group assignment while the application keeps polling.

Here is the sequence when orders.placed.in becomes available:

Discovery and assignment take time. A successful topic-creation response does not mean that every interested consumer has already incorporated it. metadata.max.age.ms influences periodic discovery for pattern matching in this client, but lowering it is not an end-to-end deadline for receiving the first record.

The group can assign the new partitions to other eligible members. This individual consumer need not receive data from every matching topic.

Controlling the Match

A broad expression such as orders.* can include retry topics, archived inputs, or a future event format that the handler cannot understand. That can cause duplicate business processing or repeated deserialization failures without any application deployment.

Use patterns when all matching topics belong to the intended input contract. Test both names that should match and names that should stay out. Keep retry and output topics outside the matched family, especially when consuming a record can cause the application to publish another one.

Matching a name does not grant access to it. Kafka’s authorization rules still apply. Likewise, a pattern does not invent or create every possible matching topic; it discovers existing topics available through the cluster’s metadata and access rules.

Kafka 4.3.1 also provides SubscriptionPattern, which sends a regular expression for broker-side matching. This API requires group.protocol=consumer and a supporting broker; it is not a drop-in replacement for Pattern on the classic consumer configured above.

Create a separate consumer with group.protocol=consumer and import org.apache.kafka.clients.consumer.SubscriptionPattern. The equivalent subscription is:

The expression must be compatible with Google RE2/J, the regular-expression implementation this API uses. Constructing SubscriptionPattern stores the expression; the broker validates it. An incompatible expression can therefore cause an error during a subsequent poll() rather than at construction time.

Keep the distinction clear when reading configurations or debugging discovery. Java Pattern and SubscriptionPattern differ in where matching occurs and which expression syntax they accept. The client metadata setting described for classic pattern subscriptions does not control broker-side pattern refresh.

For a small, known input set, an explicit topic list remains easy to review. Use pattern discovery when automatic inclusion of new topics is part of the application’s intended behavior.

A running service may receive a configuration update that adds a topic or removes one. Apply that update on the thread that owns the consumer. A configuration watcher should pass the desired change to that thread rather than calling consumer APIs concurrently.

For an application that processes each poll result sequentially, a convenient point to change the subscription is after finishing the returned batch and successfully committing its completed progress. Then call subscribe() with the complete replacement list and continue the normal loop. Do this when the desired inputs change, not on every iteration.

The change can cause a rebalance, a redistribution of partition ownership among group members. If the application has handed records to workers, changing the subscription does not cancel those workers. It needs to account for their unfinished work and stop or reconcile work for partitions it no longer owns. Subscription changes cannot make an external side effect and an offset commit atomic.

Unsubscribing

To clear the consumer’s subscription and assignment, call:

This consumer gives up its subscribed work, allowing the group to redistribute it. Passing an empty list to subscribe() also unsubscribes the consumer. With no subscription or manual assignment, calling poll() is an invalid state; subscribe to topics or assign partitions before polling again.

Unsubscribing does not delete topics or erase the group’s committed offsets. It also does not close the consumer’s resources. Use the consumer’s close API when the application is finished with the instance.

If the objective is to stop delivering more records temporarily while preserving membership and assignment, pause() is the relevant mechanism. The application must still poll to maintain group activity. Removing a subscription changes ownership and is a different operation from temporarily pausing an assigned partition.

Switching Subscription Forms

The Java client does not let you combine an active explicit-list subscription with a pattern subscription. Clear the previous mode before switching.

After accounting for any in-progress work, the owner thread can switch a classic consumer from a list to a Java pattern like this:

This intentionally releases the previous subscription. It is not a seamless transfer of application work. Replacing one explicit list with another explicit list does not require this intermediate unsubscribe.

Selecting a topic and selecting a position within it are separate decisions. Adding a topic does not always mean starting at its beginning, and removing then restoring a topic does not reset its saved progress.

Suppose the group previously processed orders.cancelled, partition 0, and committed offset 43. It then removed that topic from its subscription. A week later, the application adds it back.

If the committed offset still exists and is valid, a consumer that receives that partition assignment can resume at 43. If no valid saved offset exists, the configured auto.offset.reset policy applies. With earliest, it starts at the earliest available position, which may be later than 0 because retention can remove old data.

The same rule applies when a pattern discovers a topic. A group with no saved position for its new partitions uses its starting-position policy. A latest policy can skip records already present when the consumer chooses those initial positions, including records producers published during discovery and assignment.

For the regional reporting service, that choice affects completeness. If every retained order must contribute to the report, beginning new inputs at their end can omit orders. If the application deliberately wants only newly arriving data, starting at the end may be appropriate. Choose that behavior explicitly instead of assuming the subscription call determines it.

Some tools need an exact partition rather than a share of a topic. A diagnostic reader, for example, may need to inspect orders.placed, partition 0, without joining the reporting group’s assignment process.

On a fresh consumer, after importing org.apache.kafka.common.TopicPartition, direct selection looks like this:

This establishes a manual assignment. It does not choose a particular offset; the tool must also decide how to obtain its starting position. For an independent inspection tool, disable automatic commits and avoid committing under the reporting group’s identity. Manual assignment can still use Kafka-based offset storage when you configure a group ID.

Kafka does not coordinate exclusive ownership between manually assigned readers. Two processes can both assign themselves partition 0 and read the same records. The application or an external system must arrange failover and prevent unwanted overlap.

The table summarizes the operational difference:

Manual assignment and automatic assignment through subscription are mutually exclusive on one consumer. Use unsubscribe() before switching between them, after dealing with outstanding work. For a long-lived service that should share work and recover through a consumer group, topic subscription is usually the appropriate choice.

Explicit subscriptions name the complete topic set, while patterns include topics that match a naming rule. Subscriptions express the desired inputs; group assignments determine which partitions each consumer actually reads.

Apply subscription changes through the consumer’s owner thread, account for unfinished work, and choose starting-position behavior separately. Use manual assignment when the application needs direct partition control and can take responsibility for ownership and recovery.