Kafka

0%Certificate0/110

Join the Community

What Is Apache Kafka?

Kafka vs Traditional Message Queues

Common Kafka Use Cases

Kafka Architecture at a Glance

Running Kafka Locally

Your First Producer and Consumer

Understanding Kafka Topics

Kafka Records and Record Structure

Topic Partitions

Record Keys and Partitioning

Offsets and Log Position

Ordering Guarantees

Choosing the Number of Partitions

Kafka Producer Architecture

Building a Producer

Partitioners and Record Routing

Producer Batching

Producer Acknowledgments

Producer Retries and Failure Handling

Kafka Consumer Architecture

Building a Kafka Consumer

Subscribing to Topics

Consumer Poll Loop

Consumer Groups

Partition Assignment

Consumer Group Rebalancing

Committing and Managing Offsets

How Consumer Groups Work Internally

Group Coordinator

Consumer Group States

Heartbeats and Consumer Liveness

Rebalance Triggers

Eager vs Cooperative Rebalancing

The New Consumer Rebalance Protocol

Static Membership and Reducing Rebalances

Kafka Broker Architecture

How Kafka Distributes Topics Across Brokers

Partition Leaders and Followers

Replication Factor

In-Sync Replicas

Leader Election

Broker Failure and Recovery

Designing for High Availability

Kafka as a Distributed Commit Log

How Kafka Stores Records on Disk

Offset and Timestamp Indexes

Sequential I/O and Kafka Performance

Page Cache and Zero-Copy

Message Delivery Guarantees

At-Most-Once Processing

At-Least-Once Processing

Duplicate Message Problem

Idempotent Producers

Kafka Transactions

Exactly-Once Semantics

Designing Idempotent Consumers

Designing Good Kafka Events

Event Keys and Partitioning Strategy

Commands vs Events

Event Notification vs Event-Carried State Transfer

Serialization Formats: JSON, Avro, and Protobuf

Schema Registry Concepts

Schema Evolution and Compatibility

Event Versioning and Contract Evolution

What Is Kafka Connect?

Source vs Sink Connectors

Kafka Connect Architecture

Workers, Connectors, and Tasks

Standalone vs Distributed Connect

Database Change Data Capture

Debezium and the Transactional Outbox Pattern

Building Reliable Data Pipelines

What Is Stream Processing?

Kafka Streams Fundamentals

KStream vs KTable

Stateless Transformations

Stateful Processing and State Stores

Aggregations, Windows, and Joins

Repartitioning and Internal Topics

Kafka Streams Fault Tolerance and Exactly-Once Processing

Kafka Cluster Metadata

Why Kafka Moved Away from ZooKeeper

Understanding KRaft

Controllers and Brokers

Metadata Quorum

Raft-Based Metadata Replication

Controller Failure and Leader Election

Kafka Cluster Startup and Metadata Changes

Kafka Security Model

TLS and Encryption in Transit

Authentication with SASL

SASL Mechanisms

Kafka ACLs and Authorization

Securing Producers and Consumers

Quotas and Resource Isolation

Multi-Tenant Kafka Clusters

Kafka Performance Mental Model

Throughput vs Latency Tuning

Producer and Consumer Performance Tuning

Consumer Lag and Backpressure Tuning

Kafka Metrics and Observability

Capacity Planning and Partition Sizing

Reassigning Partitions and Scaling Clusters

Upgrades, Disaster Recovery, and Multi-Cluster Kafka

Kafka

Welcome

Course Roadmap
Join the Community

Kafka Fundamentals

What Is Apache Kafka?
Kafka vs Traditional Message Queues
Common Kafka Use Cases
Kafka Architecture at a Glance
Running Kafka Locally
Your First Producer and Consumer

Topics and Partitions

Understanding Kafka Topics
Kafka Records and Record Structure
Topic Partitions
Record Keys and Partitioning
Offsets and Log Position
Ordering Guarantees
Choosing the Number of Partitions

Kafka Producers

Kafka Producer Architecture
Building a Producer
Serializers
Partitioners and Record Routing
Producer Batching
Producer Acknowledgments
Producer Retries and Failure Handling

Kafka Consumers

Kafka Consumer Architecture
Building a Kafka Consumer
Subscribing to Topics
Consumer Poll Loop
Consumer Groups
Partition Assignment
Consumer Group Rebalancing
Committing and Managing Offsets

Consumer Groups

How Consumer Groups Work Internally
Group Coordinator
Consumer Group States
Heartbeats and Consumer Liveness
Rebalance Triggers
Eager vs Cooperative Rebalancing
The New Consumer Rebalance Protocol
Static Membership and Reducing Rebalances

Kafka Brokers

Kafka Broker Architecture
How Kafka Distributes Topics Across Brokers
Partition Leaders and Followers
Replication Factor
In-Sync Replicas
Leader Election
Broker Failure and Recovery
Designing for High Availability

Kafka Storage Internals

Kafka as a Distributed Commit Log
How Kafka Stores Records on Disk
Log Segments
Offset and Timestamp Indexes
Sequential I/O and Kafka Performance
Page Cache and Zero-Copy
Log Retention
Log Compaction

Delivery Semantics

Message Delivery Guarantees
At-Most-Once Processing
At-Least-Once Processing
Duplicate Message Problem
Idempotent Producers
Kafka Transactions
Exactly-Once Semantics
Designing Idempotent Consumers

Event and Schema Design

Designing Good Kafka Events
Event Keys and Partitioning Strategy
Commands vs Events
Event Notification vs Event-Carried State Transfer
Serialization Formats: JSON, Avro, and Protobuf
Schema Registry Concepts
Schema Evolution and Compatibility
Event Versioning and Contract Evolution

Kafka Connect

What Is Kafka Connect?
Source vs Sink Connectors
Kafka Connect Architecture
Workers, Connectors, and Tasks
Standalone vs Distributed Connect
Database Change Data Capture
Debezium and the Transactional Outbox Pattern
Building Reliable Data Pipelines

Stream Processing with Kafka

What Is Stream Processing?
Kafka Streams Fundamentals
KStream vs KTable
Stateless Transformations
Stateful Processing and State Stores
Aggregations, Windows, and Joins
Repartitioning and Internal Topics
Kafka Streams Fault Tolerance and Exactly-Once Processing

Kafka Cluster

Kafka Cluster Metadata
Why Kafka Moved Away from ZooKeeper
Understanding KRaft
Controllers and Brokers
Metadata Quorum
Raft-Based Metadata Replication
Controller Failure and Leader Election
Kafka Cluster Startup and Metadata Changes

Security and Multi-Tenancy

Kafka Security Model
TLS and Encryption in Transit
Authentication with SASL
SASL Mechanisms
Kafka ACLs and Authorization
Securing Producers and Consumers
Quotas and Resource Isolation
Multi-Tenant Kafka Clusters

Kafka in Production

Kafka Performance Mental Model
Throughput vs Latency Tuning
Producer and Consumer Performance Tuning
Consumer Lag and Backpressure Tuning
Kafka Metrics and Observability
Capacity Planning and Partition Sizing
Reassigning Partitions and Scaling Clusters
Upgrades, Disaster Recovery, and Multi-Cluster Kafka

Designing Idempotent Consumers

High Priority

Premium Content

Subscribe to unlock full access to this content and more premium articles.

Exactly-Once Semanti...Designing Good Kafka...