Learn
Practice
Newsletter
Resources
Animations
New
F
Toggle theme
0
F
0
Toggle menu
Data Engineering Roadmap
What is Data Engineering & Role of a Data Engineer
Data Types & Schemas
ETL vs ELT
Foundations
Basics of Databases & Data Modeling
Batch vs Stream Processing
Relational Databases (PostgreSQL, MySQL, SQL Server)
NoSQL Databases (MongoDB, Cassandra, DynamoDB)
OLTP vs OLAP
Popular Warehouses (Snowflake, BigQuery, Redshift)
Databases & Data Warehousing
SQL Mastery (Joins, Window Functions, CTEs)
Data Warehousing Concepts
Data Lakes vs Data Warehouses
ETL Concepts
Streaming Pipelines
dbt (Data Build Tool)
Data Pipelines & ETL Tools
Batch Pipelines
Apache Airflow
Informatica / Talend / Fivetran
Hadoop Ecosystem (HDFS, MapReduce, YARN)
Spark Streaming & Structured Streaming
Parquet, ORC & Avro File Formats
Big Data Ecosystem
Apache Spark (RDDs, DataFrames, SparkSQL)
Hive & Presto
Message Queues (RabbitMQ, ActiveMQ)
Pulsar vs Kafka
Event-driven Architectures
Streaming & Messaging Systems
Apache Kafka
Real-time Stream Processing (Flink, Storm)
AWS for Data Engineering (S3, Glue, Redshift, EMR, Kinesis)
Azure for Data Engineering (Synapse, Data Lake, Event Hubs)
Orchestration & Workflow Management
Cloud Platforms & Orchestration
GCP for Data Engineering (BigQuery, Dataflow, Pub/Sub)
Containerization (Docker, Kubernetes)
Data Validation & Quality Checks
Data Catalogs
Observability & Monitoring in Data Pipelines
Data Governance & Quality
Data Lineage & Metadata Management
GDPR & Compliance
Lakehouse Architecture (Delta Lake, Iceberg, Hudi)
Streaming Joins & State Management
Cost Optimization Strategies
Advanced Topics
Data Mesh
Real-time Analytics (ClickHouse, Druid)
Data Security & Encryption
Build a Data Pipeline with Airflow + Spark
Data Warehouse for E-commerce Analytics
Log Processing System
Projects
Real-time Streaming Pipeline with Kafka + Flink
ETL Pipeline with dbt + Snowflake
IoT Data Streaming Project
SQL Query Challenges
Data Modeling Scenarios
Big Data Case Studies (Netflix, Uber, Airbnb)
Interview Preparation
ETL Design Questions
System Design for Data Pipelines
Data Engineering Roadmap
