AlgoMaster Logo

Design Instagram

High Prioritymedium24 min readUpdated June 23, 2026
Listen to this chapter
Unlock Audio

In this chapter, we will walk through the high-level design of a photo-sharing platform like Instagram.

While Instagram supports a wide range of features including direct messaging, Reels, and Stories, this article will primarily focus on the core functionality of photo and video sharing.

Let’s start by clarifying the requirements.

1. Requirement Clarification

Before diving into the design, lets outline the functional and non-functional requirements.

Functional Requirements
  1. Users can upload photos and videos.
  2. Users can add captions to their posts.
  3. Users can follow/unfollow other users.
  4. Users can like, share, and comment on posts.
  5. Support for multiple images/videos in a single post (carousel).
  6. Users can view a personalized feed consisting of posts from accounts they follow.
  7. Users can search by username and hashtag.
Non-Functional Requirements
  1. **Low Latency: **The feed should load fast (~100ms).
  2. **High Availability: **The system should be available 24/7 with minimal downtime.
  3. **Eventual Consistency: **A slight delay in users seeing the latest posts from accounts they follow is acceptable.
  4. **High Scalability: **Handle millions of concurrent users and billions of posts.
  5. **High Durability: **The uploaded photos/videos shouldn’t be lost.

Out of Scope

  1. Direct messaging.
  2. Short-form video content (Reels).
  3. Push Notifications for likes, comments, and follows.

2. Capacity Estimation

User Base

  • Total Monthly Active Users (MAUs): 2 billion
  • Daily Active Users (DAUs): → 500 million users/day

Estimating Read & Write Requests

Post Uploads (Writes)

  • 100M media uploads/day
  • Each upload generates metadata writes (DB + cache)
  • Total write requests: 100M uploads + 100M metadata writes = 200M writes/day

Feed Reads

  • Assume an average user scrolls through 100 posts per session
  • 500 million DAUs × 100 posts viewed = 50B feed requests/day
  • Assuming 80% of feed reads are served from cache, backend reads = 10B DB reads/day

Estimating Storage Requirements

Assumptions

  • 20% of DAUs (100M) upload media every day
  • 80% of uploads are photos, 20% are videos
  • Average photo size: 1MB
  • Average video size: 10 MB

Daily Storage Calculation

  • Photos: (100M × 80%) × 1 MB = 80 TB/day
  • Videos: (100M × 20%) × 10 MB = 200 TB/day
  • Total storage per day: 280 TB/day

Database Storage

  • Metadata per post: ~500 bytes (caption, timestamp, author, engagement counts)
  • Total posts in a year: 100M × 365 = 36B posts
  • Metadata storage per year: 90 TB/year

Caching Requirements

  • Hot cache size: Store recent & popular 1 billion posts
  • Assume each cached post takes 2 KB (post data + engagement counts)
  • Cache size = 2 TB for active posts (Redis/Memcached)

3. High Level Design

The diagram below maps the major components and their connections, from the clients and API Gateway through the core services, data stores, and CDN.

Components

1. Clients (Web, Mobile)

Users interact with the platform through web browsers or mobile apps. The client applications handle video playback, user interactions like likes and comments, and UI rendering, communicating with backend services through an API Gateway or Load Balancer.

2. Load Balancer / API Gateway

Acts as the single entry point for all client requests. It distributes incoming traffic across multiple service instances for high availability and scalability, and enforces rate limiting, authentication, and authorization before forwarding requests to downstream services.

3. User Service

Stores and manages user authentication, profile data, and social connections (follow/unfollow).

4. Post Service

Handles photo/video uploads and stores metadata such as caption, user info, and timestamps. It coordinates the upload of media files from the user's device to Object Storage (e.g., AWS S3) and updates metadata in a database. It also uses a message queue such as Kafka to notify the Feed Service when a new post is created.

5. Feed Service

Precomputes and stores user feeds in a high-performance cache such as Redis or Memcached to enable fast retrieval, and queries the database if a feed is not cached.

6. Engagement Service

Manages likes, comments, and shares, and writes engagement data to a high-throughput database asynchronously via a message queue.

7. Search Service

Allows users to search for other users, hashtags, and posts. It uses Elasticsearch to index and retrieve data quickly, and supports autocomplete and full-text search for a better user experience.

8. Message Queue

Decouples services and enables event-driven processing. It notifies the Feed Service of new posts and updates engagement data asynchronously.

9. Object Storage & CDN

Photos and videos are stored in distributed object storage such as S3 or Google Cloud Storage, and a CDN (Cloudflare, AWS CloudFront) ensures fast delivery globally.

4. Database Design

A large-scale content platform like Instagram requires handling both structured data (e.g., user accounts, post metadata) and unstructured/semistructured data (e.g., photos, videos, search indexes).

Typically, you’ll combine multiple database solutions to handle different workloads.

Given the requirements, we will use a relational database (e.g., PostgreSQL, MySQL) for structured data and a NoSQL database (Cassandra, DynamoDB, or Elasticsearch) for feed storage and search indexing.

4.1 Relational Database for Structured Data

Given the structured nature of user profiles and posts metadata, a relational database (like PostgreSQL or MySQL) is often well-suited.

  • Users Table: Stores user account details.
  • Posts Table: Stores metadata related to posts.
  • Media Table: Stores photo/video metadata, but not the actual files.
  • Comments Table: Stores post comments.
  • Shares Table: Stores post shares.
  • Followers Table: Maintains the follow/unfollow relationship. Stores engagement score from followers to help with ranking posts in the feed.

4.2 NoSQL Databases for High-Volume Data

While relational databases are ideal for structured data, they struggle with high-velocity writes and large scale distributed workloads. NoSQL databases like Cassandra, DynamoDB, or Redis provide horizontal scalability and high availability.

To reduce feed generation latency, a denormalized feed table stores precomputed timelines:

This table is updated asynchronously via Kafka when a user posts, and cached in Redis for quick retrieval.

Using Graph Databases for Social Connections

To support complex relationship queries, such as mutual friends, suggested followers, and influencer ranking, we can use a graph database like Neo4j or Amazon Neptune.

They efficiently model follower-following relationships with nodes and edges.

Example Query: "People You May Know"

This allows real-time friend suggestions without complex SQL joins.

4.3 Search Indexes

To support fast and scalable search queries, we can use Elasticsearch, a distributed, real-time search engine optimized for full-text searches.

Each user profile and post metadata can be stored as a document in an Elasticsearch index, allowing quick lookups and advanced filtering.

Example: Storing User Data in Elasticsearch

To support trending hashtags and keyword searches, we can store hashtags in a separate Elasticsearch index.

Example:

4.4 Media Storage

Instagram handles petabytes of photos/videos, requiring a durable and low-latency storage solution.

A distributed object storage system, such as Amazon S3, is well-suited for storing media files. It supports pre-signed URLs, enabling users to upload media directly without routing through application servers, reducing load and latency.

To ensure high durability, media files are stored in multiple replicas across different data centers, protecting against data loss.

To further optimize read latency, content can be cached closer to users using a Content Delivery Network (CDN) like Cloudflare or Amazon CloudFront. This reduces load times and improves the user experience, especially for frequently accessed media.

5. API Design

5.1 Get User Profile

Fetching a user's profile requires a valid JWT token; the response returns core account fields including follower and following counts.

Response:

5.2 Follow a User

A POST to the follow endpoint records the relationship between the authenticated user and the target account, and no request body is required.

5.3 Create a New Post

This endpoint accepts multipart form data so that both caption text and media files can be submitted in a single request.

Form Data:

Response:

5.4 Get a Post by ID

This endpoint retrieves the full metadata for a single post, including media URLs, like count, and comment count.

Response:

5.5 Get User Feed

Paginated via page and limit query parameters, this endpoint returns a personalized list of posts from accounts the authenticated user follows.

Response:

5.6 Like a Post

Sending a POST to this endpoint records a like from the authenticated user on the specified post, with no request body needed.

5.7 Comment on a Post

Posting a comment requires a JSON body containing the comment text; the endpoint accepts application/json as the content type.

5.8 Get Comments for a Post

Comments are returned in paginated order; each item includes the commenter's ID, username, text content, and timestamp.

Response:

5.9 Search Users

A GET request with the q query parameter returns matching user profiles ranked by relevance, powered by Elasticsearch under the hood.

Response:

6. Design Deep Dive

6.1 Photo/Video Upload

  1. User Initiates the Upload
    1. The user selects one or more photos or videos and enters a caption.
    2. The client (mobile app/web browser) sends an upload request to the API Gateway.
  2. API Gateway Handles the Request
    1. The API gateway authenticates and validates the request.
    2. Routes the request to the Post Service.
  3. Post Service Generates a Pre-signed URL
    1. Instead of uploading media directly through the backend, the Post Service generates pre-signed URLs from Object Storage (one per media file).
    2. It sends the pre-signed URLs back to the client.
  4. Client Uploads Media to Object Storage
    1. The client directly uploads each file in parallel to Object Storage via the pre-signed URLs.
    2. This reduces backend load and enables faster parallel uploads.
    3. Once all uploads are complete, the client sends a confirmation request to the backend with all media URLs.
  5. Post Service Saves Metadata in the Database
    1. The Post Service stores post metadata (caption, timestamp, user ID) in the Posts table and stores each media file separately in the Media Table.
  6. Kafka Publishes a "New Post" Event
    1. The Post Service sends an event to Kafka, notifying the Feed Service.

6.2 Newsfeed Generation

Since users follow both normal users and celebrities, the system must mix posts efficiently.

Fan-out-on-write (Push Model) for Normal Users

For normal users with a manageable number of followers, we use fan-out-on-write, meaning posts are pushed to followers’ feeds at the time of posting.

How It Works

  1. User A posts a new photo/video.
  2. The Post Service sends an event to Kafka, notifying the Feed Service.
  3. The Feed Service identifies the users followers (e.g., 500 followers).
  4. The post is immediately inserted into each follower’s timeline, stored in Redis (hot cache).
  5. When followers open their feeds, posts are instantly available, ensuring low-latency reads.

Example: LPUSH - Add Post to Followers’ Feeds

Suppose user 12345 (John Doe), who has 500 followers, posts a new photo. The Feed Service pushes this post to all 500 followers' feeds.

Here, John's post is pushed to the feeds of followers 56789, 67890, and 78901, along with 497 other followers.

Example: Fetching a User’s Feed (LRANGE - Get Recent Posts)

Benefits: Reads are fast since followers' feeds are pre-loaded, and the approach works well for small and medium-sized accounts.

Challenges: It becomes inefficient for users with millions of followers, such as celebrities. Writing a single post requires copying it to potentially millions of timelines, which leads to high write amplification.

Fan-out-on-read (Pull Model) for Celebrities

For celebrities and influencers, where a single post may need to reach millions of followers, preloading into every follower’s feed is impractical.

Instead, a fan-out-on-read (pull model) is used.

How It Works

  1. When a user requests their newsfeed, the Feed Service dynamically retrieves normal users’ posts from Redis (precomputed feeds) and celebrity posts from a hot cache (Redis) or a persistent store (PostgreSQL).
  2. The system merges both types of posts in real-time before serving the feed.

Benefits: It avoids massive write operations, which keeps the system scalable, and it serves fresh data when users request their feeds.

Challenges: Read latency is slightly higher than the push model, and it requires caching optimization to reduce database lookups.

Indexing New Content

  1. A New Post/User is Created
    1. A user uploads a post or creates an account.
    2. The Post/User stores metadata in the database.
    3. The Post/User Service publishes an event to Kafka.
  2. Search Service Updates Elasticsearch Index
    1. The Search Service consumes Kafka events and adds new users, posts, or hashtags to Elasticsearch.

Search Request

  1. User Initiates a Search Request
    1. The user types a query in the search bar (e.g., "john_doe" or "#travel").
    2. The client (mobile/web) sends a request. The request is routed via the API Gateway to the Search Service.
  2. Search Service Queries Elasticsearch
    1. The Search Service first checks Redis Cache for recent searches. If not found, queries Elasticsearch for relevant results.
    2. Elasticsearch performs full-text search, prefix matching and ranking based on engagement/popularity.
  3. Elasticsearch Returns Results
    1. Elasticsearch returns ranked results matching the query.
    2. The Search Service formats the response.
  4. Search Results are Cached in Redis
    1. The Search Service caches frequent queries in Redis for faster lookups.
    2. Next time a user searches for the same query, the result is served from Redis instantly.

6.4 Like, Comments and Shares

The Engagement Service processes like, comment and share requests.

It sends a Kafka event to update the DB asynchronously.

Like event:

Share event:

Comment event:

To optimize the latency for popular posts, we can cache like / share count and top comments.

7. Follow-ups

7.1 Scalability

Scalability ensures the system can handle increasing load without degrading performance.

Horizontal Scaling (Scale Out)

Use distributed databases like Cassandra or DynamoDB to spread data across nodes, and deploy multiple instances of each service behind a load balancer to handle user requests.

Sharding

Shard large datasets to split them across nodes:

  • User Data → Shard by user_id mod N
  • Posts → Shard by post_id mod N
  • Followers Table → Shard by follower_id mod N

Microservices Architecture

Break the system into independent services such as the Feed Service, Post Service, and User Service to improve maintainability and scalability. Use message queues like Kafka or RabbitMQ to handle high-throughput operations asynchronously, including notifications, updates, and feed generation.

7.2 Availability

Availability ensures that Instagram remains accessible 24/7, even in the face of failures. Given its global user base, the platform must achieve atleast 99.99% uptime.

Redundancy & Replication

Maintain replicated databases across multiple regions, such as PostgreSQL replicas and Cassandra multi-region clusters, and deploy multiple application servers across different availability zones (AZs).

Failover Mechanisms

Use automatic failover in databases, such as a leader-follower setup in PostgreSQL or multi-leader Cassandra clusters, and implement circuit breakers to gracefully degrade service if a dependency fails.

7.3 Durability

Durability ensures that data, especially user-generated content (photos, videos, comments, likes), is never lost, even in case of system failures.

Distributed Object Storage

Store media in Amazon S3 or Google Cloud Storage, which replicates data across multiple locations to prevent loss.

Database Replication & Backups

Use multi-region replication across Cassandra, DynamoDB, or PostgreSQL replicas for disaster recovery, and perform regular backups to prevent accidental data loss.

Write-Ahead Logging (WAL) & Event Sourcing

Implement WAL in databases so that changes are recorded before they are committed, and use event sourcing to log user actions like new posts and likes so that state can be rebuilt if necessary.

Quiz

Design Instagram Quiz

20 quizzes