Design an Image Search System
An image search system finds images that match a text description or an uploaded image. It converts images and queries into learned embeddings, retrieves nearby candidates from a vector index, and refines them with keyword search, reranking, and filters before returning ranked results.
Imagine opening an image search engine and typing, "A golden retriever running through a snowy forest." Within a fraction of a second, the system returns hundreds of images that match your description.
Now, instead of typing, you upload a picture of a chair and ask the system to find similar chairs. Again, it returns relevant results almost instantly.
This looks simple from the user's perspective. But behind the scenes, the system needs to understand the contents of millions, or even billions, of images, understand what the user is looking for, and efficiently find the most relevant matches.
And unlike traditional text search, we can't simply look for matching keywords. The system needs to understand visual features, objects, colors, scenes, and even relationships between different objects. This makes image search an interesting machine learning system design problem.
In this article, we'll design an image search system from scratch, following the same approach we would use in an actual Machine Learning Engineer interview. We'll start by clarifying the requirements, then explore how to represent images using embeddings, how to train and evaluate our models, and how to build a scalable retrieval and ranking system.
Along the way, we'll discuss important trade-offs involving accuracy, latency, storage, and cost.