{"title":"Design an Image Search System","description":"","content":"\n> **QUESTION**\n>\n> An image search system retrieves visually or semantically relevant images for a query, where the query can be a text string (\"red leather jacket\") or another image uploaded by the user. \n>\n> Unlike text search, the core representations are pixels, not tokens, which means the retrieval backbone rests on computer vision and multi-modal embeddings rather than inverted indexes.\n\n\n---\n\n# 1. Problem Formulation\n\nBefore committing to an architecture, lets clarify the scope. Image search covers a wide range of products, and each has different scale and quality requirements. A conversation with the interviewer helps pin down the design.\n\n### Clarifying Questions\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"What kind of image corpus are we searching over? Curated stock photos, user-generated content from a social platform, or general web images?\"\n>\n> **Interviewer:** \"General web images. Think Google Images or Bing Image Search, indexed from the open web.\"\n>\n> **Candidate:** \"Do users query with text, with another image, or both?\"\n>\n> **Interviewer:** \"Both. Text queries are the dominant path, but reverse image search is a first-class feature.\"\n>\n> **Candidate:** \"What's the scale of the image corpus and the query load?\"\n>\n> **Interviewer:** \"About 10 billion images, with 50-100 million new or updated images per day. Peak query load is around 50,000 QPS.\"\n>\n> **Candidate:** \"What's the latency requirement?\"\n>\n> **Interviewer:** \"End-to-end under 300ms at the 99th percentile. Image queries can tolerate slightly more than text queries because users expect some processing time when they upload a photo.\"\n>\n> **Candidate:** \"What safety requirements apply? Are we filtering NSFW or illegal content?\"\n>\n> **Interviewer:** \"Yes. The system must filter illegal content (CSAM) at ingestion, and support a SafeSearch toggle (strict, moderate, off) for adult and violent content at query time.\"\n>\n> **Candidate:** \"How important is personalization? Should results vary by user history?\"\n>\n> **Interviewer:** \"Modest personalization is fine (location, recent searches), but relevance should dominate. Don't rely on user profiles.\"\n>\n> **Candidate:** \"Do we have click logs and human relevance judgments?\"\n>\n> **Interviewer:** \"Yes. Billions of click and view events per day, and a smaller set of about 300K human-rated (query, image) pairs from trained annotators.\"\n\n\nWith scope pinned down, the requirements follow.\n\n### Requirements\n\n#### **Functional Requirements:**\n\n- Retrieve ranked images for text queries across both literal (\"golden retriever puppy\") and conceptual (\"cozy winter morning\") intent\n- Retrieve visually similar images for image queries uploaded by the user\n- Support filtering by resolution, aspect ratio, color, image type (photo, illustration, GIF), and license\n- Enforce SafeSearch at three tiers (strict, moderate, off)\n- Deduplicate near-identical results so the top page is diverse\n\n#### **Non-Functional Requirements:**\n\n- End-to-end p99 latency under 300ms\n- Index 10 billion images with new or updated content searchable within 10-15 minutes\n- Handle 50,000 queries per second at peak with graceful tail-latency behavior\n- Relevance quality of NDCG@10 above 0.40, with a hard floor on safety violations (zero CSAM, NSFW leakage under 0.1% at the strict setting)\n\n### ML Problem Framing\n\nThis is a multi-stage retrieval and ranking problem over a shared multi-modal embedding space. The key insight is that a well-trained model, such as CLIP or SigLIP, encodes text and images into the same vector space, so a text query and an image query both become 512-dimensional vectors that compare against the same index of 10 billion image vectors. One index, two query entry points.\n\nServing happens in stages because no single model can score 10 billion images per query under 300ms:\n\n1. **Retrieval** narrows 10B images to roughly 2,000 candidates via ANN search in the shared embedding space.\n2. **Ranking** scores those 2,000 with a more expressive cross-encoder that sees the query and each candidate jointly, plus dense features, producing the top 200.\n3. **Re-ranking** enforces diversity, applies safety filters, boosts recent content for trending queries, and produces the final top 50 displayed as a grid.\n\n\n```mermaid\nflowchart LR\n A[\"10B
Images\"]:::primary --> B[\"Retrieval
~2,000\"]:::teal\n B --> C[\"Ranking
~200\"]:::orange\n C --> D[\"Re-ranking
~50\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nAt the retrieval stage the objective is recall: the top 2,000 candidates should contain almost every image a user might find relevant. Missing a great image here can't be fixed downstream. At the ranking stage the objective flips to precision and ordering: given the 2,000 candidates, get the best images to the top. The two objectives need different model shapes, which is why the pipeline uses a lightweight dual-encoder for retrieval and a heavier cross-encoder for ranking.\n\n### Metrics\n\n#### **Offline Metrics:**\n\n\n| Metric | What It Measures | Target |\n|--------|------------------|--------|\n| NDCG@10 | Ranking quality of top 10 results, accounting for position | > 0.40 |\n| MRR | Rank of the first highly relevant image | > 0.50 |\n| Recall@2000 | Fraction of relevant images captured by retrieval | > 0.92 |\n| mAP (reverse image) | Mean average precision for image-to-image queries on a labeled near-duplicate set | > 0.80 |\n\n\nNDCG@10 is the primary offline metric for text-query relevance. Reverse image search needs its own metric because the task is different: given a query image, find the most visually similar images. mAP on a near-duplicate benchmark (e.g., Copydays, INSTRE) measures this directly. Recall@2000 separates retrieval quality from ranking quality. If retrieval only covers 70% of the truly relevant images, no amount of ranker improvement will fix the top 10.\n\n#### **Online Metrics:**\n\n\n| Metric | What It Measures | Why It Matters |\n|--------|------------------|----------------|\n| Click-through rate (CTR) | Fraction of queries where a user clicks or taps an image | Top-line attractiveness signal |\n| Image dwell time | Seconds spent on the image preview or source page | Distinguishes curiosity clicks from satisfied clicks |\n| Zero-result rate | Queries returning no safe, usable results | Signals coverage or safety-filter over-blocking |\n| Reformulation rate | Queries followed by a rephrased or refined search | Proxy for dissatisfaction |\n| Safety incident rate | User reports of harmful or inappropriate content per million queries | Compliance and trust guardrail |\n\n\nClick-through on images is noisier than on web links because users scan the grid and click quickly. Dwell time on the preview is a better satisfaction signal. Safety incident rate is a hard guardrail, not a quality metric, and it must never regress during an experiment regardless of how much relevance improves.\n\n---\n\n# 2. System Architecture\n\nThe system has two planes: an offline pipeline that ingests images, generates embeddings, and trains models, and an online pipeline that handles queries in real time. The planes communicate through shared indexes and feature stores.\n\n\n```mermaid\nflowchart TD\n subgraph Online Pipeline\n Q[Text or Image
Query]:::primary --> CACHE[Query
Cache]:::teal\n CACHE --> |miss| QU[Query
Understanding]:::teal\n CACHE --> |hit| RES[Results]:::green\n QU --> ENC[Text or Vision
Encoder]:::orange\n ENC --> ANN[ANN Retrieval
Base + Live]:::orange\n ANN --> DEDUP[Near-Duplicate
Collapse]:::teal\n DEDUP --> FF[Feature
Fetch]:::teal\n FF --> RANK[Cross-Encoder
+ GBM Ranker]:::orange\n RANK --> SAFE[Safety &
Diversity Filter]:::teal\n SAFE --> RES\n end\n\n subgraph Offline Pipeline\n IMG[Image
Corpus]:::primary --> PROC[Preprocess
& Thumbnail]:::teal\n PROC --> NSFW[NSFW &
CSAM Filter]:::orange\n NSFW --> VE[Vision
Encoder]:::orange\n VE --> IDX[(Image
Embedding Index)]:::teal\n PROC --> PH[Perceptual
Hash]:::teal\n PH --> PHIDX[(Hash
Index)]:::teal\n CLICKS[Click &
View Logs]:::primary --> TRAIN[Model
Training]:::orange\n JUDGE[Human
Judgments]:::primary --> TRAIN\n TRAIN --> VE\n TRAIN --> RANK\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nTracing a query through the online path: a text or image query arrives and hits a query cache. On a miss, query understanding runs spell correction and intent classification for text, or EXIF inspection and resizing for images. \n\nThe appropriate encoder (text or vision) produces a 512-dimensional vector, and ANN search retrieves the top 2,000 nearest vectors from both the base and live tiers in parallel. A near-duplicate collapse step groups visually identical results so the grid shows variety rather than the same image under ten different URLs. \n\nThe ranker scores the remaining candidates using dense features, and a final safety and diversity filter produces the top 50. Image queries additionally attach a perceptual hash match step upfront to handle the common case where the uploaded image already lives in the corpus.\n\nThe offline path ingests new and updated images, runs them through preprocessing (resize, normalize, thumbnail), strips unsafe content at the filter stage, then generates both embedding vectors and perceptual hashes for the two indexes. Click and view logs feed the training loop, which refreshes the vision encoder periodically and the cross-encoder ranker on a separate schedule.\n\n### Scale Numbers\n\nA back-of-the-envelope pass grounds every decision below.\n\n#### **Query load**\n\nAt 50,000 peak QPS with a Gaussian-ish daily curve, average QPS is roughly 15,000-20,000. Daily volume lands at 1.5-2 billion queries. Roughly 10-15% of those are image uploads (reverse image search), which are more expensive to serve because they require a full vision-encoder forward pass at query time.\n\n#### **Embedding storage**\n\nTen billion images, each a 512-dim float32 vector, is `10,000,000,000 × 512 × 4 bytes = 20.5 TB` for raw embeddings. Product quantization at 64 bytes per vector compresses this to roughly 640 GB. The HNSW graph and PQ codebooks add another 20-30%, so the serving index lands near 0.8-1.0 TB, which is feasible to shard in-memory across a few dozen nodes.\n\n#### **Raw image storage**\n\nTen billion images at an average of 200 KB each is 2 PB. This lives in blob storage (object store backed by SSD for hot access, standard storage for cold), never on the serving hot path. Serving fetches thumbnails on demand.\n\n#### **Thumbnail tier**\n\nFor each image the system maintains 2-3 pre-generated thumbnails (grid size ~300px, preview size ~800px, full size ~1200px). At roughly 20 KB per grid thumbnail, the hot thumbnail tier is ~200 TB and is served through a CDN with a high cache hit rate because the head of the query distribution surfaces the same images repeatedly.\n\n#### **Query-time GPU budget**\n\nA ViT-B/16 vision encoder has about 86M parameters. On a modern A100 GPU it processes a 224×224 image in 1-3ms with small-batch serving. At 50K QPS with 10% image queries (5K image-query QPS) and 2ms per image, one GPU handles ~500 QPS, so the image-query path needs 10-15 GPUs plus redundancy.\n\nThe text encoder (a distilled BERT-like tower) is cheaper, 0.5-1ms per query, needing 10-20 GPUs at peak. The cross-encoder ranker is the biggest consumer: scoring 2,000 candidates per query at 20-30ms per query with batching puts it at 40-80 GPUs. Total online GPU fleet lands at 70-120.\n\n#### **Training data volume**\n\nClick and view logs at 2B queries per day with an average of 4-5 image impressions viewed per query yield roughly 10 billion (query, impression, action) records daily. After filtering for high-confidence clicks with dwell over 5 seconds, 80-200M strong positive pairs per day are available for training.\n\nThese numbers set the hard edges of the design: the base vector index must be quantized, raw pixels stay off the hot path, image queries are first-class cost centers, and the ranker dominates GPU spend.\n\n### Data Flow\n\n\n```mermaid\nflowchart LR\n subgraph Ingestion\n RAW[Raw
Images]:::primary --> NORM[Decode &
Resize]:::teal\n NORM --> SAFE[NSFW &
CSAM Check]:::orange\n SAFE --> EMB[Vision
Encoder]:::orange\n SAFE --> HASH[Perceptual
Hash]:::orange\n end\n\n subgraph Index Build\n EMB --> VIDX[(ANN
Index)]:::green\n HASH --> HIDX[(Hash
Index)]:::green\n end\n\n subgraph Training Loop\n CLOGS[Click
Logs]:::primary --> LBL[Label
Extraction]:::teal\n LBL --> PAIRS[Training
Pairs]:::teal\n PAIRS --> CLIP[Dual-Encoder
Training]:::orange\n PAIRS --> CET[Cross-Encoder
Training]:::orange\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nIngestion runs continuously. Fresh images reach the index within 10-15 minutes via an incremental pipeline that writes to a live tier, and a weekly base rebuild re-encodes the full corpus to pick up model updates. \n\nThe training loop runs on its own cadence: the dual-encoder refreshes every 2-4 weeks, and the cross-encoder refreshes every 4-8 weeks because it sees query-image interactions and is less sensitive to corpus drift.\n\n---\n\n# 3. Data\n\n### Data Sources\n\nFive sources feed the system:\n\n1. **Image corpus.** The raw images with their URLs, source pages, dimensions, format (JPEG, PNG, WebP, GIF, SVG), file size, EXIF metadata (camera, timestamp, optional GPS), and license information. The crawler refreshes at different cadences: news sites every few hours, e-commerce catalogs daily, long-tail web pages weekly or monthly.\n2. **Surrounding text.** Each image typically comes with alt-text, a caption, the page title, nearby paragraph text, and anchor text from inbound links that point to the image. This text is the single most important signal for text-query retrieval and must be captured at ingestion.\n3. **Click and view logs.** Billions of (query, impression_position, clicked, dwell_time, reformulated) records per day. These are the primary training signal. A click on an image at grid position 5 with 20 seconds of dwell on the source page is a strong positive. A click at position 1 with a 2-second bounce is likely a negative.\n4. **Human relevance judgments.** Roughly 300K (query, image, relevance_grade) tuples rated by trained annotators on a 4-point scale (perfect, good, acceptable, bad). Judges also tag safety violations and image-quality issues. Expensive but unbiased.\n5. **Co-click and co-view graph.** When users click multiple images for the same query within a short window, those images are similar in the user's eye even if their pixels differ. Aggregated across millions of sessions, this graph supplies a behavioral notion of visual similarity that complements the pixel-based encoder.\n\n### Image Representation\n\nRaw images are not ready to be consumed by a neural network. Decoding, resizing, normalization, and format handling all sit between the byte stream and the encoder. Mistakes here silently degrade retrieval quality because the online query image and the offline indexed image must go through identical transforms.\n\n\n```mermaid\nflowchart LR\n RAW[Raw Bytes
JPEG/PNG/WebP/GIF]:::primary --> DEC[Decode]:::teal\n DEC --> ORIENT[EXIF
Orientation]:::teal\n ORIENT --> RESIZE[Resize to
256 short side]:::orange\n RESIZE --> CROP[Center Crop
224 x 224]:::orange\n CROP --> NORM[Normalize
ImageNet mean/std]:::teal\n NORM --> ENC[Vision
Encoder]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nA few details matter more than they look. EXIF orientation is not applied by every decoder, so a photo that looks upright in a browser can arrive at the model rotated ninety degrees. Apply orientation before resizing. For transparent PNGs, flatten onto a neutral gray background because flattening onto white biases the encoder toward bright scenes.\n\nFor animated GIFs, index only the middle frame (motion rarely changes the semantic content, and indexing every frame multiplies storage). For SVGs, rasterize at the target resolution. Keep everything in sRGB color space. Mixed color spaces (Display P3 photos from iPhones, Adobe RGB scans) produce subtly different embeddings if not normalized.\n\nResolution trades quality for cost. At 224×224, a ViT-B/16 runs in 1-3ms per image. At 384×384 the same model needs 4-8x more compute but captures finer detail (fabric texture, small text, subtle color differences). Production systems often train and serve at 224×224 for general search, and reserve 384×384 for a high-fidelity tier used by shopping or professional search where the extra detail is worth the cost.\n\n### Multi-Field Signals\n\nImage search is rarely pure image search. A web image exists in a context: alt-text, caption, page title, anchor text, and body text near the image all carry signal. Ignoring them leaves obvious wins on the table.\n\n\n| Field | Typical Reliability | Weight in Retrieval | Notes |\n|-------|---------------------|---------------------|-------|\n| Alt-text | High when present | Strong | Describes the image directly; missing on 30-60% of web images |\n| Caption | High | Strong | Short, descriptive, often curated |\n| Page title | Medium | Medium | Topical but not image-specific |\n| Anchor text of inbound links | Medium | Medium | Condenses how others describe the image |\n| Body text near image | Low | Weak | Noisy; nearby does not imply about |\n| EXIF and filename | Low | Very Weak | Occasionally useful (brand, location) but often junk |\n\n\nProduction systems fuse these signals in two places. At retrieval, the text query matches against a text vector that represents the concatenated surrounding text fields, which gives a second retrieval path alongside the image-embedding path. At ranking, per-field match features (alt-text query overlap, caption match, title match) feed the ranker so it can reason about which field matched and how strongly. \n\nGoogle Images and Bing Image Search both lean heavily on surrounding text for text queries. Purely visual retrieval (query text → image embedding directly via CLIP) works well for conceptual queries but loses to text-based retrieval for named-entity queries like \"Barcelona FC logo\" where the surrounding text pins the image to an entity the visual model might miss.\n\n### Feature Engineering\n\nThe ranker needs features from three buckets. Pure embedding similarity is not enough to sort the final candidate set, because it doesn't know about resolution, source reputation, or user behavior.\n\n**Query features** describe the query itself:\n\n- Query length (number of tokens for text, aspect ratio for image)\n- Predicted intent (navigational, informational, shopping, people, scenery)\n- Is the query a named entity (person, logo, place)\n- Detected language\n- For image queries: dominant colors, detected object categories, detected text (OCR)\n\n**Image features** describe static properties of each image:\n\n- Resolution and aspect ratio\n- Aesthetic score from a learned quality model\n- JPEG quality / compression level\n- Dominant colors (5-bin histogram)\n- Detected object categories from an object detector\n- Source domain authority (page-level PageRank)\n- Image age (days since first crawl)\n- Historical click-through rate for this image across all queries\n\n**Cross features** describe the query-image interaction:\n\n- CLIP similarity score (visual embedding dot product with query embedding)\n- Surrounding-text BM25 score\n- Alt-text match score\n- Page title match score\n- OCR match score (is the query text literally present in the image)\n- Co-click affinity from the behavioral graph\n\n### Feature Table\n\n\n| Feature | Type | Source | Freshness | Description |\n|---------|------|--------|-----------|-------------|\n| query_length | Int | Query | Real-time | Tokens in text query, or aspect ratio for image query |\n| query_intent | Categorical | Intent classifier | Real-time | navigational / informational / shopping / people / scenery |\n| query_entity | Categorical | NER | Real-time | Detected named entity class, if any |\n| image_width_px | Int | Ingestion | At ingestion | Pixel width of the indexed image |\n| image_aspect | Float | Ingestion | At ingestion | Aspect ratio (width / height) |\n| image_aesthetic | Float | Quality model | Weekly | Learned aesthetic score from a small CNN, 0-1 |\n| image_colors_top5 | Vector | Color extractor | At ingestion | Dominant 5-color histogram |\n| image_objects | Vector | Object detector | At ingestion | Multi-hot vector over 1,000 object classes |\n| domain_authority | Float | Link graph | Weekly | Source page authority, 0-1 |\n| image_age_days | Int | Crawler | Daily | Days since first crawl |\n| image_historical_ctr | Float | Click logs | Daily | Aggregated CTR across all queries, last 30 days |\n| clip_similarity | Float | Vision + text encoder | Real-time | Dot product of query and image embeddings |\n| alt_text_bm25 | Float | BM25 engine | Real-time | BM25 over alt-text field |\n| surrounding_text_bm25 | Float | BM25 engine | Real-time | BM25 over concatenated surrounding text |\n| ocr_match | Float | OCR + matcher | Real-time | Fraction of query tokens present in OCR-extracted image text |\n| co_click_affinity | Float | Behavioral graph | Daily | Log-weighted co-click score between query and image |\n\n\nFeatures like `image_colors_top5` and `image_objects` are computed once at ingestion and cached. The query-dependent features (`clip_similarity`, `bm25`, `ocr_match`) are computed per request and dominate feature-fetch latency.\n\n---\n\n# 4. Model\n\n### Baseline Approach\n\nA respectable baseline uses a pretrained vision backbone (ResNet50 trained on ImageNet) to produce an image embedding, and a text encoder (TF-IDF or a small sentence encoder) to produce a text embedding over the surrounding text. \n\nFor a text query, retrieve the top images whose surrounding text matches via BM25, then re-rank by image-aesthetic score and source authority. For an image query, retrieve by cosine similarity against the pretrained ResNet50 vectors.\n\nThis baseline works for narrow cases. For an image query that's a near-duplicate of something in the corpus, ResNet50 similarity surfaces it. For a text query where the desired image happens to have strong alt-text, BM25 surfaces it. \n\nWhere the baseline collapses is on conceptual text queries (\"nostalgic summer evening\", \"minimalist kitchen\") and on image queries that need semantic rather than pixel similarity (\"find more images of the same landmark in different weather\"). The ResNet50 embedding encodes textures and objects, not concepts, and there's no shared space between text and images, so the system can't route text queries through the image embedding at all.\n\n### Advanced Approach\n\nThe production design uses a dual-encoder trained contrastively on massive (image, text) pairs, in the style of CLIP or SigLIP. One transformer encodes images, another encodes text, and the two are trained so that a matching (image, caption) pair sits close in the shared embedding space while mismatched pairs sit far apart.\n\nAfter training, a text query and an image query both produce a 512-dimensional vector in the same space, so a single ANN index over image vectors serves both modalities.\n\n\n```mermaid\nflowchart LR\n subgraph Dual-Encoder\n IT[Image]:::primary --> IVE[Vision
Transformer]:::teal\n IVE --> IV[\"image_vec
(512-d)\"]:::primary\n TT[Text]:::primary --> TVE[Text
Transformer]:::teal\n TVE --> TV[\"text_vec
(512-d)\"]:::primary\n IV --> DOT[\"dot(image, text)\"]:::orange\n TV --> DOT\n DOT --> LOSS[Contrastive
Loss]:::green\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe two towers don't share weights. Text and images have entirely different structure (tokens vs pixels, different distributional statistics), so each encoder specializes on its modality. \n\nAt query time, only the relevant encoder runs: the text tower for text queries, the vision tower for image queries. Image vectors are precomputed offline and live in the ANN index, so the online cost is dominated by encoding the query and running the nearest-neighbor search.\n\n#### **Vision encoder choice**\n\nThe vision encoder is the biggest single lever for cost, quality, and latency. Choices range from small CNNs to large transformers, and the pick depends on the scale and budget.\n\n\n| Encoder | Params | Latency (A100, 224×224) | Embedding Quality | Notes |\n|---------|--------|-------------------------|-------------------|-------|\n| ResNet50 | 25M | 0.5-1ms | Baseline | Cheap, ImageNet-pretrained, shows its age on fine-grained tasks |\n| EfficientNet-B3 | 12M | 0.7-1.5ms | Good | Strong accuracy per param; slower per FLOP due to depthwise convs |\n| ViT-B/16 (CLIP) | 86M | 1-3ms | Strong | Industry default; flexible and scales well |\n| ViT-L/14 (CLIP) | 307M | 4-8ms | Excellent | Used by OpenAI CLIP; significantly better retrieval but 3-4x cost |\n| SigLIP-SO400M | 400M | 6-12ms | State of the art | Sigmoid loss scales better than softmax; Google uses this for Gemini vision |\n\n\nThe design here uses ViT-B/16 as the primary vision encoder because it balances quality and cost. ViT-L/14 sits behind a feature flag for a high-fidelity tier used by shopping search. SigLIP-SO400M is the frontier choice when quality matters more than GPU bill.\n\n#### **ANN index design**\n\nRetrieval only works if the index returns top-K nearest neighbors across 10B vectors in well under 20ms. Brute-force search is out: one query against 10B 512-dim vectors is 5 trillion multiplications per query, which no GPU fleet will absorb at 50K QPS. The index uses HNSW over product-quantized vectors, the same recipe as most large-scale retrieval systems.\n\n\n| Approach | Recall@1000 | Latency | Memory | Fit |\n|----------|-------------|---------|--------|-----|\n| Brute force | 100% | Seconds | 20 TB | No |\n| HNSW (full precision) | 97-99% | 1-2ms | 20+ TB in RAM | Too expensive |\n| IVF + Flat | 85-92% | 5-10ms | 20+ TB | Lower recall, still expensive |\n| HNSW + PQ (64 bytes/vec) | 90-95% | 2-5ms | 0.8-1 TB | Production choice |\n| IVF + PQ | 80-88% | 3-8ms | 0.8-1 TB | Faster to build, lower recall |\n\n\nEach 512-dim float32 vector (2 KB raw) is split into 64 sub-vectors of 8 floats, and each sub-vector is replaced by an 8-bit codebook index trained via k-means. That's 64 bytes per image, a 32x compression. \n\nDistance lookups use precomputed tables so quantized search is actually faster than raw search, not slower. The recall trade-off (90-95% vs 97-99% for full HNSW) is acceptable because the cross-encoder downstream can rescue candidate-order mistakes.\n\nOne quirk of image embeddings: they cluster tighter than text embeddings because natural image distributions have strong modes (faces, logos, landscapes, product photos). \n\nHNSW tuning parameters (`M`, `efConstruction`, `efSearch`) typically need to be pushed higher than for text retrieval to avoid getting trapped in dense local regions. A common starting point is `M=48`, `efConstruction=200`, `efSearch=128`.\n\n#### **Cross-encoder ranking**\n\nThe second stage scores the top 2,000 candidates with a model that sees the query and the candidate jointly. For text queries, a vision-language cross-encoder like a fine-tuned BLIP-2 reranker takes the query tokens and the image patch tokens through cross-attention, producing a relevance score from a CLS token. \n\nFor image queries, the \"query\" side is another set of vision tokens from the query image, and the cross-encoder learns to score candidate-to-query pair similarity that captures visual correspondence beyond a single global dot product.\n\n\n```mermaid\nflowchart LR\n QT[\"Query
(text or image)\"]:::primary --> CENC[Cross-Encoder
Transformer]:::teal\n CT[Candidate
Image Tokens]:::primary --> CENC\n CENC --> CLS[CLS
Embedding]:::teal\n CLS --> CES[CE Score]:::orange\n DF[Dense Features
resolution, aesthetic,
domain, CTR, ...]:::primary --> MERGE[Feature
Concat]:::teal\n CES --> MERGE\n MERGE --> GBM[Gradient-Boosted
Ranker]:::orange\n GBM --> SCORE[Final
Score]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe cross-encoder's relevance score alone isn't the final output. It captures query-image fit from pixels and text but can't see the features from §3 (resolution, aesthetic score, source authority, historical CTR, behavioral co-click). \n\nThose signals carry information pixels can't reveal, so the cross-encoder score is concatenated with the dense features and fed to a gradient-boosted tree model (LightGBM or XGBoost) trained on human relevance labels and click-derived pseudo-labels.\n\nThis two-stage setup keeps the expensive cross-encoder focused on what it does best (visual-textual fit) and leaves feature interactions to a cheap tree model that's easy to retrain.\n\nThe three architecture patterns (dual-encoder, late-interaction, cross-encoder) sit on a quality-cost spectrum:\n\n\n| Property | Dual-Encoder (CLIP) | Late-Interaction (FILIP) | Cross-Encoder (BLIP-reranker) |\n|----------|---------------------|--------------------------|-------------------------------|\n| Query-image interaction | None (independent encoding) | Token-level MaxSim at query time | Full cross-attention |\n| Inference cost per pair | ~0.1ms | ~1-2ms | ~10-20ms |\n| Document precomputation | Single vector per image | Many vectors per image (one per patch token) | Not precomputable |\n| Scales to billions of images | Yes (via ANN) | Partial (index size explodes) | No, ranking only |\n| Relevance quality | Good | Very good | Excellent |\n| Pipeline role | Retrieval | Middle-stage re-ranker | Final ranking |\n\n\nThe design here pairs dual-encoder retrieval with cross-encoder ranking. Late-interaction models are worth considering when the ranking stage is cost-constrained and a larger index footprint is acceptable. Pinterest's visual search has used late-interaction patterns to push retrieval quality up without paying for a full cross-encoder per query.\n\n### Training\n\n#### **Dual-encoder training**\n\nThe loss is a symmetric InfoNCE: for a batch of N (image, text) pairs, the model learns to make each image vector closest to its own text vector and far from the other N-1 text vectors, and vice versa. \n\nLarge batch sizes are critical because the quality of contrastive learning scales with the number of in-batch negatives. CLIP used a batch size of 32,768. In-house systems at major companies often train with batch sizes of 16K-64K on hundreds of GPUs for weeks.\n\n\n```python\n# Symmetric InfoNCE contrastive loss for dual-encoder training\ndef clip_loss(image_vecs, text_vecs, temperature=0.07):\n image_vecs = F.normalize(image_vecs, dim=-1)\n text_vecs = F.normalize(text_vecs, dim=-1)\n logits = image_vecs @ text_vecs.T / temperature\n labels = torch.arange(logits.size(0), device=logits.device)\n loss_i2t = F.cross_entropy(logits, labels)\n loss_t2i = F.cross_entropy(logits.T, labels)\n return (loss_i2t + loss_t2i) / 2\n```\n\n\nTraining happens in two phases. Phase one is large-scale pretraining on web-scraped (image, alt-text) pairs, typically billions of pairs, which teaches the model broad visual-linguistic correspondence. Phase two is fine-tuning on domain-specific data: query-click pairs from the search product, human-labeled relevance data, and curated high-quality captions. \n\nHard-negative mining matters here. In-batch negatives are mostly easy (a random image in a batch rarely matches a given query's text), so after initial training, the system mines hard negatives from its own top-K retrieval results (images that rank high but weren't clicked) and adds them to training batches. Three to five rounds of hard-negative mining is typical.\n\n#### **Cross-encoder training**\n\nThe cross-encoder trains on a mix of human judgments (300K labeled pairs) and click-derived signals. For each query, candidates with different relevance grades or click outcomes form training tuples, and the model learns to rank them correctly with a pairwise or listwise loss. \n\nLambdaMART-style losses work well because they directly optimize NDCG. The cross-encoder needs fewer examples than the dual-encoder (hundreds of thousands vs billions) because it sees the full interaction between query and image and learns more from each example.\n\n---\n\n# 5. Serving\n\n### Inference Pipeline\n\nA single query flows through the system along one of two paths depending on its modality. The paths converge after the query encoding step and share all downstream stages.\n\n\n```mermaid\nflowchart LR\n Q[Query]:::primary --> |text| TEXT[Text
Encoder
5ms]:::teal\n Q --> |image| VIS[Vision
Encoder
20ms]:::orange\n TEXT --> ANN[ANN
Retrieval
~2000 images
15ms]:::orange\n VIS --> PHFILT[Perceptual
Hash Match
5ms]:::teal\n PHFILT --> ANN\n ANN --> DEDUP[Near-Duplicate
Collapse
5ms]:::teal\n DEDUP --> FF[Feature
Fetch
10ms]:::teal\n FF --> RANK[Cross-Encoder
+ GBM Rank
60ms]:::orange\n RANK --> SAFE[Safety
Filter
5ms]:::teal\n SAFE --> DIV[Diversify &
Re-rank
10ms]:::teal\n DIV --> THUMB[Thumbnail
Fetch
10ms]:::teal\n THUMB --> RES[Top 50]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nText queries take the cheaper path: the text encoder is a distilled transformer that runs in 5ms or less. Image queries are more expensive because the full vision encoder runs per request (20ms for ViT-B/16 including decode and preprocessing). \n\nAn image query also runs a perceptual hash lookup in parallel: if the uploaded image matches something already in the corpus (very common, since users often upload an image they found somewhere on the web and want to know its source), the hash match returns the exact URL within 5ms and can short-circuit ANN retrieval for that subtree. Both paths converge at the ANN step, and downstream processing is modality-agnostic.\n\nA query cache sits in front of the pipeline. For text queries, the cache key is the normalized query text plus SafeSearch level and coarse user context (location, language). The head of the query distribution is heavy-tailed: the top few thousand text queries account for 15-25% of traffic and their top results change slowly, so caching responses for 5-30 minutes cuts backend load substantially and improves p50 for common queries. \n\nImage queries are harder to cache because the uploaded image is typically unique, but the perceptual hash of the uploaded image can serve as a cache key with high hit rates for viral or meme content.\n\n### Batch vs Real-Time\n\nDifferent components run in different modes based on cost and freshness requirements.\n\n\n| Component | Mode | Why |\n|-----------|------|-----|\n| Image embedding (vision encoder) | Batch | 10B images, precompute once, update incrementally |\n| ANN index build | Batch (weekly) + incremental (minutes) | Full rebuild is expensive; live tier handles new content |\n| Perceptual hash index | Near-real-time | New images get a hash at ingestion and appear in the hash index within minutes |\n| Query encoding (text or vision) | Real-time | Depends on the query; cannot precompute |\n| Cross-encoder ranking | Real-time | Depends on query-image pair; cannot precompute |\n| Static features (resolution, aesthetic, object tags) | Batch | Computed once at ingestion, cached in the feature store |\n| Dynamic features (CLIP similarity, BM25, OCR match) | Real-time | Query-dependent |\n| Domain authority, historical CTR | Batch (daily/weekly) | Aggregated over long windows, updated on a schedule |\n| NSFW / CSAM filtering | Ingestion + real-time | Ingestion filter prevents storage; query-time filter handles edge cases and policy updates |\n\n\nThe biggest operational headache is keeping the embedding index fresh when the vision encoder itself changes. A model refresh invalidates every precomputed vector and forces a full re-encode of 10B images, which takes 12-24 hours on a dedicated GPU cluster. \n\nThis is much more expensive than a typical text-search embedding refresh and drives the migration pattern discussed in Monitoring.\n\n### Latency Budget\n\n\n| Component | Budget (p99) | Notes |\n|-----------|--------------|-------|\n| Query understanding | 5ms | Spell correction, intent, SafeSearch decision |\n| Text encoder | 5ms | Distilled transformer, only on text queries |\n| Vision encoder | 25ms | ViT-B/16 at 224×224, only on image queries |\n| Perceptual hash lookup | 5ms | Image queries only; runs in parallel with vision encoder |\n| ANN retrieval | 15ms | Sharded HNSW+PQ, fan-out with hedging |\n| Near-duplicate collapse | 5ms | Group images by cluster ID |\n| Feature fetch | 10ms | Parallel lookups against feature store |\n| Cross-encoder ranking | 60ms | Largest bucket; 2,000 candidates batched on GPU |\n| Safety filter | 5ms | Per-candidate NSFW score lookup + threshold |\n| Diversify and re-rank | 10ms | Grid diversity, freshness boost, domain cap |\n| Thumbnail fetch | 10ms | CDN hit for display URLs |\n| Network overhead | 10ms | Internal RPCs between services |\n| **Total (text query)** | **~145ms** | Leaves 150ms of headroom below 300ms p99 |\n| **Total (image query)** | **~165ms** | Adds vision-encoder cost |\n\n\nThe budget intentionally targets ~150ms median rather than the full 300ms requirement. The headroom absorbs tail variance: GC pauses, shard stragglers, cold feature-store reads, and transient network hiccups each commonly add 20-50ms at the 99th percentile. Budgeting directly to the SLA would mean routine tail violations.\n\nThe cross-encoder is the dominant cost. If it becomes a bottleneck, the first lever is reducing the candidate set from 2,000 to 1,000, roughly halving ranking time. The second is distilling the cross-encoder into a smaller model. The third, and most architectural, is inserting a cheap middle-stage re-ranker (late-interaction or a small MLP on dense features) to trim 2,000 down to 500 before the expensive cross-encoder runs.\n\n### Sharding and Distribution\n\nAt 10 billion images and a 1 TB compressed index, the embedding index does not fit on a single node. The perceptual hash index is smaller (a few hundred bytes per image, so 1-2 TB for the full corpus) but also shards for query parallelism. Every query fans out to all shards, each shard searches its slice, and a root aggregator merges top-K candidates globally.\n\n\n```mermaid\nflowchart TD\n Q[Query Vector]:::primary --> R[Query
Router]:::teal\n R --> S1[Shard 1
ANN + Hash]:::orange\n R --> S2[Shard 2
ANN + Hash]:::orange\n R --> S3[Shard ...]:::orange\n R --> SN[Shard N
ANN + Hash]:::orange\n S1 --> |top-150| AGG[Root
Aggregator]:::teal\n S2 --> |top-150| AGG\n S3 --> |top-150| AGG\n SN --> |top-150| AGG\n AGG --> |global top-2000| RANK[Ranking
Stage]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Sharding strategy**\n\nShard by image ID, not by embedding region. Region-based sharding (routing a query to the shard whose centroids are closest) is tempting because it reads fewer shards per query, but it creates load-balancing nightmares as the query distribution shifts. \n\nImage-ID sharding is the industry default: every query hits every shard, but each shard handles a uniform slice, load is naturally balanced, and adding capacity means adding shards rather than rebalancing centroids. At 25 shards and 400M images per shard, each shard holds 25-40 GB of compressed index plus HNSW graph, which fits comfortably in the RAM of a mid-sized machine.\n\n#### **Fan-out and tail latency**\n\nWith 25 shards and a 15ms per-shard budget, the retrieval step completes in `max(shard_latency) + merge_time`, not `sum`, provided there's enough network bandwidth. This assumes every shard responds quickly. \n\nA single slow shard drags the whole query. Mitigation uses two techniques. First, hedged requests: after the p95 of typical shard latency (say 12ms), the router sends a backup request to a replica and takes whichever response arrives first. \n\nHedging costs a small extra load (a few percent of requests are duplicated) and eliminates most of the long tail. Second, partial-result tolerance: each shard can return partial results if it runs out of time, and the aggregator accepts a few missing shards rather than blocking. Losing 1 of 25 shards drops recall by roughly 4% in the worst case, which is acceptable when the alternative is a 300ms timeout.\n\n#### **Replication and capacity**\n\nEach shard runs on at least three replicas across availability zones. The router load-balances reads across replicas and fails over on timeout. For writes (new image ingestion), updates apply to the leader and asynchronously replicate, with staleness bounded to under a few seconds. At 50K peak QPS and 25 shards with 2.5-3x replication, the retrieval tier needs 75-100 nodes. \n\nAdd 70-120 GPU nodes for query encoding and cross-encoder ranking, plus a feature-store tier and a CDN layer, and the online serving footprint lands at roughly 250-350 nodes. Cost is dominated by GPUs, as it usually is for deep-learning-heavy retrieval systems.\n\n---\n\n# 6. Deep Dives\n\n### Near-Duplicate and Exact-Duplicate Detection\n\nThe same image appears on the web in dozens of variants: cropped thumbnails, watermarked versions, JPEG reencodes at different quality levels, resized copies, color-adjusted edits, and frames with different aspect ratios.\n\nA naive retrieval ranks all of them as near-top matches because their embeddings sit right next to each other, which produces a result grid that looks embarrassing: the same photograph eight times with slight differences. Handling duplicates is a retrieval problem, a ranking problem, and a storage problem at once.\n\n\n```mermaid\nflowchart LR\n NEW[New
Image]:::primary --> PHASH[Perceptual
Hash]:::teal\n PHASH --> HLOOK[Hash
Index Lookup]:::orange\n HLOOK --> |exact match| TAG1[Tag as
Exact Dup]:::green\n HLOOK --> |no match| EMB[Embed]:::teal\n EMB --> ELOOK[ANN
Near-Neighbor]:::orange\n ELOOK --> |sim > 0.97| TAG2[Tag as
Near Dup]:::green\n ELOOK --> |sim <= 0.97| UNIQ[Unique
Cluster]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThree techniques handle different parts of the problem:\n\n\n| Technique | What It Catches | Cost | Weakness |\n|-----------|-----------------|------|----------|\n| Cryptographic hash (SHA-256) | Byte-identical duplicates | Trivial | Breaks under any pixel modification, including reencode |\n| Perceptual hash (pHash, dHash, aHash) | Visually similar images under resize, minor color shifts, JPEG recompression | Very cheap (hashing + Hamming distance) | Fails on crops, rotations, strong edits |\n| Deep hashing / embedding similarity | Semantic near-duplicates, different crops of the same photo, color-adjusted variants | GPU encoder + ANN lookup | Expensive per image |\n\n\nProduction systems combine all three. At ingestion, every image gets a SHA-256 hash (for exact match dedup in storage) and a perceptual hash stored in a dedicated Hamming-distance index (typically an LSH-based structure). \n\nThe CLIP embedding also gets stored. At query time, the near-duplicate collapse step clusters retrieved candidates: first by perceptual hash (Hamming distance under 6 on a 64-bit pHash groups near-identical images), then by embedding similarity (cosine similarity above 0.97 groups semantic near-duplicates).\n\n\n```python\n# Perceptual-hash based near-duplicate clustering on retrieved candidates\nfrom collections import defaultdict\n\ndef cluster_near_duplicates(candidates, phash_threshold=6, sim_threshold=0.97):\n clusters = defaultdict(list)\n for cand in candidates:\n placed = False\n for head_id, members in clusters.items():\n head = members[0]\n if hamming(cand.phash, head.phash) <= phash_threshold:\n members.append(cand); placed = True; break\n if cosine(cand.emb, head.emb) >= sim_threshold:\n members.append(cand); placed = True; break\n if not placed:\n clusters[cand.id].append(cand)\n return [max(members, key=lambda c: c.quality_score) for members in clusters.values()]\n```\n\n\nA cluster is collapsed in the result grid: only the best representative is shown, and the others are accessible via a \"see more sizes\" link. \"Best\" is a ranking decision driven by resolution, source authority, and license clarity. Wikimedia Commons beats a random blog with the same photo. A 1024×768 version beats a 320×240 thumbnail.\n\nThe trade-off is the dedup threshold. Too aggressive (cluster at similarity 0.85), and legitimately different images get grouped together; the grid loses variety. Too loose (cluster at 0.99), and the same photo shows up five times.\n\nDifferent products tune this differently: Google Images is aggressive because the grid is a preview into the web and users want variety, while reverse image search tools like TinEye are loose because users want to find every copy of the image.\n\nOne subtle detail often missed: near-duplicate detection at ranking time is necessary even with dedup at ingestion, because a query can surface 2,000 candidates that happen to include multiple clusters. The per-query collapse runs against the retrieved candidate set, not the whole corpus, so it's cheap (O(k log k) for k candidates) but essential.\n\n### Handling the Two Query Modalities\n\nText-to-image and image-to-image retrieval look symmetric (both produce a query vector, both query the same ANN index), but their failure modes are very different.\n\n**Text queries are ambiguous.** \"Jaguar\" might mean the car or the animal. \"Apple\" is the fruit or the company. \"Pitch\" is sports, sales, or sound. The text encoder picks one interpretation and the retrieval reflects it, so users whose intent differs see irrelevant results. \n\nThe mitigation is upstream: intent classification tags the query, query expansion generates paraphrases for each plausible intent, and the retrieval fans out to both interpretations with separate ANN queries. The top-K from each intent merge via rank fusion, and the ranker learns which intent matches the click pattern. Pinterest publishes their use of a similar pattern for ambiguous visual search queries.\n\n**Image queries are visually concrete but semantically underspecified.** A user uploads a photo of a red Nike running shoe on a wooden floor with morning light. What do they want? Similar red shoes? Similar Nike shoes? Similar flooring? Similar lighting? The raw image embedding encodes all of these at once, and a naive retrieval returns a mix that often misses the user's actual intent. Four strategies help:\n\n1. **Region proposals.** An object detector (a lightweight model like YOLO or DETR-small) runs over the query image and identifies salient objects. The UI offers chips: \"Search shoe\", \"Search floor\", \"Search person\". The user picks one, and the system re-runs retrieval using only the cropped region's embedding. Pinterest Lens and Google Lens both use this pattern, and it dramatically improves satisfaction on multi-object images.\n2. **Attribute-aware embeddings.** Beyond a single global image embedding, index separate embeddings per attribute: color, category, style. At query time, the user can toggle which attribute dominates, effectively reweighting the search. This is standard in e-commerce visual search (Amazon, eBay, ASOS).\n3. **Multi-vector indexing.** Store multiple embeddings per image, one per detected object. Index size balloons, typically 3-5x, but retrieval gets much more precise. Worth the storage for product-heavy corpora.\n4. **Query-time attention masks.** The user draws a box or taps a region on the uploaded image, and the encoder computes the embedding only over that region. Most intuitive, and it's the direction mobile visual search has taken.\n\nThe design here supports region proposals as the default UI for image queries, falls back to whole-image embedding when no object is detected clearly, and keeps multi-vector indexing as a feature-flagged capability for shopping search. The cost is a modest object detector at ingestion (~5ms per image, adding to the embedding cost) and at query time (~8ms per image query).\n\n### Safety and NSFW Filtering\n\nContent safety is not a \"nice to have\" for an image search system. It's a hard requirement with legal and reputational consequences. The system enforces safety at two points: at ingestion, before content reaches the index, and at query time, as a re-ranking filter.\n\n\n```mermaid\nflowchart LR\n IN[Incoming
Image]:::primary --> HASH[CSAM Hash
Check]:::orange\n HASH --> |match| REJ1[Reject
& Report]:::green\n HASH --> |clear| NSFW[NSFW
Classifier]:::orange\n NSFW --> |score > 0.9| REJ2[Reject]:::green\n NSFW --> |score 0.5-0.9| SOFT[Soft Tag
for Safe Search]:::teal\n NSFW --> |score < 0.5| CLEAN[Clean
Tag]:::teal\n SOFT --> IDX[Index with
Safety Metadata]:::teal\n CLEAN --> IDX\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Ingestion filters**\n\nEvery image is hashed with PhotoDNA or similar perceptual-hash technology and checked against the NCMEC hash database of known CSAM. Matches are rejected, stored, and reported per legal obligation. \n\nThis check is non-negotiable and runs before anything else. Then a multi-label NSFW classifier scores the image on several axes: explicit (nudity, sexual content), violent (gore, weapons in threatening contexts), drugs, and offensive content. High-confidence rejections (score > 0.9) never enter the index. Medium scores (0.5-0.9) get soft tags that determine visibility under different SafeSearch levels.\n\n\n| Category | Detection Approach | Action |\n|----------|--------------------|--------|\n| CSAM | NCMEC hash match + in-house classifier | Reject, report, legal hold |\n| Explicit adult | Multi-label CNN classifier | Reject at > 0.9, soft tag 0.5-0.9 |\n| Violence and gore | Classifier | Reject at > 0.9, soft tag 0.5-0.9 |\n| Weapons in threatening contexts | Object detector + context classifier | Soft tag |\n| Hate symbols | Symbol detection + context | Reject with human review escalation |\n| Self-harm imagery | Classifier | Reject, surface support resources |\n\n\n#### **Query-time filters**\n\nSafeSearch has three modes. \"Strict\" excludes everything above a low NSFW threshold (0.2), surfacing a clean grid suitable for minors and workplaces. \"Moderate\" is the default for most regions and excludes content above 0.5. \n\n\"Off\" shows everything including soft-tagged content, for users who explicitly opt in. The filter runs as part of the final re-ranking stage because candidate-level scores are cheap (a feature lookup, not a model inference at query time).\n\n#### **The false-positive problem**\n\nAggressive NSFW classifiers misfire on art, medical imagery, classical sculpture, and breastfeeding photos. A painted nude in a museum catalog is not pornography, but a generic classifier labels it as such. Production systems handle this with three strategies.\n\nFirst, curated allow-lists: domains like Wikipedia, museum sites, and medical resources get lower thresholds or bypass certain filters entirely.\n\nSecond, classifier ensembles: an \"art\" classifier, a \"medical\" classifier, and an \"NSFW\" classifier vote together, and content flagged by the NSFW model but strongly flagged by art or medical gets a downgraded tag rather than outright rejection.\n\nThird, human review workflows for borderline cases, especially high-traffic uploads where a false positive has user-visible impact.\n\nThe trade-off is transparency. Aggressive filtering suppresses legitimate content; quiet demotion is hard to audit. Keep a full decision log (what was filtered, which classifier fired, what the threshold was, whether a human reviewed) so that audits and appeals have data to work with. Sample a small fraction of filtered content for human review to keep the classifier calibrated over time.\n\n### Visual Similarity vs Semantic Intent\n\nA user uploads a photo of a vintage 1960s-style orange velvet armchair. What they want is more vintage 1960s-style orange velvet armchairs. What a plain CLIP-similarity search returns is a mix: some similar armchairs, some orange fabrics, some 1960s-era objects in general, and some chairs of any era because the embedding encodes all of these aspects at once. This mismatch between pure visual similarity and user intent is the core quality problem for reverse image search.\n\nThree approaches improve intent alignment:\n\n#### **Attribute prediction and re-ranking**\n\nTrain small classifiers that predict image attributes (category, color, style, era, material) and include these predictions as features in the ranker. At query time, predict attributes for the uploaded image, and rank candidates higher when their attributes match. A candidate armchair with matching color and era scores above a lamp of the same color, even if the raw embedding similarity is close.\n\n\n```python\n# Attribute-aware reranking for reverse image search\ndef rerank_with_attributes(query_img, candidates, base_scores):\n q_attrs = attribute_model.predict(query_img)\n reranked = []\n for cand, base in zip(candidates, base_scores):\n c_attrs = cand.attributes # precomputed at ingestion\n category_bonus = 0.15 if c_attrs.category == q_attrs.category else 0\n color_bonus = 0.10 * color_similarity(c_attrs.color, q_attrs.color)\n style_bonus = 0.08 if c_attrs.style == q_attrs.style else 0\n reranked.append(base + category_bonus + color_bonus + style_bonus)\n return reranked\n```\n\n\n#### **Object detection and category pinning**\n\nAn object detector identifies what the query image contains (a single armchair, for example) and restricts retrieval to images that contain the same object category. This is a hard filter rather than a soft boost, and it works well when the intent is clear from a dominant object. It fails on scene-level queries like \"beach at sunset\" where no single object defines intent.\n\n#### **User clarification**\n\nThe UI asks. After an initial retrieval, the system offers refinement chips: \"More like this color\", \"More of this style\", \"Same era\", \"Same shape\". Each chip re-runs retrieval with a different weighting of attributes. This is the simplest approach and often the best, because the user knows their intent better than any model.\n\nThe design here uses a blend: object detection filters candidates to matching categories when detection is confident, attribute-aware reranking handles the soft weighting, and clarification chips surface in the UI when ambiguity is high (detected by looking at the entropy of top candidate attributes). Amazon's visual search and Pinterest Lens both combine these strategies, with Pinterest especially leaning on object-level embeddings as a first-class retrieval path.\n\nOne failure mode worth naming: collapsing all attributes into a single score can silently hurt relevance for users whose intent sits off-axis from the average. If most users who upload armchairs want category matches, the ranker learns to weight category heavily, and the minority of users who wanted \"that particular shade of orange across any furniture\" get worse results. Maintaining separate re-ranker variants per detected category, and measuring per-segment quality, catches this drift before it hardens into a quality regression.\n\n### Freshness for Trending Content\n\nThe corpus changes constantly. News events produce bursts of images within hours (protests, sports events, breaking news), memes spread in minutes, and trending visual queries shift daily. A weekly rebuild of a 10B-image index won't capture fresh content, so the system uses a two-tier index similar to the pattern in text search, but with image-specific twists.\n\n\n```mermaid\nflowchart LR\n NEW[New/Updated
Images]:::primary --> LIVE[(Live Tier
~150M images
last 48h)]:::orange\n HIST[Historical
Corpus]:::primary --> BASE[(Base Tier
~10B images
weekly rebuild)]:::teal\n Q[Query]:::primary --> R[Retrieval
Layer]:::teal\n R --> LIVE\n R --> BASE\n LIVE --> M[Merge &
Dedup]:::teal\n BASE --> M\n M --> RANK[Ranking]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Base tier**\n\nHolds all 10B images in the PQ-compressed HNSW index. Rebuilt end-to-end weekly. Optimized for scale and steady-state quality.\n\n#### **Live tier**\n\nCovers everything ingested in the last 48 hours, roughly 100-200M images. Uses full-precision HNSW because the size fits in RAM uncompressed and the recall bump matters for fresh content. Rebuilt every 1-2 hours, with streaming writes continuously adding new images that get searchable within 10-15 minutes of ingestion.\n\nAt query time both tiers are searched in parallel, results merge and dedupe by URL and perceptual hash, and the union passes to the cross-encoder. When the weekly base rebuild completes, images that were in the live tier migrate to the base tier, and the live tier resets to its new 48-hour window.\n\nThe cost is real: fan-out doubles (two retrieval paths instead of one), embedding versioning gets trickier because live and base tiers must use the same encoder, and the pipeline has more moving parts to monitor.\n\nThe benefit is that breaking news imagery reaches users within minutes instead of days, which is the difference between a useful image search for trending queries and a stale one. Google and Bing both operate variants of this two-tier pattern.\n\n---\n\n# 7. Evaluation and Iteration\n\n### Offline Evaluation\n\nBuild a test set from human relevance judgments covering both query modalities and a mix of query types. Head text queries (\"cat\", \"sunset\") are frequent and well-served; torso queries (\"art deco lamp\") have moderate frequency; tail queries (\"1970s brown rotary phone on wooden desk\") are rare and often poorly served.\n\nFor image queries, stratify by detected object category (products, people, scenery, logos, memes) because performance varies dramatically across these segments.\n\n#### **Constructing the evaluation set**\n\nUniform sampling oversamples the head (the top 1,000 queries are a huge fraction of traffic) and under-represents the tail, which is where the model actually breaks.\n\nUse stratified sampling: draw separate samples from head, torso, and tail bands by frequency quantile, plus targeted samples by intent class (shopping, people, scenery, named entities) and image-query object category. Aim for 3,000-8,000 labeled queries with 20-50 judged images each, which is enough power to detect NDCG differences of 0.01-0.02.\n\n#### **Labeling**\n\nAnnotators rate (query, image) pairs on a 4-point scale (perfect, good, acceptable, bad) and separately flag safety violations, quality issues, and duplicates. Every pair gets at least three annotators, and disagreements route to a senior reviewer.\n\nTrack inter-annotator agreement (Cohen's kappa) as a quality signal. Kappa below 0.5 means the guidelines are ambiguous and the labels are noisier than the differences between model versions. Refresh the evaluation set quarterly to catch distribution shift, and version it so past experiments remain reproducible.\n\nCompute NDCG@10 on the full test set and on slices. Slice analysis catches regressions that hide in the average.\n\n\n```python\nfrom sklearn.metrics import ndcg_score\n\nndcg_results = []\nfor query_id, group in test_set.groupby(\"query_id\"):\n true_rel = group[\"human_label\"].values\n pred_scores = group[\"model_score\"].values\n ndcg = ndcg_score([true_rel], [pred_scores], k=10)\n ndcg_results.append({\n \"query_id\": query_id,\n \"ndcg\": ndcg,\n \"modality\": group[\"modality\"].iloc[0],\n \"segment\": group[\"segment\"].iloc[0],\n })\n\nresults_df = pd.DataFrame(ndcg_results)\nprint(results_df.groupby([\"modality\", \"segment\"])[\"ndcg\"].mean())\n```\n\n\n**Output:**\n\n\n```plaintext\nmodality segment\ntext head 0.54\ntext torso 0.43\ntext tail 0.36\ntext shopping 0.48\ntext people 0.51\nimage products 0.62\nimage scenery 0.39\nimage people 0.44\nimage memes 0.71\n```\n\n\nReverse-image NDCG is usually higher on products and memes (concrete visual match) and lower on scenery and people (where intent is harder to pin down). If the numbers diverge much more than this, something's miscalibrated.\n\n### Online Evaluation\n\nA/B testing works, but two patterns specific to image search are worth calling out.\n\n#### **Novelty effects**\n\nUsers click more on new image grid layouts or orderings simply because they're different, not because they're better. This is especially strong in visual interfaces where the grid composition is part of the experience. Run experiments for at least 14 days to let novelty wear off. For ranking-only experiments (no UI changes), a week usually suffices.\n\n#### **Interleaving**\n\nInstead of showing different users different result grids, interleave results from control and treatment into a single grid and measure click attribution.\n\nInterleaving detects ranking quality differences with 10-100x fewer queries than traditional A/B tests, which matters when safety guardrails make it expensive to run large experiments. Bing publishes detailed work on interleaving for image search.\n\n#### **Guardrails**\n\nMonitor secondary metrics during every experiment: p99 latency, zero-result rate, safety incident rate, and crash rate. A relevance improvement that bumps p99 by 30ms or triples the zero-result rate is not shippable, regardless of NDCG. The safety incident rate is a non-negotiable guardrail: any experiment that increases it rolls back immediately.\n\n#### **The offline-online gap**\n\nOffline NDCG improves, online CTR and session success refuse to move, or worse, move the wrong way. This happens for a few predictable reasons in image search. Annotators rate images on topical relevance, but users also weigh visual aesthetics, resolution, whitespace, and grid composition that judges don't see. \n\nThe labeling distribution doesn't match real traffic, so an improvement concentrated on the tail looks strong offline and invisible online. And some image-search metrics (dwell on source page, reformulation) move slowly and need large sample sizes to detect. \n\nThe practical response is to track both offline and online metrics side by side, treat offline NDCG as a guardrail rather than the sole source of truth, and when the two disagree, trust the online signal and investigate what the offline set missed.\n\n### Monitoring\n\nProduction image search systems degrade silently. Queries keep coming, but relevance erodes gradually as the world changes, new visual trends emerge, and the model stays fixed.\n\n#### **What to monitor:**\n\n- **Retrieval recall.** Periodically sample queries, run them through the full pipeline against the complete corpus (expensive), and check whether the retrieval stage captures the top-ranked images. Dropping recall means the ANN index or the vision encoder is going stale.\n- **Embedding drift.** Track the average cosine similarity between query embeddings and clicked image embeddings over time. A falling trend means the embedding space is losing alignment with user intent.\n- **Modality distribution shift.** Monitor the split between text and image queries, and within image queries the distribution of detected object categories. A spike in a new category (a viral product launch, a news event) signals the model may not handle it well.\n- **Image-quality-score distribution.** If the aesthetic score distribution of top-ranked results drifts, the ranker's weighting on quality may be miscalibrated.\n- **Click entropy.** If users increasingly click on results at grid position 6+ instead of 1-2, the top-of-grid ranking is degrading.\n- **Latency p99 by component.** Track each stage separately. A latency spike in the cross-encoder usually means the candidate set grew unexpectedly, often because retrieval is returning more over-clustered results that escape dedup.\n- **Safety-classifier rates.** Track the rate at which safety classifiers fire, both at ingestion and at serving. A sudden jump suggests a policy change, a classifier drift, or a coordinated attempt to push harmful content.\n\nSet automated alerts on these metrics and retrain models when offline evaluation shows degradation. The dual-encoder retrains every 2-4 weeks on fresh click data. The cross-encoder retrains less often (every 4-8 weeks) because it sees the full query-image interaction and is less sensitive to corpus drift.\n\n#### **Embedding versioning and migration**\n\nRetraining the vision encoder produces a new embedding space that is mathematically incompatible with the old one. New query vectors cannot be compared against old image vectors and still produce meaningful distances.\n\nThis means every dual-encoder retrain invalidates the entire 20 TB image index, and re-encoding 10 billion images on a GPU cluster takes 12-24 hours even with aggressive batching. The migration pattern is a shadow rebuild, done carefully:\n\n1. Freeze the new vision-encoder checkpoint and record its version hash.\n2. Re-encode the full corpus into a new ANN index running in parallel with the old one. The new index is \"dark\" during this period: built, validated, but not serving traffic.\n3. While the rebuild runs (12-24 hours on a dedicated GPU fleet separate from serving), the production system continues to use the old index with the old query encoder.\n4. Validate the new index against an offline evaluation set and a canary traffic slice (1-5% of queries). Compare NDCG and online metrics on the canary before committing.\n5. Cut over atomically: both query encoding and retrieval switch to the new version in one deploy. Hold the old index online for 24-48 hours as a rollback target, then retire it.\n\nThe live tier undergoes the same cutover, but because it rebuilds every 1-2 hours, its migration cost is negligible. The expensive part is the base-tier re-encoding, which runs on a GPU pool that is deliberately separate from the serving fleet so training load doesn't degrade serving latency. A common and costly mistake is to mix old and new embeddings in the index. Serving some newer images with the new encoder and older images with the old one produces silently inconsistent score distributions, and recall degrades in ways that don't show up in aggregate metrics. Keep each index pure to a single embedding version.","pageType":"ml-system-design"}

Design an Image Search System

Ashish Pratap Singh

Get Premium