Before committing to an architecture, lets clarify the scope. Image search covers a wide range of products, and each has different scale and quality requirements. A conversation with the interviewer helps pin down the design.

Clarifying Questions

With scope pinned down, the requirements follow.

Requirements

Functional Requirements:

Non-Functional Requirements:

ML Problem Framing

This is a multi-stage retrieval and ranking problem over a shared multi-modal embedding space. The key insight is that a well-trained model, such as CLIP or SigLIP, encodes text and images into the same vector space, so a text query and an image query both become 512-dimensional vectors that compare against the same index of 10 billion image vectors. One index, two query entry points.

Serving happens in stages because no single model can score 10 billion images per query under 300ms:

At the retrieval stage the objective is recall: the top 2,000 candidates should contain almost every image a user might find relevant. Missing a great image here can't be fixed downstream. At the ranking stage the objective flips to precision and ordering: given the 2,000 candidates, get the best images to the top. The two objectives need different model shapes, which is why the pipeline uses a lightweight dual-encoder for retrieval and a heavier cross-encoder for ranking.

Metrics

Offline Metrics:

NDCG@10 is the primary offline metric for text-query relevance. Reverse image search needs its own metric because the task is different: given a query image, find the most visually similar images. mAP on a near-duplicate benchmark (e.g., Copydays, INSTRE) measures this directly. Recall@2000 separates retrieval quality from ranking quality. If retrieval only covers 70% of the truly relevant images, no amount of ranker improvement will fix the top 10.

Online Metrics:

Click-through on images is noisier than on web links because users scan the grid and click quickly. Dwell time on the preview is a better satisfaction signal. Safety incident rate is a hard guardrail, not a quality metric, and it must never regress during an experiment regardless of how much relevance improves.

The system has two planes: an offline pipeline that ingests images, generates embeddings, and trains models, and an online pipeline that handles queries in real time. The planes communicate through shared indexes and feature stores.

Tracing a query through the online path: a text or image query arrives and hits a query cache. On a miss, query understanding runs spell correction and intent classification for text, or EXIF inspection and resizing for images.

The appropriate encoder (text or vision) produces a 512-dimensional vector, and ANN search retrieves the top 2,000 nearest vectors from both the base and live tiers in parallel. A near-duplicate collapse step groups visually identical results so the grid shows variety rather than the same image under ten different URLs.

The ranker scores the remaining candidates using dense features, and a final safety and diversity filter produces the top 50. Image queries additionally attach a perceptual hash match step upfront to handle the common case where the uploaded image already lives in the corpus.

The offline path ingests new and updated images, runs them through preprocessing (resize, normalize, thumbnail), strips unsafe content at the filter stage, then generates both embedding vectors and perceptual hashes for the two indexes. Click and view logs feed the training loop, which refreshes the vision encoder periodically and the cross-encoder ranker on a separate schedule.

Scale Numbers

A back-of-the-envelope pass grounds every decision below.

Query load

At 50,000 peak QPS with a Gaussian-ish daily curve, average QPS is roughly 15,000-20,000. Daily volume lands at 1.5-2 billion queries. Roughly 10-15% of those are image uploads (reverse image search), which are more expensive to serve because they require a full vision-encoder forward pass at query time.

Embedding storage

Ten billion images, each a 512-dim float32 vector, is 10,000,000,000 × 512 × 4 bytes = 20.5 TB for raw embeddings. Product quantization at 64 bytes per vector compresses this to roughly 640 GB. The HNSW graph and PQ codebooks add another 20-30%, so the serving index lands near 0.8-1.0 TB, which is feasible to shard in-memory across a few dozen nodes.

Raw image storage

Ten billion images at an average of 200 KB each is 2 PB. This lives in blob storage (object store backed by SSD for hot access, standard storage for cold), never on the serving hot path. Serving fetches thumbnails on demand.

Thumbnail tier

For each image the system maintains 2-3 pre-generated thumbnails (grid size ~300px, preview size ~800px, full size ~1200px). At roughly 20 KB per grid thumbnail, the hot thumbnail tier is ~200 TB and is served through a CDN with a high cache hit rate because the head of the query distribution surfaces the same images repeatedly.

Query-time GPU budget

A ViT-B/16 vision encoder has about 86M parameters. On a modern A100 GPU it processes a 224×224 image in 1-3ms with small-batch serving. At 50K QPS with 10% image queries (5K image-query QPS) and 2ms per image, one GPU handles ~500 QPS, so the image-query path needs 10-15 GPUs plus redundancy.

The text encoder (a distilled BERT-like tower) is cheaper, 0.5-1ms per query, needing 10-20 GPUs at peak. The cross-encoder ranker is the biggest consumer: scoring 2,000 candidates per query at 20-30ms per query with batching puts it at 40-80 GPUs. Total online GPU fleet lands at 70-120.

Training data volume

Click and view logs at 2B queries per day with an average of 4-5 image impressions viewed per query yield roughly 10 billion (query, impression, action) records daily. After filtering for high-confidence clicks with dwell over 5 seconds, 80-200M strong positive pairs per day are available for training.

These numbers set the hard edges of the design: the base vector index must be quantized, raw pixels stay off the hot path, image queries are first-class cost centers, and the ranker dominates GPU spend.

Data Flow

Ingestion runs continuously. Fresh images reach the index within 10-15 minutes via an incremental pipeline that writes to a live tier, and a weekly base rebuild re-encodes the full corpus to pick up model updates.

The training loop runs on its own cadence: the dual-encoder refreshes every 2-4 weeks, and the cross-encoder refreshes every 4-8 weeks because it sees query-image interactions and is less sensitive to corpus drift.

Data Sources

Five sources feed the system:

Image Representation

Raw images are not ready to be consumed by a neural network. Decoding, resizing, normalization, and format handling all sit between the byte stream and the encoder. Mistakes here silently degrade retrieval quality because the online query image and the offline indexed image must go through identical transforms.

A few details matter more than they look. EXIF orientation is not applied by every decoder, so a photo that looks upright in a browser can arrive at the model rotated ninety degrees. Apply orientation before resizing. For transparent PNGs, flatten onto a neutral gray background because flattening onto white biases the encoder toward bright scenes.

For animated GIFs, index only the middle frame (motion rarely changes the semantic content, and indexing every frame multiplies storage). For SVGs, rasterize at the target resolution. Keep everything in sRGB color space. Mixed color spaces (Display P3 photos from iPhones, Adobe RGB scans) produce subtly different embeddings if not normalized.

Resolution trades quality for cost. At 224×224, a ViT-B/16 runs in 1-3ms per image. At 384×384 the same model needs 4-8x more compute but captures finer detail (fabric texture, small text, subtle color differences). Production systems often train and serve at 224×224 for general search, and reserve 384×384 for a high-fidelity tier used by shopping or professional search where the extra detail is worth the cost.

Multi-Field Signals

Image search is rarely pure image search. A web image exists in a context: alt-text, caption, page title, anchor text, and body text near the image all carry signal. Ignoring them leaves obvious wins on the table.

Production systems fuse these signals in two places. At retrieval, the text query matches against a text vector that represents the concatenated surrounding text fields, which gives a second retrieval path alongside the image-embedding path. At ranking, per-field match features (alt-text query overlap, caption match, title match) feed the ranker so it can reason about which field matched and how strongly.

Google Images and Bing Image Search both lean heavily on surrounding text for text queries. Purely visual retrieval (query text → image embedding directly via CLIP) works well for conceptual queries but loses to text-based retrieval for named-entity queries like "Barcelona FC logo" where the surrounding text pins the image to an entity the visual model might miss.

Feature Engineering

The ranker needs features from three buckets. Pure embedding similarity is not enough to sort the final candidate set, because it doesn't know about resolution, source reputation, or user behavior.

Query features describe the query itself:

Image features describe static properties of each image:

Cross features describe the query-image interaction:

Feature Table

Features like image_colors_top5 and image_objects are computed once at ingestion and cached. The query-dependent features (clip_similarity, bm25, ocr_match) are computed per request and dominate feature-fetch latency.

Baseline Approach

A respectable baseline uses a pretrained vision backbone (ResNet50 trained on ImageNet) to produce an image embedding, and a text encoder (TF-IDF or a small sentence encoder) to produce a text embedding over the surrounding text.

For a text query, retrieve the top images whose surrounding text matches via BM25, then re-rank by image-aesthetic score and source authority. For an image query, retrieve by cosine similarity against the pretrained ResNet50 vectors.

This baseline works for narrow cases. For an image query that's a near-duplicate of something in the corpus, ResNet50 similarity surfaces it. For a text query where the desired image happens to have strong alt-text, BM25 surfaces it.

Where the baseline collapses is on conceptual text queries ("nostalgic summer evening", "minimalist kitchen") and on image queries that need semantic rather than pixel similarity ("find more images of the same landmark in different weather"). The ResNet50 embedding encodes textures and objects, not concepts, and there's no shared space between text and images, so the system can't route text queries through the image embedding at all.

Advanced Approach

The production design uses a dual-encoder trained contrastively on massive (image, text) pairs, in the style of CLIP or SigLIP. One transformer encodes images, another encodes text, and the two are trained so that a matching (image, caption) pair sits close in the shared embedding space while mismatched pairs sit far apart.

After training, a text query and an image query both produce a 512-dimensional vector in the same space, so a single ANN index over image vectors serves both modalities.

The two towers don't share weights. Text and images have entirely different structure (tokens vs pixels, different distributional statistics), so each encoder specializes on its modality.

At query time, only the relevant encoder runs: the text tower for text queries, the vision tower for image queries. Image vectors are precomputed offline and live in the ANN index, so the online cost is dominated by encoding the query and running the nearest-neighbor search.

Vision encoder choice

The vision encoder is the biggest single lever for cost, quality, and latency. Choices range from small CNNs to large transformers, and the pick depends on the scale and budget.

The design here uses ViT-B/16 as the primary vision encoder because it balances quality and cost. ViT-L/14 sits behind a feature flag for a high-fidelity tier used by shopping search. SigLIP-SO400M is the frontier choice when quality matters more than GPU bill.

ANN index design

Retrieval only works if the index returns top-K nearest neighbors across 10B vectors in well under 20ms. Brute-force search is out: one query against 10B 512-dim vectors is 5 trillion multiplications per query, which no GPU fleet will absorb at 50K QPS. The index uses HNSW over product-quantized vectors, the same recipe as most large-scale retrieval systems.

Each 512-dim float32 vector (2 KB raw) is split into 64 sub-vectors of 8 floats, and each sub-vector is replaced by an 8-bit codebook index trained via k-means. That's 64 bytes per image, a 32x compression.

Distance lookups use precomputed tables so quantized search is actually faster than raw search, not slower. The recall trade-off (90-95% vs 97-99% for full HNSW) is acceptable because the cross-encoder downstream can rescue candidate-order mistakes.

One quirk of image embeddings: they cluster tighter than text embeddings because natural image distributions have strong modes (faces, logos, landscapes, product photos).

HNSW tuning parameters (M, efConstruction, efSearch) typically need to be pushed higher than for text retrieval to avoid getting trapped in dense local regions. A common starting point is M=48, efConstruction=200, efSearch=128.

Cross-encoder ranking

The second stage scores the top 2,000 candidates with a model that sees the query and the candidate jointly. For text queries, a vision-language cross-encoder like a fine-tuned BLIP-2 reranker takes the query tokens and the image patch tokens through cross-attention, producing a relevance score from a CLS token.

For image queries, the "query" side is another set of vision tokens from the query image, and the cross-encoder learns to score candidate-to-query pair similarity that captures visual correspondence beyond a single global dot product.

The cross-encoder's relevance score alone isn't the final output. It captures query-image fit from pixels and text but can't see the features from §3 (resolution, aesthetic score, source authority, historical CTR, behavioral co-click).

Those signals carry information pixels can't reveal, so the cross-encoder score is concatenated with the dense features and fed to a gradient-boosted tree model (LightGBM or XGBoost) trained on human relevance labels and click-derived pseudo-labels.

This two-stage setup keeps the expensive cross-encoder focused on what it does best (visual-textual fit) and leaves feature interactions to a cheap tree model that's easy to retrain.

The three architecture patterns (dual-encoder, late-interaction, cross-encoder) sit on a quality-cost spectrum:

The design here pairs dual-encoder retrieval with cross-encoder ranking. Late-interaction models are worth considering when the ranking stage is cost-constrained and a larger index footprint is acceptable. Pinterest's visual search has used late-interaction patterns to push retrieval quality up without paying for a full cross-encoder per query.

Training

Dual-encoder training

The loss is a symmetric InfoNCE: for a batch of N (image, text) pairs, the model learns to make each image vector closest to its own text vector and far from the other N-1 text vectors, and vice versa.

Large batch sizes are critical because the quality of contrastive learning scales with the number of in-batch negatives. CLIP used a batch size of 32,768. In-house systems at major companies often train with batch sizes of 16K-64K on hundreds of GPUs for weeks.

Training happens in two phases. Phase one is large-scale pretraining on web-scraped (image, alt-text) pairs, typically billions of pairs, which teaches the model broad visual-linguistic correspondence. Phase two is fine-tuning on domain-specific data: query-click pairs from the search product, human-labeled relevance data, and curated high-quality captions.

Hard-negative mining matters here. In-batch negatives are mostly easy (a random image in a batch rarely matches a given query's text), so after initial training, the system mines hard negatives from its own top-K retrieval results (images that rank high but weren't clicked) and adds them to training batches. Three to five rounds of hard-negative mining is typical.

Cross-encoder training

The cross-encoder trains on a mix of human judgments (300K labeled pairs) and click-derived signals. For each query, candidates with different relevance grades or click outcomes form training tuples, and the model learns to rank them correctly with a pairwise or listwise loss.

LambdaMART-style losses work well because they directly optimize NDCG. The cross-encoder needs fewer examples than the dual-encoder (hundreds of thousands vs billions) because it sees the full interaction between query and image and learns more from each example.

Inference Pipeline

A single query flows through the system along one of two paths depending on its modality. The paths converge after the query encoding step and share all downstream stages.

Text queries take the cheaper path: the text encoder is a distilled transformer that runs in 5ms or less. Image queries are more expensive because the full vision encoder runs per request (20ms for ViT-B/16 including decode and preprocessing).

An image query also runs a perceptual hash lookup in parallel: if the uploaded image matches something already in the corpus (very common, since users often upload an image they found somewhere on the web and want to know its source), the hash match returns the exact URL within 5ms and can short-circuit ANN retrieval for that subtree. Both paths converge at the ANN step, and downstream processing is modality-agnostic.

A query cache sits in front of the pipeline. For text queries, the cache key is the normalized query text plus SafeSearch level and coarse user context (location, language). The head of the query distribution is heavy-tailed: the top few thousand text queries account for 15-25% of traffic and their top results change slowly, so caching responses for 5-30 minutes cuts backend load substantially and improves p50 for common queries.

Image queries are harder to cache because the uploaded image is typically unique, but the perceptual hash of the uploaded image can serve as a cache key with high hit rates for viral or meme content.

Batch vs Real-Time

Different components run in different modes based on cost and freshness requirements.

The biggest operational headache is keeping the embedding index fresh when the vision encoder itself changes. A model refresh invalidates every precomputed vector and forces a full re-encode of 10B images, which takes 12-24 hours on a dedicated GPU cluster.

This is much more expensive than a typical text-search embedding refresh and drives the migration pattern discussed in Monitoring.

Latency Budget

The budget intentionally targets ~150ms median rather than the full 300ms requirement. The headroom absorbs tail variance: GC pauses, shard stragglers, cold feature-store reads, and transient network hiccups each commonly add 20-50ms at the 99th percentile. Budgeting directly to the SLA would mean routine tail violations.

The cross-encoder is the dominant cost. If it becomes a bottleneck, the first lever is reducing the candidate set from 2,000 to 1,000, roughly halving ranking time. The second is distilling the cross-encoder into a smaller model. The third, and most architectural, is inserting a cheap middle-stage re-ranker (late-interaction or a small MLP on dense features) to trim 2,000 down to 500 before the expensive cross-encoder runs.

Sharding and Distribution

At 10 billion images and a 1 TB compressed index, the embedding index does not fit on a single node. The perceptual hash index is smaller (a few hundred bytes per image, so 1-2 TB for the full corpus) but also shards for query parallelism. Every query fans out to all shards, each shard searches its slice, and a root aggregator merges top-K candidates globally.

Sharding strategy

Shard by image ID, not by embedding region. Region-based sharding (routing a query to the shard whose centroids are closest) is tempting because it reads fewer shards per query, but it creates load-balancing nightmares as the query distribution shifts.

Image-ID sharding is the industry default: every query hits every shard, but each shard handles a uniform slice, load is naturally balanced, and adding capacity means adding shards rather than rebalancing centroids. At 25 shards and 400M images per shard, each shard holds 25-40 GB of compressed index plus HNSW graph, which fits comfortably in the RAM of a mid-sized machine.

Fan-out and tail latency

With 25 shards and a 15ms per-shard budget, the retrieval step completes in max(shard_latency) + merge_time, not sum, provided there's enough network bandwidth. This assumes every shard responds quickly.

A single slow shard drags the whole query. Mitigation uses two techniques. First, hedged requests: after the p95 of typical shard latency (say 12ms), the router sends a backup request to a replica and takes whichever response arrives first.

Hedging costs a small extra load (a few percent of requests are duplicated) and eliminates most of the long tail. Second, partial-result tolerance: each shard can return partial results if it runs out of time, and the aggregator accepts a few missing shards rather than blocking. Losing 1 of 25 shards drops recall by roughly 4% in the worst case, which is acceptable when the alternative is a 300ms timeout.

Replication and capacity

Each shard runs on at least three replicas across availability zones. The router load-balances reads across replicas and fails over on timeout. For writes (new image ingestion), updates apply to the leader and asynchronously replicate, with staleness bounded to under a few seconds. At 50K peak QPS and 25 shards with 2.5-3x replication, the retrieval tier needs 75-100 nodes.

Add 70-120 GPU nodes for query encoding and cross-encoder ranking, plus a feature-store tier and a CDN layer, and the online serving footprint lands at roughly 250-350 nodes. Cost is dominated by GPUs, as it usually is for deep-learning-heavy retrieval systems.

Near-Duplicate and Exact-Duplicate Detection

The same image appears on the web in dozens of variants: cropped thumbnails, watermarked versions, JPEG reencodes at different quality levels, resized copies, color-adjusted edits, and frames with different aspect ratios.

A naive retrieval ranks all of them as near-top matches because their embeddings sit right next to each other, which produces a result grid that looks embarrassing: the same photograph eight times with slight differences. Handling duplicates is a retrieval problem, a ranking problem, and a storage problem at once.

Three techniques handle different parts of the problem:

Production systems combine all three. At ingestion, every image gets a SHA-256 hash (for exact match dedup in storage) and a perceptual hash stored in a dedicated Hamming-distance index (typically an LSH-based structure).

The CLIP embedding also gets stored. At query time, the near-duplicate collapse step clusters retrieved candidates: first by perceptual hash (Hamming distance under 6 on a 64-bit pHash groups near-identical images), then by embedding similarity (cosine similarity above 0.97 groups semantic near-duplicates).

A cluster is collapsed in the result grid: only the best representative is shown, and the others are accessible via a "see more sizes" link. "Best" is a ranking decision driven by resolution, source authority, and license clarity. Wikimedia Commons beats a random blog with the same photo. A 1024×768 version beats a 320×240 thumbnail.

The trade-off is the dedup threshold. Too aggressive (cluster at similarity 0.85), and legitimately different images get grouped together; the grid loses variety. Too loose (cluster at 0.99), and the same photo shows up five times.

Different products tune this differently: Google Images is aggressive because the grid is a preview into the web and users want variety, while reverse image search tools like TinEye are loose because users want to find every copy of the image.

One subtle detail often missed: near-duplicate detection at ranking time is necessary even with dedup at ingestion, because a query can surface 2,000 candidates that happen to include multiple clusters. The per-query collapse runs against the retrieved candidate set, not the whole corpus, so it's cheap (O(k log k) for k candidates) but essential.

Handling the Two Query Modalities

Text-to-image and image-to-image retrieval look symmetric (both produce a query vector, both query the same ANN index), but their failure modes are very different.

Text queries are ambiguous. "Jaguar" might mean the car or the animal. "Apple" is the fruit or the company. "Pitch" is sports, sales, or sound. The text encoder picks one interpretation and the retrieval reflects it, so users whose intent differs see irrelevant results.

The mitigation is upstream: intent classification tags the query, query expansion generates paraphrases for each plausible intent, and the retrieval fans out to both interpretations with separate ANN queries. The top-K from each intent merge via rank fusion, and the ranker learns which intent matches the click pattern. Pinterest publishes their use of a similar pattern for ambiguous visual search queries.

Image queries are visually concrete but semantically underspecified. A user uploads a photo of a red Nike running shoe on a wooden floor with morning light. What do they want? Similar red shoes? Similar Nike shoes? Similar flooring? Similar lighting? The raw image embedding encodes all of these at once, and a naive retrieval returns a mix that often misses the user's actual intent. Four strategies help:

The design here supports region proposals as the default UI for image queries, falls back to whole-image embedding when no object is detected clearly, and keeps multi-vector indexing as a feature-flagged capability for shopping search. The cost is a modest object detector at ingestion (~5ms per image, adding to the embedding cost) and at query time (~8ms per image query).

Safety and NSFW Filtering

Content safety is not a "nice to have" for an image search system. It's a hard requirement with legal and reputational consequences. The system enforces safety at two points: at ingestion, before content reaches the index, and at query time, as a re-ranking filter.

Ingestion filters

Every image is hashed with PhotoDNA or similar perceptual-hash technology and checked against the NCMEC hash database of known CSAM. Matches are rejected, stored, and reported per legal obligation.

This check is non-negotiable and runs before anything else. Then a multi-label NSFW classifier scores the image on several axes: explicit (nudity, sexual content), violent (gore, weapons in threatening contexts), drugs, and offensive content. High-confidence rejections (score > 0.9) never enter the index. Medium scores (0.5-0.9) get soft tags that determine visibility under different SafeSearch levels.

Query-time filters

SafeSearch has three modes. "Strict" excludes everything above a low NSFW threshold (0.2), surfacing a clean grid suitable for minors and workplaces. "Moderate" is the default for most regions and excludes content above 0.5.

"Off" shows everything including soft-tagged content, for users who explicitly opt in. The filter runs as part of the final re-ranking stage because candidate-level scores are cheap (a feature lookup, not a model inference at query time).

The false-positive problem

Aggressive NSFW classifiers misfire on art, medical imagery, classical sculpture, and breastfeeding photos. A painted nude in a museum catalog is not pornography, but a generic classifier labels it as such. Production systems handle this with three strategies.

First, curated allow-lists: domains like Wikipedia, museum sites, and medical resources get lower thresholds or bypass certain filters entirely.

Second, classifier ensembles: an "art" classifier, a "medical" classifier, and an "NSFW" classifier vote together, and content flagged by the NSFW model but strongly flagged by art or medical gets a downgraded tag rather than outright rejection.

Third, human review workflows for borderline cases, especially high-traffic uploads where a false positive has user-visible impact.

The trade-off is transparency. Aggressive filtering suppresses legitimate content; quiet demotion is hard to audit. Keep a full decision log (what was filtered, which classifier fired, what the threshold was, whether a human reviewed) so that audits and appeals have data to work with. Sample a small fraction of filtered content for human review to keep the classifier calibrated over time.

Visual Similarity vs Semantic Intent

A user uploads a photo of a vintage 1960s-style orange velvet armchair. What they want is more vintage 1960s-style orange velvet armchairs. What a plain CLIP-similarity search returns is a mix: some similar armchairs, some orange fabrics, some 1960s-era objects in general, and some chairs of any era because the embedding encodes all of these aspects at once. This mismatch between pure visual similarity and user intent is the core quality problem for reverse image search.

Three approaches improve intent alignment:

Attribute prediction and re-ranking

Train small classifiers that predict image attributes (category, color, style, era, material) and include these predictions as features in the ranker. At query time, predict attributes for the uploaded image, and rank candidates higher when their attributes match. A candidate armchair with matching color and era scores above a lamp of the same color, even if the raw embedding similarity is close.

Object detection and category pinning

An object detector identifies what the query image contains (a single armchair, for example) and restricts retrieval to images that contain the same object category. This is a hard filter rather than a soft boost, and it works well when the intent is clear from a dominant object. It fails on scene-level queries like "beach at sunset" where no single object defines intent.

User clarification

The UI asks. After an initial retrieval, the system offers refinement chips: "More like this color", "More of this style", "Same era", "Same shape". Each chip re-runs retrieval with a different weighting of attributes. This is the simplest approach and often the best, because the user knows their intent better than any model.

The design here uses a blend: object detection filters candidates to matching categories when detection is confident, attribute-aware reranking handles the soft weighting, and clarification chips surface in the UI when ambiguity is high (detected by looking at the entropy of top candidate attributes). Amazon's visual search and Pinterest Lens both combine these strategies, with Pinterest especially leaning on object-level embeddings as a first-class retrieval path.

One failure mode worth naming: collapsing all attributes into a single score can silently hurt relevance for users whose intent sits off-axis from the average. If most users who upload armchairs want category matches, the ranker learns to weight category heavily, and the minority of users who wanted "that particular shade of orange across any furniture" get worse results. Maintaining separate re-ranker variants per detected category, and measuring per-segment quality, catches this drift before it hardens into a quality regression.

Freshness for Trending Content

The corpus changes constantly. News events produce bursts of images within hours (protests, sports events, breaking news), memes spread in minutes, and trending visual queries shift daily. A weekly rebuild of a 10B-image index won't capture fresh content, so the system uses a two-tier index similar to the pattern in text search, but with image-specific twists.

Base tier

Holds all 10B images in the PQ-compressed HNSW index. Rebuilt end-to-end weekly. Optimized for scale and steady-state quality.

Live tier

Covers everything ingested in the last 48 hours, roughly 100-200M images. Uses full-precision HNSW because the size fits in RAM uncompressed and the recall bump matters for fresh content. Rebuilt every 1-2 hours, with streaming writes continuously adding new images that get searchable within 10-15 minutes of ingestion.

At query time both tiers are searched in parallel, results merge and dedupe by URL and perceptual hash, and the union passes to the cross-encoder. When the weekly base rebuild completes, images that were in the live tier migrate to the base tier, and the live tier resets to its new 48-hour window.

The cost is real: fan-out doubles (two retrieval paths instead of one), embedding versioning gets trickier because live and base tiers must use the same encoder, and the pipeline has more moving parts to monitor.

The benefit is that breaking news imagery reaches users within minutes instead of days, which is the difference between a useful image search for trending queries and a stale one. Google and Bing both operate variants of this two-tier pattern.

Offline Evaluation

Build a test set from human relevance judgments covering both query modalities and a mix of query types. Head text queries ("cat", "sunset") are frequent and well-served; torso queries ("art deco lamp") have moderate frequency; tail queries ("1970s brown rotary phone on wooden desk") are rare and often poorly served.

For image queries, stratify by detected object category (products, people, scenery, logos, memes) because performance varies dramatically across these segments.

Constructing the evaluation set

Uniform sampling oversamples the head (the top 1,000 queries are a huge fraction of traffic) and under-represents the tail, which is where the model actually breaks.

Use stratified sampling: draw separate samples from head, torso, and tail bands by frequency quantile, plus targeted samples by intent class (shopping, people, scenery, named entities) and image-query object category. Aim for 3,000-8,000 labeled queries with 20-50 judged images each, which is enough power to detect NDCG differences of 0.01-0.02.

Labeling

Annotators rate (query, image) pairs on a 4-point scale (perfect, good, acceptable, bad) and separately flag safety violations, quality issues, and duplicates. Every pair gets at least three annotators, and disagreements route to a senior reviewer.

Track inter-annotator agreement (Cohen's kappa) as a quality signal. Kappa below 0.5 means the guidelines are ambiguous and the labels are noisier than the differences between model versions. Refresh the evaluation set quarterly to catch distribution shift, and version it so past experiments remain reproducible.

Compute NDCG@10 on the full test set and on slices. Slice analysis catches regressions that hide in the average.

Output:

Reverse-image NDCG is usually higher on products and memes (concrete visual match) and lower on scenery and people (where intent is harder to pin down). If the numbers diverge much more than this, something's miscalibrated.

Online Evaluation

A/B testing works, but two patterns specific to image search are worth calling out.

Novelty effects

Users click more on new image grid layouts or orderings simply because they're different, not because they're better. This is especially strong in visual interfaces where the grid composition is part of the experience. Run experiments for at least 14 days to let novelty wear off. For ranking-only experiments (no UI changes), a week usually suffices.

Interleaving

Instead of showing different users different result grids, interleave results from control and treatment into a single grid and measure click attribution.

Interleaving detects ranking quality differences with 10-100x fewer queries than traditional A/B tests, which matters when safety guardrails make it expensive to run large experiments. Bing publishes detailed work on interleaving for image search.

Guardrails

Monitor secondary metrics during every experiment: p99 latency, zero-result rate, safety incident rate, and crash rate. A relevance improvement that bumps p99 by 30ms or triples the zero-result rate is not shippable, regardless of NDCG. The safety incident rate is a non-negotiable guardrail: any experiment that increases it rolls back immediately.

The offline-online gap

Offline NDCG improves, online CTR and session success refuse to move, or worse, move the wrong way. This happens for a few predictable reasons in image search. Annotators rate images on topical relevance, but users also weigh visual aesthetics, resolution, whitespace, and grid composition that judges don't see.

The labeling distribution doesn't match real traffic, so an improvement concentrated on the tail looks strong offline and invisible online. And some image-search metrics (dwell on source page, reformulation) move slowly and need large sample sizes to detect.

The practical response is to track both offline and online metrics side by side, treat offline NDCG as a guardrail rather than the sole source of truth, and when the two disagree, trust the online signal and investigate what the offline set missed.

Monitoring

Production image search systems degrade silently. Queries keep coming, but relevance erodes gradually as the world changes, new visual trends emerge, and the model stays fixed.

What to monitor:

Set automated alerts on these metrics and retrain models when offline evaluation shows degradation. The dual-encoder retrains every 2-4 weeks on fresh click data. The cross-encoder retrains less often (every 4-8 weeks) because it sees the full query-image interaction and is less sensitive to corpus drift.

Embedding versioning and migration

Retraining the vision encoder produces a new embedding space that is mathematically incompatible with the old one. New query vectors cannot be compared against old image vectors and still produce meaningful distances.

This means every dual-encoder retrain invalidates the entire 20 TB image index, and re-encoding 10 billion images on a GPU cluster takes 12-24 hours even with aggressive batching. The migration pattern is a shadow rebuild, done carefully:

The live tier undergoes the same cutover, but because it rebuilds every 1-2 hours, its migration cost is negligible. The expensive part is the base-tier re-encoding, which runs on a GPU pool that is deliberately separate from the serving fleet so training load doesn't degrade serving latency. A common and costly mistake is to mix old and new embeddings in the index. Serving some newer images with the new encoder and older images with the old one produces silently inconsistent score distributions, and recall degrades in ways that don't show up in aggregate metrics. Keep each index pure to a single embedding version.