{"title":"Exercise: Choosing Embedding Models","description":null,"content":"# Exercise 1: Rank Documents by Relevance\n\nYou are given a query and five candidate sentences. Embed all of them with `text-embedding-3-small`, score each sentence against the query using cosine similarity, and print the sentences from most to least relevant.\n\nThis is the ranking test you run to compare candidate models before committing to one. It shows the model's judgment directly: sentences that share meaning with the query should rank near the top even when they share few or no words with it.\n\n### Requirements\n\n- Embed the query and all five sentences. Batching them in one API call is fine.\n- Score each sentence against the query with cosine similarity.\n- Print the sentences in descending order of similarity, each with its score.\n- Do not hardcode the ordering. It must come from the computed scores.\n\nThe starter code sets up the client, the data, and a cosine helper. Fill in the embedding and ranking steps.\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses the platform-provided API key\n\nquery = \"How do I fix a leaking faucet?\"\n\ndocuments = [\n \"Plumbing repair guide for the home\",\n \"Stop a dripping tap step by step\",\n \"Best chocolate cake recipe\",\n \"How to train for a marathon\",\n \"Replacing a worn washer in a faucet\",\n]\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n mag_a = math.sqrt(sum(x * x for x in a))\n mag_b = math.sqrt(sum(y * y for y in b))\n return dot / (mag_a * mag_b)\n\n# TODO: embed the query and all documents with text-embedding-3-small\n\n# TODO: score each document against the query with cosine_similarity\n\n# TODO: sort documents by score, highest first\n\n# TODO: print each document and its score, most relevant first\n```\n\n\n### Expected Output\n\nThe exact scores depend on the model, but the order should be stable. The two faucet-related sentences and the plumbing guide should rank above the cake and marathon sentences.\n\n\n```plaintext\n0.61 Replacing a worn washer in a faucet\n0.55 Stop a dripping tap step by step\n0.48 Plumbing repair guide for the home\n0.13 How to train for a marathon\n0.09 Best chocolate cake recipe\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI()\n\nquery = \"How do I fix a leaking faucet?\"\n\ndocuments = [\n \"Plumbing repair guide for the home\",\n \"Stop a dripping tap step by step\",\n \"Best chocolate cake recipe\",\n \"How to train for a marathon\",\n \"Replacing a worn washer in a faucet\",\n]\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n mag_a = math.sqrt(sum(x * x for x in a))\n mag_b = math.sqrt(sum(y * y for y in b))\n return dot / (mag_a * mag_b)\n\n# Embed the query and all documents in a single batched call\nresponse = client.embeddings.create(\n model=\"text-embedding-3-small\",\n input=[query] + documents,\n)\nvectors = [item.embedding for item in response.data]\nquery_vector, doc_vectors = vectors[0], vectors[1:]\n\n# Score each document against the query\nscored = [\n (cosine_similarity(query_vector, doc_vectors[i]), documents[i])\n for i in range(len(documents))\n]\n\n# Sort by score, highest first, and print\nscored.sort(key=lambda pair: pair[0], reverse=True)\nfor score, text in scored:\n print(f\"{score:.2f} {text}\")\n```\n\n\nPutting the query first in the batch lets you split the returned vectors into the query vector and the document vectors with a single slice. The ranking comes entirely from the computed cosine scores, so the same code ranks any query against any document set without changes.\n\n---\n\n# Exercise 2: Compare Two Embedding Models\n\nChoosing a model means comparing candidates on the same data. Rank the documents with two different embedding models and look at whether the orderings agree. When they disagree, that is the signal that the model choice actually matters for your task.\n\n### Requirements\n\n- Write a `rank(model_name)` helper that embeds the query and documents and returns the documents sorted by similarity.\n- Run it with `text-embedding-3-small` and `text-embedding-3-large`.\n- Print both rankings side by side.\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nquery = \"How do I fix a leaking faucet?\"\n\ndocuments = [\n \"Plumbing repair guide for the home\",\n \"Stop a dripping tap step by step\",\n \"Best chocolate cake recipe\",\n \"How to train for a marathon\",\n \"Replacing a worn washer in a faucet\",\n]\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\ndef rank(model_name):\n # TODO: embed query + documents with model_name, return documents sorted by cosine\n ...\n\n# TODO: rank with \"text-embedding-3-small\" and \"text-embedding-3-large\" and print both orderings\n```\n\n\n### Expected Output\n\nBoth models put the faucet and plumbing documents on top, but the exact order or score gaps may differ.\n\n\n```plaintext\ntext-embedding-3-small\n 1. Replacing a worn washer in a faucet\n 2. Stop a dripping tap step by step\n 3. Plumbing repair guide for the home\n 4. How to train for a marathon\n 5. Best chocolate cake recipe\ntext-embedding-3-large\n 1. Replacing a worn washer in a faucet\n 2. Stop a dripping tap step by step\n 3. Plumbing repair guide for the home\n 4. Best chocolate cake recipe\n 5. How to train for a marathon\n```\n\n\n### Solution\n\n\n**main.py**\n\n```python\nimport math\nfrom openai import OpenAI\n\nclient = OpenAI() # Uses OPENAI_API_KEY from environment\n\nquery = \"How do I fix a leaking faucet?\"\n\ndocuments = [\n \"Plumbing repair guide for the home\",\n \"Stop a dripping tap step by step\",\n \"Best chocolate cake recipe\",\n \"How to train for a marathon\",\n \"Replacing a worn washer in a faucet\",\n]\n\ndef cosine_similarity(a, b):\n dot = sum(x * y for x, y in zip(a, b))\n return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))\n\ndef rank(model_name):\n response = client.embeddings.create(model=model_name, input=[query] + documents)\n vectors = [item.embedding for item in response.data]\n qv, dvs = vectors[0], vectors[1:]\n scored = sorted(((cosine_similarity(qv, dvs[i]), documents[i]) for i in range(len(documents))), reverse=True)\n return [text for _, text in scored]\n\nfor model_name in [\"text-embedding-3-small\", \"text-embedding-3-large\"]:\n print(model_name)\n for i, text in enumerate(rank(model_name), start=1):\n print(f\" {i}. {text}\")\n```\n\n\nRunning both models through the same `rank` helper isolates the model as the only variable, so any difference in ordering is attributable to the embedding, not the code. On easy cases the two often agree, which tells you the cheaper, smaller model is good enough; where they diverge on harder pairs is where the larger model may earn its extra cost and latency. That comparison, not the dimension count, is how you actually pick.","pageType":"ai-engineering"}

Get Premium