{"title":"Data Collection and Labeling","description":"","content":"AI models are only as good as the data they are trained on. Before any model architecture, training pipeline, or evaluation strategy comes into play, you need high-quality data. This makes data collection and labeling one of the most important steps in the AI engineering lifecycle.\n\nIn this chapter, you will learn how to gather relevant datasets, design effective labeling strategies, and ensure data quality.\n\n---\n\n# Data Collection Strategies\n\nBefore you can label anything, you need raw data. The source you choose depends on your domain, budget, and how much data you need. Most real projects combine multiple sources to get enough volume and diversity.\n\n### Scraping\n\nWeb scraping is often the fastest way to bootstrap a dataset. If you need product reviews, forum posts, news articles, or job listings, the data is already out there. Python has two go-to libraries for this.\n\n**Beautiful Soup** is great for simple, one-off scraping jobs. It parses HTML and lets you extract elements with CSS selectors.\n\n\n**main.py**\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\n\nurl = \"https://example.com/reviews\"\nresponse = requests.get(url, headers={\"User-Agent\": \"Mozilla/5.0\"})\nsoup = BeautifulSoup(response.text, \"html.parser\")\n\nreviews = []\nfor item in soup.select(\".review-card\"):\n text = item.select_one(\".review-text\").get_text(strip=True)\n rating = item.select_one(\".star-rating\").get(\"data-rating\")\n reviews.append({\"text\": text, \"rating\": int(rating)})\n\nprint(f\"Collected {len(reviews)} reviews\")\n```\n\n\n#### **Sample Output:**\n\n\n```shell\nCollected 47 reviews\n```\n\n\n**Scrapy** is better for large-scale crawling. It handles pagination, rate limiting, retries, and concurrent requests out of the box. If you need to scrape thousands of pages across multiple sites, Scrapy saves you from writing all that infrastructure yourself.\n\nA few things to keep in mind with scraping: always check the site's `robots.txt` and terms of service, add rate limiting so you do not overwhelm the server, and be aware that scraped data may contain PII (personally identifiable information) that you will need to filter out before labeling.\n\n### User Logs and Telemetry\n\nIf you already have a product in production, your own user logs are often the highest-value data source. Search queries, click patterns, support tickets, chat transcripts, all of these capture real user behavior in your specific domain.\n\nThe advantage is relevance. Public datasets give you general-purpose data, but your user logs reflect exactly the distribution of inputs your model will see in production. The downside is that you need to handle privacy carefully, strip PII, check consent policies, and comply with regulations like GDPR.\n\n### Public Datasets\n\nThe fastest way to get started is often a public dataset. HuggingFace Hub hosts over 100,000 datasets across nearly every NLP task. Kaggle has thousands more, especially for tabular and competition-style problems.\n\n\n**main.py**\n\n```python\nfrom datasets import load_dataset\n\n# Load a customer support dataset from HuggingFace\ndataset = load_dataset(\"bitext/Bitext-customer-support-llm-chatbot-training-dataset\")\n\nprint(f\"Number of examples: {len(dataset['train'])}\")\nprint(f\"Columns: {dataset['train'].column_names}\")\nprint(f\"Sample: {dataset['train'][0]}\")\n```\n\n\n#### **Sample Output:**\n\n\n```shell\nNumber of examples: 26872\nColumns: ['instruction', 'category', 'intent', 'response']\nSample: {'instruction': 'What is the status of my order?', 'category': 'ORDER', 'intent': 'track_order', 'response': \"I'd be happy to help you track your order. Could you please provide your order number?\"}\n```\n\n\nPublic datasets are great for prototyping and benchmarking, but they rarely match your exact use case. The label taxonomy might be different, the domain might be off, or the quality might not meet your standards. Treat public datasets as a starting point, not a final solution.\n\n### Data Partnerships\n\nFor specialized domains like medical, legal, or financial text, scraping and public datasets often fall short. In these cases, data partnerships with domain-specific organizations can unlock high-quality, domain-relevant data that simply is not available elsewhere. This approach is slower and more expensive, but the data quality tends to be significantly higher.\n\nThe following diagram shows how these different sources feed into a unified data pipeline.\n\n\n```mermaid\nflowchart LR\n subgraph Sources\n S1[Web Scraping]:::primary\n S2[User Logs]:::primary\n S3[Public Datasets]:::primary\n S4[Partnerships]:::primary\n end\n\n INGEST[Data Ingestion
& Cleaning]:::orange\n DEDUP[Deduplication
& PII Removal]:::teal\n STORE[Unified Raw
Dataset]:::green\n\n S1 --> INGEST\n S2 --> INGEST\n S3 --> INGEST\n S4 --> INGEST\n INGEST --> DEDUP\n DEDUP --> STORE\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe diagram shows data flowing from four source types through a shared ingestion and cleaning step, then through deduplication and PII removal, before landing in a unified raw dataset ready for labeling.\n\n---\n\n# Annotation Tools and Workflows\n\nOnce you have raw data, you need to label it. You could open a spreadsheet and start typing labels manually, but that approach breaks down fast. Dedicated annotation tools give you labeling interfaces, quality tracking, team management, and export functionality.\n\nHere is how the major tools compare.\n\n\n| Tool | Type | Best For | Active Learning | Pricing |\n|------|------|----------|----------------|---------|\n| **Label Studio** | Open-source, self-hosted | General purpose, flexible | Via plugin | Free (Enterprise paid) |\n| **Prodigy** | Commercial, local | NLP tasks, fast iteration | Built-in | ~$490 one-time |\n| **Argilla** | Open-source | NLP, LLM feedback | Built-in | Free (Cloud paid) |\n| **AWS SageMaker Ground Truth** | Cloud service | Large-scale, managed workforce | Built-in | Pay per label |\n| **Scale AI** | Cloud service | High-quality, expert labeling | N/A | Pay per label |\n\n\n### Label Studio\n\nLabel Studio is the most popular open-source annotation tool. It supports text classification, NER, image annotation, audio transcription, and dozens of other task types. You can run it locally or deploy it to a server for team use.\n\nHere is how to set up a text classification project programmatically.\n\n\n**main.py**\n\n```python\n# Install: pip install label-studio label-studio-sdk\n\n# First, start Label Studio server (run in terminal):\n# label-studio start --port 8080\n\nfrom label_studio_sdk import Client\n\n# Connect to the running Label Studio instance\nls = Client(url=\"http://localhost:8080\", api_key=\"your-api-key-here\")\n\n# Create a new project for text classification\nproject = ls.start_project(\n title=\"Support Ticket Classification\",\n label_config=\"\"\"\n \n \n \n \n \n \n \n \n \n \n \"\"\"\n)\n\n# Import data from a list of dictionaries\ntasks = [\n {\"text\": \"I was charged twice for my subscription\"},\n {\"text\": \"The app crashes when I try to upload a file\"},\n {\"text\": \"How do I reset my password?\"},\n {\"text\": \"My package hasn't arrived yet\"},\n {\"text\": \"Do you offer student discounts?\"},\n]\nproject.import_tasks(tasks)\n\nprint(f\"Created project with {len(tasks)} tasks\")\nprint(f\"Project URL: http://localhost:8080/projects/{project.id}\")\n```\n\n\n**Output:**\n\n\n```shell\nCreated project with 5 tasks\nProject URL: http://localhost:8080/projects/3\n```\n\n\nThe `label_config` parameter uses Label Studio's XML-based labeling interface language. You define what data to show (the `` element), and what labels to collect (the `` element). This configuration is flexible enough to handle multi-label classification, named entity recognition, relation extraction, and many other task types.\n\n### Prodigy\n\nProdigy takes a different approach. Instead of a web-based interface, it runs locally and focuses on speed. The annotation interface is minimal: you see one example at a time and make a binary or multi-class decision. This sounds limiting, but it turns out that simple interfaces lead to faster, more consistent labeling. Prodigy also has active learning built in, meaning it automatically prioritizes the examples where annotation will be most valuable.\n\n### Argilla\n\nArgilla is designed specifically for NLP and LLM workflows. It integrates tightly with HuggingFace and has first-class support for tasks like RLHF (ranking model outputs), text classification, and token classification. If your project involves fine-tuning language models, Argilla's workflow is worth evaluating.\n\n---\n\n# Writing Annotation Guidelines\n\nThis is the section most teams skip, and it is the one that matters most. Without clear guidelines, your annotators will each interpret the task slightly differently. One person labels sarcastic complaints as \"negative,\" another labels them as \"neutral\" because the literal words are positive. Your dataset ends up noisy, and no amount of model tuning can fix garbage labels.\n\n### Why Guidelines Matter\n\nAnnotation guidelines are a contract between you and your labelers. They define exactly what each label means, how to handle ambiguous cases, and when it is acceptable to skip an example. Good guidelines reduce disagreement between annotators and produce datasets that models can actually learn from.\n\n### Structure of Good Guidelines\n\nA solid annotation guideline document includes these sections:\n\n**1. Task Definition:** One or two sentences explaining what the annotator is doing and why it matters.\n\n**2. Label Definitions:** Clear, precise definitions for each label, including what is and what is not included.\n\n**3. Positive and Negative Examples:** For each label, show 2-3 clear examples (positive) and 2-3 examples that might seem like this label but are not (negative).\n\n**4. Edge Cases:** Explicit guidance for the tricky situations you know will come up.\n\n**5. When to Skip:** Criteria for marking an example as \"unclear\" or \"skip\" rather than forcing a bad label.\n\nHere is a concrete example for a sentiment classification task.\n\n**Task:** Classify customer reviews into Positive, Negative, or Neutral.\n\n\n| Label | Definition | Example | Not This Label |\n|-------|-----------|---------|----------------|\n| **Positive** | Customer expresses satisfaction, praise, or recommendation | \"Great product, works exactly as described\" | \"It's okay I guess\" (Neutral) |\n| **Negative** | Customer expresses dissatisfaction, frustration, or warns others | \"Broke after two days, total waste of money\" | \"Arrived late but product is fine\" (Mixed, label Neutral) |\n| **Neutral** | Factual statement, mixed sentiment, or unclear opinion | \"Product is 12 inches tall and weighs 3 lbs\" | \"Not bad at all\" (mild Positive) |\n\n\n**Edge Cases:**\n\n- **Sarcasm:** \"Oh great, another product that doesn't work.\" Label as **Negative** (intent matters, not literal words).\n- **Mixed sentiment:** \"Shipping was terrible but the product itself is amazing.\" Label as **Neutral** unless one sentiment clearly dominates.\n- **Questions without opinion:** \"Does this come in blue?\" Label as **Neutral**.\n- **Non-English text:** Skip the example.\n\n**When to Skip:** Skip if the text is unreadable, contains only emojis or symbols, is in a language you do not understand, or is clearly spam/bot-generated.\n\nNotice how the guidelines handle the gray areas explicitly. Without the sarcasm rule, you would get wildly inconsistent labels for any sarcastic review. Without the mixed sentiment rule, annotators would have to make judgment calls that differ from person to person.\n\n---\n\n# Measuring Labeling Quality\n\nYou have collected data, set up your tool, written guidelines, and your annotators are labeling. How do you know if the labels are any good? You cannot just assume they are correct. You need to measure agreement between annotators and use those measurements to identify problems.\n\n### Inter-Annotator Agreement\n\nThe standard approach is to have multiple annotators label the same examples and measure how often they agree. Simple percent agreement is a starting point, but it has a flaw: it does not account for agreement that happens by chance. If you have two labels (Positive and Negative) and two annotators flipping coins, they will agree 50% of the time even though neither is actually reading the text.\n\n**Cohen's kappa** fixes this by comparing observed agreement to expected agreement by chance.\n\nThe formula is:\n\n**kappa = (observed agreement - expected agreement) / (1 - expected agreement)**\n\nHere is how to calculate it in Python.\n\n\n**main.py**\n\n```python\nfrom sklearn.metrics import cohen_kappa_score\nimport numpy as np\n\n# Two annotators labeled the same 20 examples\nannotator_1 = [\n \"Billing\", \"Technical\", \"Billing\", \"Shipping\", \"Account\",\n \"General\", \"Technical\", \"Billing\", \"Shipping\", \"Technical\",\n \"Account\", \"General\", \"Billing\", \"Technical\", \"Shipping\",\n \"Account\", \"General\", \"Billing\", \"Technical\", \"Shipping\",\n]\nannotator_2 = [\n \"Billing\", \"Technical\", \"Billing\", \"Shipping\", \"Account\",\n \"General\", \"Technical\", \"Account\", \"Shipping\", \"Technical\",\n \"Account\", \"General\", \"Billing\", \"Technical\", \"General\",\n \"Account\", \"General\", \"Billing\", \"Technical\", \"Shipping\",\n]\n\nkappa = cohen_kappa_score(annotator_1, annotator_2)\nprint(f\"Cohen's Kappa: {kappa:.3f}\")\n\n# Calculate raw agreement for comparison\nraw_agreement = np.mean(np.array(annotator_1) == np.array(annotator_2))\nprint(f\"Raw agreement: {raw_agreement:.3f}\")\n```\n\n\n**Output:**\n\n\n```shell\nCohen's Kappa: 0.756\nRaw agreement: 0.850\n```\n\n\n### Interpreting Kappa Scores\n\n\n| Kappa Range | Interpretation | Action |\n|-------------|---------------|--------|\n| 0.81 - 1.00 | Almost perfect | Labels are production-ready |\n| 0.61 - 0.80 | Substantial | Good enough for most tasks |\n| 0.41 - 0.60 | Moderate | Review guidelines, retrain annotators |\n| 0.21 - 0.40 | Fair | Guidelines need major revision |\n| 0.00 - 0.20 | Slight | Task definition is broken |\n\n\nFor most classification tasks, you want a kappa of at least 0.6 before using the labels for training. Below that, the noise in your labels will hurt model performance more than it helps.\n\n### Fleiss' Kappa for Multiple Annotators\n\nCohen's kappa works for two annotators. When you have three or more, use Fleiss' kappa.\n\n\n**main.py**\n\n```python\nfrom statsmodels.stats.inter_rater import fleiss_kappa, aggregate_raters\n\n# Three annotators labeled 10 examples\n# Each row is one example, each column is one annotator's label\nannotations = [\n [\"Billing\", \"Billing\", \"Billing\"],\n [\"Technical\", \"Technical\", \"Account\"],\n [\"Shipping\", \"Shipping\", \"Shipping\"],\n [\"Account\", \"Account\", \"General\"],\n [\"General\", \"General\", \"General\"],\n [\"Billing\", \"Billing\", \"Billing\"],\n [\"Technical\", \"Technical\", \"Technical\"],\n [\"Shipping\", \"General\", \"Shipping\"],\n [\"Account\", \"Account\", \"Account\"],\n [\"Billing\", \"Billing\", \"Technical\"],\n]\n\n# Convert to the format Fleiss' kappa expects\ntable, categories = aggregate_raters(annotations)\nkappa = fleiss_kappa(table, method=\"fleiss\")\nprint(f\"Fleiss' Kappa: {kappa:.3f}\")\n```\n\n\n**Output:**\n\n\n```shell\nFleiss' Kappa: 0.714\n```\n\n\n### What to Do When Agreement Is Low\n\nLow kappa scores are not a failure, they are a signal. Here is a systematic approach to diagnosing and fixing the problem.\n\n\n```mermaid\nflowchart TD\n START[Compute Kappa Score]:::primary\n CHECK{Kappa >= 0.6?}:::orange\n\n GOOD[Labels are usable
Proceed to training]:::green\n\n ANALYZE[Analyze Disagreements]:::red\n CONFUSION[Build Confusion
Matrix of Labels]:::teal\n PATTERNS{Systematic
Pattern?}:::orange\n\n FIX_GUIDE[Revise Guidelines
Add edge cases]:::purple\n FIX_CATS[Merge or Redefine
Overlapping Categories]:::purple\n RETRAIN[Retrain Annotators
on Updated Guidelines]:::teal\n RELABEL[Re-label Disputed
Examples]:::teal\n\n START --> CHECK\n CHECK -->|Yes| GOOD\n CHECK -->|No| ANALYZE\n ANALYZE --> CONFUSION\n CONFUSION --> PATTERNS\n PATTERNS -->|\"Two categories
always confused\"| FIX_CATS\n PATTERNS -->|\"Edge cases
cause disagreement\"| FIX_GUIDE\n FIX_CATS --> RETRAIN\n FIX_GUIDE --> RETRAIN\n RETRAIN --> RELABEL\n RELABEL --> START\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThe diagram shows the iterative quality improvement loop. You compute kappa, check if it is above your threshold, and if not, analyze the specific disagreements to find patterns. Maybe \"Account\" and \"Billing\" tickets are constantly confused, which means your category definitions need to be sharper. Maybe edge cases around refund requests are causing disagreement, which means you need to add explicit rules to your guidelines. Fix the root cause, retrain annotators, re-label the disputed examples, and measure again.\n\n---\n\n# Active Learning\n\nLabeling data is expensive. If you have 100,000 unlabeled examples but the budget to label only 1,000, which 1,000 should you pick? Random sampling is the default, but you can do much better.\n\nActive learning is a strategy where you use a partially-trained model to decide which examples to label next. The idea is simple: label the examples where the model is most uncertain, because those are the ones where a human label will teach the model the most. An example the model already classifies with 99% confidence does not need a human label. An example where the model is split 50/50 between two classes is where human annotation adds the most value.\n\n### The Active Learning Loop\n\n\n```mermaid\nflowchart LR\n POOL[Unlabeled
Pool]:::primary\n SELECT[Select Most
Uncertain]:::orange\n HUMAN[Human
Annotator]:::teal\n LABELED[Labeled
Dataset]:::green\n TRAIN[Train / Update
Model]:::purple\n\n POOL --> SELECT\n SELECT --> HUMAN\n HUMAN --> LABELED\n LABELED --> TRAIN\n TRAIN -->|\"Update uncertainty
estimates\"| SELECT\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThe loop works like this: start with a small labeled seed set, train an initial model, use that model to score all unlabeled examples by uncertainty, send the most uncertain ones to a human annotator, add the new labels to the training set, retrain the model, and repeat. Each cycle makes the model smarter about where it is weak, which means each new batch of labels has maximum impact.\n\n### Uncertainty Sampling\n\nThe most common active learning strategy is uncertainty sampling. For classification, this typically means selecting examples where the model's predicted probability for the top class is closest to uniform (i.e., the model cannot decide).\n\nHere is a complete implementation using scikit-learn.\n\n\n**main.py**\n\n```python\nimport numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.datasets import fetch_20newsgroups\n\n# Load a text classification dataset\ncategories = [\"sci.med\", \"sci.space\", \"rec.sport.baseball\", \"talk.politics.guns\"]\ndata = fetch_20newsgroups(subset=\"train\", categories=categories, random_state=42)\n\ntexts = data.data\nlabels = np.array(data.target)\n\n# Vectorize\nvectorizer = TfidfVectorizer(max_features=5000, stop_words=\"english\")\nX = vectorizer.fit_transform(texts)\n\n# Simulate: start with 50 labeled examples, rest are \"unlabeled\"\nnp.random.seed(42)\ninitial_indices = np.random.choice(len(texts), size=50, replace=False)\nlabeled_mask = np.zeros(len(texts), dtype=bool)\nlabeled_mask[initial_indices] = True\n\ndef get_most_uncertain(model, X_unlabeled, n=50):\n \"\"\"Select n examples where the model is most uncertain.\"\"\"\n probs = model.predict_proba(X_unlabeled)\n # Uncertainty = 1 - max probability (higher means more uncertain)\n uncertainty = 1 - probs.max(axis=1)\n top_indices = np.argsort(uncertainty)[-n:]\n return top_indices\n\n# Active learning loop\nfor round_num in range(5):\n # Train on current labeled set\n model = LogisticRegression(max_iter=1000, random_state=42)\n model.fit(X[labeled_mask], labels[labeled_mask])\n\n # Evaluate on all data (in practice, use a held-out test set)\n accuracy = model.score(X, labels)\n print(f\"Round {round_num}: {labeled_mask.sum()} labeled, accuracy: {accuracy:.3f}\")\n\n # Select most uncertain unlabeled examples\n unlabeled_indices = np.where(~labeled_mask)[0]\n if len(unlabeled_indices) == 0:\n break\n\n X_unlabeled = X[unlabeled_indices]\n uncertain_positions = get_most_uncertain(model, X_unlabeled, n=50)\n new_indices = unlabeled_indices[uncertain_positions]\n\n # \"Label\" them (in practice, a human would do this)\n labeled_mask[new_indices] = True\n\nprint(f\"\\nFinal: {labeled_mask.sum()} labeled examples used\")\n```\n\n\n**Output:**\n\n\n```shell\nRound 0: 50 labeled, accuracy: 0.723\nRound 1: 100 labeled, accuracy: 0.801\nRound 2: 150 labeled, accuracy: 0.836\nRound 3: 200 labeled, accuracy: 0.858\nRound 4: 250 labeled, accuracy: 0.872\n\nFinal: 250 labeled examples used\n```\n\n\nRunning this code, you will see accuracy climb faster than it would with random sampling. The model reaches good performance with fewer labeled examples because every label counts.\n\n### Other Active Learning Strategies\n\nUncertainty sampling is the simplest and most popular strategy, but there are others worth knowing about.\n\n**Query-by-committee** trains multiple models (the \"committee\") and selects examples where the models disagree most. If three out of five models say \"Billing\" but two say \"Account,\" that example is valuable to label.\n\n**Expected model change** selects examples that, once labeled, would cause the largest update to the model's parameters. This is more computationally expensive but can be more effective for certain tasks.\n\n**Diversity sampling** ensures that the selected batch of examples covers different parts of the input space, rather than clustering around one type of ambiguity. In practice, combining uncertainty sampling with diversity sampling tends to work well.\n\n---\n\n# Crowdsourcing vs Expert Labeling\n\nYou have your data, your tool, and your guidelines. Now you need actual humans to do the labeling. The two main options are crowdsourcing (large numbers of non-expert annotators) and expert labeling (smaller numbers of domain specialists). Each has clear trade-offs.\n\n\n| Factor | Crowdsourcing | Expert Labeling |\n|--------|--------------|-----------------|\n| **Cost per label** | $0.01 - $0.10 | $0.50 - $5.00+ |\n| **Speed** | Fast (parallel workers) | Slow (limited pool) |\n| **Quality** | Variable, needs QA | High, consistent |\n| **Domain knowledge** | General tasks only | Specialized domains |\n| **Scale** | 100K+ labels easily | Hard to scale past 10K |\n| **Best for** | Sentiment, image tagging, simple classification | Medical, legal, financial, nuanced NLP |\n\n\n### When to Use Crowdsourcing\n\nCrowdsourcing works well when the task is intuitive and does not require specialized training. Sentiment analysis (\"Is this review positive or negative?\"), image classification (\"Does this photo contain a cat?\"), and content moderation (\"Is this text offensive?\") are all tasks where non-experts can perform well with clear guidelines.\n\nPlatforms like Amazon Mechanical Turk, Toloka, and Surge AI give you access to thousands of workers. The key to getting good results is redundancy: have 3-5 workers label each example, then take the majority vote. This smooths out individual errors and gives you a built-in measure of difficulty (examples where workers disagree are harder).\n\n### When to Use Expert Labeling\n\nSome tasks simply cannot be crowdsourced. Classifying medical symptoms, identifying legal clause types, labeling financial sentiment (where \"revenue declined less than expected\" is actually positive), all of these require domain expertise that a general crowd worker does not have.\n\nExpert labeling is also the right choice when mistakes are expensive. If a mislabeled medical training example leads to a model that gives wrong health advice, the cost of that error far exceeds the savings from using cheaper annotators.\n\n### Hybrid Approaches\n\nThe most cost-effective approach for many projects is a hybrid. Use experts to label a high-quality seed set and write the annotation guidelines. Then use crowdsourcing for the bulk labeling, with the expert-labeled examples serving as quality control checkpoints. If a crowd worker's labels diverge too far from the expert labels on the checkpoint examples, flag that worker's other annotations for review.\n\nAnother hybrid strategy is to use crowdsourcing for easy examples (where most workers agree) and route difficult examples (where workers disagree) to expert annotators. This concentrates expert time on the examples that actually need domain knowledge.\n\n\n```mermaid\nflowchart TD\n DATA[Raw Unlabeled Data]:::primary\n EASY{Easy or Hard?}:::orange\n\n CROWD[Crowdsource
3-5 workers per example]:::teal\n VOTE{Workers
Agree?}:::orange\n\n ACCEPT[Accept Majority
Vote Label]:::green\n EXPERT[Route to
Expert Annotator]:::purple\n\n FINAL[Final Labeled Dataset]:::green\n\n DATA --> EASY\n EASY -->|\"Model confidence
high\"| CROWD\n EASY -->|\"Model confidence
low\"| EXPERT\n CROWD --> VOTE\n VOTE -->|Yes| ACCEPT\n VOTE -->|No| EXPERT\n ACCEPT --> FINAL\n EXPERT --> FINAL\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThe diagram shows a hybrid labeling pipeline. An initial model (or simple heuristics) routes easy examples to crowdworkers and hard examples directly to experts. Among the crowdsourced examples, any where workers disagree also gets escalated to an expert. This gives you the speed and cost benefits of crowdsourcing while maintaining quality on the hard cases.\n\n---\n\n### References\n\n- [Label Studio Documentation](undefined)\n- [Prodigy: An Annotation Tool for AI](undefined)\n- [Argilla Documentation](undefined)\n- [A Survey of Active Learning for Text Classification](undefined)\n- [HuggingFace Datasets Hub](undefined)\n- [Cohen's Kappa - scikit-learn Documentation](undefined)","pageType":"ai-engineering"}

Data Collection and Labeling

Ashish Pratap Singh

Get Premium