How to Collect Data for Machine Learning: A Practical Guide for ML Teams

A dataset can be broken before collection starts. Three things do it: a label nobody pinned down, a sample that skipped the hard cases, and consent that does not cover the use. None of them show up in your metrics. All of them show up in production.

Here is the sequence that survives production: pick sources that match where the model will run, write the spec before collecting, annotate with QC that catches drift, and test for leakage before you spend the compute. 

Why Is Data Collection the Hardest Part of ML Projects?

Most ML failures are data problems wearing model clothes. The model converges, the validation accuracy looks strong, and then the system fails in production — because the data did not match the real-world distribution in ways nobody noticed.

Four Specific Failure Modes That Repeat Across Projects

The World Is Messier Than Your Logs 

Users behave differently across devices, regions, and channels. If you capture only one slice, your model learns one reality and meets another at inference time.

Labels Are Business Logic in Disguise

In supervised ML, the label definition encodes what "correct" means. If annotators interpret it differently — because the guidelines are ambiguous, because edge cases are absent, because no calibration round happened — the model trains on contradictions. Inter-annotator agreement below 0.8 on a classification task is a signal the label definition needs rework before more data is collected. [1]

Sampling Bias Enters Quietly

Your dataset reflects who showed up, who got measured, and who was excluded. That is not a philosophical point — it is a performance variable. A model trained on data skewed toward majority groups, dominant languages, or well-lit environments will systematically underperform on everything else. [2]

Legal Constraints Are Real Constraints

Some data is not collectible — or not collectible the way you want — without consent, contracts, or documented safeguards. Teams that treat privacy as a cleanup task instead of a design input create compliance gaps that are nearly impossible to fix retroactively once a dataset is distributed. 

What Types of Data Does Your ML Model Actually Need?

Before choosing sources, clarify what you are collecting. Raw data falls into three categories with different collection requirements and storage implications. [3]

Structured data lives in tables with fixed schemas — transaction records, sensor readings with defined fields, user event logs. It is the easiest to validate and the most straightforward to ingest into training pipelines.

Semi-structured data has identifiable elements but no rigid schema: JSON API responses, XML exports, CSV files with variable column sets. Consistent parsing and schema validation matter more here.

Unstructured data — text, images, audio, video, sensor streams — represents most of what modern ML models are trained on, and up to 80–90% of the total data generated in the world. [3] It is easy to collect in volume and hard to use without annotation.

One practical implication: unstructured data collection and annotation are inseparable planning items. If you budget for collection without budgeting for annotation, you will end up with raw assets that cannot enter a training pipeline.

Beyond format, every training example has three layers:

• Inputs (features): the signals the model can see — pixels, token sequences, sensor readings, event metadata
• Targets (labels): what the model should predict — a class, a bounding box, a transcript, a numeric score
• Context (metadata): timestamps, acquisition device, environment conditions, annotator ID, dataset version

Context is where unexplained model failures usually live. Without it, you cannot answer "why did accuracy drop last week?" or reproduce the training conditions for a dataset from six months ago. 

How Do You Define the ML Task Before Collecting Anything?

The common mistake: starting with available data and fitting a task to it. The fix: write a one-sentence task definition first, then design the data collection to match it.

What Does a Usable Task Definition Look Like?

Write the task in plain language a non-ML stakeholder would agree with:

• "Detect fraudulent card transactions in near real time."
• "Classify incoming support tickets into 14 routing categories."
• "Estimate delivery time at checkout given cart contents and location."
• "Flag safety policy violations in user-generated images."

Then translate it into an ML task type — classification, regression, detection, segmentation, ranking, retrieval. This determines what your labels need to be and what "ground truth" even means.

How Do You Define the Label Without Letting It Drift?

Label definition is the step teams most consistently underinvest in. A label is not a category name — it is a decision boundary. Define:

• What counts as a positive example — specifically, not abstractly
• What is ambiguous enough to be flagged as "uncertain" instead of forced into a class
• What time window defines the outcome (for churn prediction, "whenever they feel like leaving" is not a label definition)
• What the cost asymmetry is between false positives and false negatives — this affects where to set thresholds and which hard cases to prioritize in collection

If you cannot write the label definition in three sentences with at least two negative examples, you are not ready to collect. You will collect data and then discover the definition in production — at full cost.

How Do You Specify the Population You Need to Generalize To?

Define explicitly:

• Which users, regions, devices, and languages the model will serve
• Which environments the model will operate in — lighting conditions for vision, channel and accent distribution for audio, domain and register for text
• What is intentionally out of scope

This becomes your sampling plan. It also becomes your evaluation plan. Any population you do not include in the collection will become a blind spot in evaluation, which will become a failure mode in production. 

What Are the Main Data Sources for ML — and When Does Each Work?

There is no universally best source. The right choice depends on your task, legal constraints, timeline, and how closely you need to match your deployment environment.

First-Party Data: Product Logs and Internal Systems

First-party data — app telemetry, CRM records, support transcripts, transaction logs, call center audio — matches your actual users and workflows. It is the strongest foundation for production ML when the instrumentation is consistent.

When it works well: you control collection frequency and schema, the distribution matches your deployment population, and you can update the dataset continuously as your product evolves.

Where it breaks: logging is inconsistent across platforms; events lack stable identifiers across sessions or devices; the label exists but is delayed or noisy (fraud chargebacks arrive days after transactions; support tickets get reclassified after resolution).

Practical requirement: treat instrumentation as part of the ML system. Define event schemas and required fields before collection starts. If you train on whatever the logging system happened to emit, you will not be able to reproduce training conditions when you need to retrain.

Open and Public Datasets: Useful for Prototypes, Limited for Production

Open datasets — Kaggle, Hugging Face Datasets, UCI ML Repository, Google Dataset Search, domain-specific repositories — are the fastest way to test modeling feasibility and compare model families. They are rarely a direct match for production unless your problem closely resembles an existing benchmark task.

Use them for: baseline experiments, pre-training before domain-specific fine-tuning, and evaluation benchmarks where your problem aligns with an established one.

Two common traps: distribution mismatch (the dataset was collected in a different environment, demographic, or language than your deployment) and licensing restrictions (many academic datasets explicitly prohibit commercial use). Before relying on a public dataset, verify: the collection method, the consent basis, and the exact license terms. [4] CC0 and Public Domain licenses permit unrestricted commercial use; CC BY-NC licenses do not. Assuming a dataset is free for commercial use because it is publicly accessible is a compliance risk that surfaces at the wrong moment.

Third-Party Data Providers: Scale with Contracts

Data vendors provide labeled datasets, enrichment data, or raw data collected via panels, partners, or proprietary pipelines. They are most useful when you need scale quickly or niche signals — a specific language, a rare demographic, a specialized sensor modality — that you cannot collect in-house in the available time.

What to verify before engaging a provider: collection methodology, participant consent records, label guidelines and reviewer process, quality metrics you can audit (inter-annotator agreement, rework rates, gold example accuracy), security posture, and dataset documentation. [5]

If the vendor cannot explain how they prevent label drift across annotators and over time, you will inherit that drift in your training data.

Web Scraping: Powerful, Easy to Misuse

Scraping can be valuable for public web content, product catalogs, and research corpora. It is also where teams accidentally collect personal data, violate platform terms, or build pipelines that break silently when page layouts change.

Use it when: the target content is genuinely public, your use case is legally defensible, you have a plan for deduplication and licensing review, and you can detect layout drift.

Build scraping like an engineering system: schema validation, parsing tests, and monitoring for structural changes. If you cannot detect that a page layout changed, you will train on silently broken data — confidently.

A harder requirement: scraping frequently captures personally identifiable information incidentally (usernames, profile images, location data in metadata). Under GDPR and equivalent frameworks, this triggers data subject rights obligations even if collection was technically permitted. [6] Build a PII-detection pass into the pipeline before data enters storage.

Sensor and Field Collection: High Signal, High Variability

Computer vision, audio recognition, robotics, IoT, and autonomous systems depend on sensor data: cameras, LiDAR, radar, microphones, depth sensors, accelerometers. This data has the highest fidelity to the deployment environment — and the most ways to introduce inconsistency.

Critical metadata to capture: sensor model and firmware version, calibration parameters, capture settings (resolution, frame rate, exposure), environment conditions, and session ID. Without this, you cannot reproduce training conditions or diagnose field failures.

For egocentric robotics data collection — 100+ hours of first-person video across 15,000+ scenarios — we capture alongside each session the equipment configuration, environment type, and structured scenario ID. Without that metadata, footage from different sessions would appear to come from the same distribution when the policy-relevant factors vary considerably. [15]

Different hardware creates different data distributions, even for nominally identical capture tasks. If your training data comes from one sensor model and your deployment uses another, the distribution gap is a known risk that needs explicit evaluation.

Crowdsourcing and Field Collection: Scalable with Guardrails

Crowdsourcing collects human judgments, speech samples, photographs, survey responses, or annotations at scale. It is the practical path to demographic diversity and linguistic coverage that in-house collection cannot reach.

For a palm recognition dataset — 20,000 palm photo sets — we ran collection through Prolific with filters by hand characteristics and demographics, completing delivery in two months with consent documentation built into the collection protocol. [16] The Prolific model means each participant has a documented consent record, which matters for downstream use in products operating under GDPR.

Where crowdsourcing fails: vague task instructions produce label drift; incentives that reward speed over accuracy produce rushed, low-quality outputs; demographic coverage is uneven if you do not explicitly filter for it. The fix is qualification tests, gold examples with known answers, and disagreement tracking. If you do not measure annotator reliability, you will ship the average of misunderstandings. 

When Does Synthetic Data Make Sense — and When Does It Not?

Synthetic data fills coverage gaps that real-world collection cannot cost-effectively close: rare failure modes, dangerous scenarios, privacy-sensitive situations where collecting real data requires consent processes that do not scale. [7]

For tabular data with class imbalance, SMOTE (Synthetic Minority Oversampling Technique) generates synthetic samples of minority classes by interpolating between existing examples, helping balance the distribution without distorting the majority class representation. [8] For vision tasks, augmentation — rotations, flips, color jitter, cropping — expands coverage of visual variation the model needs to handle. For NLP, back-translation and paraphrase generation can diversify phrasing without changing meaning.

Two limits that repeat in practice. First, synthetic data does not replace real-world data for deployment-critical evaluation — it can train, but only real-world test sets reveal what actually happens at inference. Second, generative models used to produce synthetic data can encode the same biases as the training data they were built on. [9] Synthetic data created from a biased real-world corpus inherits those biases at scale.

The practical rule: use synthetic data to expand coverage of rare scenarios and correct imbalance; validate performance on real-world evaluation sets before any deployment decision. 

What Does a Reliable Data Collection Pipeline Look Like?

A pipeline that holds up has six sequential steps. Skipping steps or reordering them does not speed up the process — it moves the cost to later, where it is harder to fix. 

Step 1: Write a Data Specification First

A data spec is an engineering contract for your dataset. It forces the decisions that would otherwise be made implicitly, inconsistently, or too late. Include:

• Task definition and label rules, with positive and negative examples
• Required fields and data types
• Acceptable missingness rules — what can be null, what cannot
• Metadata requirements: timestamps, source identifier, acquisition method, sensor configuration
• Privacy and security constraints
• Quality checks: duplicate thresholds, resolution minimums, language filters, format requirements
• Update cadence and versioning approach

If you cannot write this spec, you are not ready to collect. You will collect "stuff" and then argue about what it means.

Step 2: Design the Sampling Plan Before Collection

Sampling is where bias enters quietly and permanently. A sampling plan answers:

• Which sources represent the real deployment population
• How you will avoid over-collecting easy or common cases at the expense of hard ones
• How you will include rare categories, edge cases, and underrepresented subgroups
• How you will handle class imbalance — whether through collection strategy, reweighting, or augmentation

For a medical imaging dataset covering alopecia diagnosis — 350 male and 150+ female participants across multiple severity stages — the sampling plan was built around clinical severity scale coverage, not just participant count. [17] Collecting 500 participants clustered at mild presentation would produce a dataset that fails on the severe cases most critical for diagnostic use.

Step 3: Build the Collection Pipeline as an Engineering System

Even modest-scale collection needs: ingestion (batch pulls, streaming events, uploads, API calls, field capture uploads); validation (schema checks, required fields, anomaly detection); storage with a clear separation between raw, processed, and labeled zones; lineage tracking (which transform produced which dataset version); and monitoring for volume shifts, missing fields, and sudden new categories.

Batch ingestion vs. streaming ingestion is a design choice made at this step, not later. Batch ingestion — pulling data in scheduled chunks — works for stable, archival, or planned collection. Streaming ingestion — continuous acquisition as data is generated — is necessary for fraud detection, real-time sensor feeds, and any task where data staleness degrades model performance. [3] Choosing the wrong model creates downstream retraining friction that compounds over the project's life.

Raw data is your evidence locker. When something breaks — and it will — raw data is what lets you answer "what changed" and reprocess correctly.

Step 4: Label and Annotate with Quality Controls

If your labels come from human annotators, your annotation guidelines are the model's training manual. Write them like you are training a new team member: include edge cases, exclusion rules, and examples of the hard calls — not just the obvious ones.

The annotation step is where first-party data problems become visible. A 363,000-file audio dataset for banking call center automation arrived with existing operator-assigned categories — but those categories had been applied inconsistently across teams. Producing clean labels for NLP automation required 50 days of structured annotation work to resolve the inconsistencies before training could begin. [18] Raw recordings existing is not the same as training-ready labels existing.

For an Arabic-language LLM evaluation dataset spanning three months of annotation work, the annotation guidelines required iterative calibration before inter-annotator agreement stabilized at a level suitable for LLM quality evaluation. [13] That refinement cycle is not wasted time — it is the work that prevents label noise from scaling with dataset size.

Quality controls that should be standard: gold items (examples with known correct answers embedded in the annotation queue), redundant labeling for ambiguous examples with an adjudication process, and disagreement tracking that feeds back into guideline updates. Track label versions. Label definitions change over time — what breaks projects is changing the definition without versioning, so examples labeled under different guidelines end up in the same training set.

Three-tier QC — annotator → reviewer → QA audit — is what we run on every project at Unidata. It is the step that determines whether dataset size translates into model quality. 

Step 5: Run Data Quality Checks Before Training

Do this before committing compute and time to training. Checks that catch the most expensive problems:

• Duplicates — exact and near-duplicate; near-duplicates cause data leakage across train and test splits [10]
• Leakage — features that directly encode the label; leakage creates models that look accurate in evaluation and fail in production [10]
• Spurious shortcuts — watermarks, UI elements, template phrases that correlate with labels for accidental reasons
• Missingness — fields that should be present but are not
• Distribution drift — how this dataset version compares to the previous one
• Segment coverage — are all the subgroups you need to serve represented?

If you run one check only, run the leakage check. A model trained on leaked data will achieve near-perfect evaluation metrics and produce unreliable predictions in production.

Step 6: Document the Dataset Before It Leaves Your Hands

Good dataset documentation answers five questions: Where did this data come from? What does it represent? What is missing? What should it not be used for? What are the known limitations and risks? [5]

A "datasheet for datasets" format — proposed by Gebru et al. and now widely adopted — provides a standard structure for this: motivation, composition, collection process, labeling process, intended uses, known limitations, privacy and licensing, and versioning. [5] It does not need to be long. It needs to exist. 

How Do You Handle Imbalanced Datasets?

Class imbalance is one of the most common data collection problems, and one of the least visible until it causes production failures. A model trained on 95% negative examples and 5% positive examples will achieve 95% accuracy by predicting "negative" for everything — and be useless for the actual task.

Three strategies, applied in combination depending on the severity of the imbalance:

Collection-level: design the sampling plan to target minority class examples explicitly. For domains where rare events are the point — fraud, equipment failure, medical edge cases — this means actively designing scenarios and recruitment criteria to surface them, not waiting for them to appear at natural frequency.

Augmentation-level: SMOTE for tabular minority classes; targeted image augmentation for vision tasks; back-translation for NLP. These expand the representation without distorting the overall distribution. [8]

Training-level: class weighting in the loss function penalizes errors on minority classes more heavily, directing model attention without changing the dataset. Precision-recall curves and F1 scores by class are more informative evaluation metrics than overall accuracy when class imbalance is present. [11]

One caution: over-sampling minority classes by simple duplication (not synthetic generation) can cause the model to memorize specific examples rather than learn the underlying pattern. Prefer SMOTE or targeted collection over naive duplication. 

What Privacy, Consent, and Licensing Rules Apply?

What Counts as Personal Data?

Under GDPR, personal data is any information relating to an identified or identifiable natural person. [6] This includes obvious identifiers — names, email addresses — and less obvious ones: location data, IP addresses, device identifiers, biometric data (face images, voice recordings, palm photos), and behavioral data that can identify individuals in context.

For any dataset that includes human subjects, participant consent documentation is not optional. It is the legal basis for processing. For our alopecia image datasets — collected from 350+ male and 150+ female participants — consent covered the specific capture purpose, data use, storage location, and retention period. [17] Retroactive consent processes on distributed datasets are not practically achievable; the consent infrastructure has to be built into the collection protocol from the start.

Privacy-by-design in practice means three things:

• Minimize collection: capture only what the model needs, not everything available
• Environment preparation: for field and egocentric collection, prepare the capture environment before recording begins — remove identifying documents, personal photographs, and any artifacts that reveal individuals' identities. This prevents most privacy issues from entering footage in the first place.
• Separate identifiers from content: pseudonymize participant records at the point of collection; store identifiers and content in separate access-controlled systems

GDPR fines for certain infringements can reach up to €20,000,000 or 4% of worldwide annual turnover. [6] That figure is not a threat; it is a design constraint. The cost of building consent infrastructure into collection is a fraction of the cost of discovering the gap at scale.

What Licensing Applies to External Data?

For open datasets, scraping targets, and vendor data, confirm before using:

• License terms for commercial use — CC0 and ODC-PDDL permit unrestricted use; CC BY requires attribution; CC BY-NC prohibits commercial use [4]
• Whether redistribution is permitted
• Whether derivative works (models trained on the data) are restricted
• Attribution requirements

If terms are ambiguous, do not assume permissiveness. Replace the source or negotiate rights explicitly. 

Should You Collect Data In-House or Work with a Data Partner?

This is a capacity decision, not a philosophical one. The right answer depends on what you are trying to build and what your team can execute at the required quality level and timeline. 

Collect in-house when: the data is tightly coupled to your product instrumentation; you need rapid iteration between model behavior, product, and data pipeline; or the dataset is strategically sensitive and needs to stay within your infrastructure.

Work with a data partner when: you need scale faster than your team can build; you need specialized collection infrastructure (medical annotation by clinicians, multi-sensor field capture, multilingual annotation across 32+ languages); or you need managed quality with auditable SLAs. For the palm recognition dataset — 20,000 photo sets in two months with consent documentation and demographic filters — building that infrastructure in-house would have taken significantly longer than engaging a partner with established crowdsourcing pipelines. [16]

For more detail on how to evaluate the build vs. buy tradeoff for your specific situation, read Build or Buy ML Dataset: What to Choose? For an overview of how much data production-grade models typically require at each stage, see the Training Data Guide.

When evaluating a data partner, ask for: the collection methodology and sampling approach; label guidelines and the annotator review process; quality metrics you can audit; security posture and access controls; dataset documentation and versioning approach. If the vendor cannot explain how they detect and prevent label drift, you will inherit it.

If you need a partner for data collection or annotation — from crowdsourcing through sensor capture and 3-tier QC — see. 

What Are the Most Common Data Collection Mistakes?

Collecting what is available instead of what is needed. Write a data spec and sampling plan first. Make availability a constraint to work around — not the driver of what you collect.

Defining labels on the fly. Freeze the label definition before annotation begins, train annotators against it, and then revise deliberately with versioning. Mid-project label redefinitions without versioning produce training sets where early and late examples mean different things.

Ignoring edge cases until production. Hard cases slow down collection. They are also where model credibility is established or lost. For forest mapping annotation — 200,000 trees across 10 species classes — ambiguous cases had to be resolved in the annotation guidelines before labeling began, not discovered mid-project. [14] Leaving edge cases undefined produces a model that fails precisely on the visually complex examples the annotation team already knew were hard.

Mixing datasets without tracking provenance. If you merge datasets, document why, how, and what transforms were applied. Source tags and collection dates are not nice-to-haves; they are the minimum needed to diagnose problems when model behavior changes between dataset versions.

Measuring only model metrics, not data health. Monitor data volume, schema stability, missing fields, and segment coverage as first-class metrics. Data drift is usually visible before accuracy drops. By the time the model metric moves, the cause is weeks in the past. 

Dataset Documentation Template

Use this as a starting structure before any dataset leaves your hands:

• Motivation: what task and decision this dataset supports
• Composition: modalities, fields, metadata included
• Collection process: how data was captured, from where, with what equipment and filters
• Labeling process: label definitions, annotator guidelines, review process, inter-annotator agreement
• Intended use: where it should be used and where it should not
• Known limitations: missing segments, noisy labels, weak coverage areas
• Privacy and licensing: consent basis, personal data handling, license constraints
• Versioning: dataset version ID, transform code reference, release notes

This does not need to be perfect on the first iteration. It needs to exist before the dataset enters a training pipeline.