What Is Crowdsourcing? Complete Guide

Crowdsourcing is the cheapest way to buy training data by the item, and one of the most expensive ways to end up with training data you can actually use. Both are true at once, which is why teams keep arriving at it and keep being surprised by the bill.

The word covers one simple move: instead of handing a job to a hired team, you post it openly and whoever wants it takes it. That stretches across Wikipedia, Kickstarter and open science competitions, but for an ML team it lands in two specific places: producing raw data, and labeling data that already exists.

It behaves very differently at each — close to irreplaceable in one, hard to defend in the other. Everything below works toward deciding which of the two, if either, your own pipeline should hand over.

Checked by Expert

Kirill Meshyk

Head of AI Data Collection

LinkedIn

• Data Collection
• AI Data Collection
• Crowdsourcing
• Biometric Data
• Synthetic Data

What is crowdsourcing? Definition and origin

Crowdsourcing means getting ideas, labor, or content from a large, distributed group of people through a public, open call. Nobody is assigned the task. It goes out to anyone who might take it on, the people who answer pick themselves, and whatever comes back is pooled, checked, then paid for or simply published.

Outsourcing hands a defined task to one named vendor under a negotiated contract. Crowdsourcing opens the same kind of task to an unbounded, anonymous group and pays per accepted submission, not by the hour or by retainer. The broadcast is the dividing line, even though both models push work outside a company's permanent staff. Researchers landed in roughly the same place: a 2012 paper gathered the competing published definitions, pulled out what they had in common, and tested the result against eleven real cases [2].

Under the mechanism sits the wisdom of the crowd. The idea goes back to Galton's 1907 note in Nature on what happens when you average many independent estimates [1]. "Independent" is the word doing the work there. The effect needs people judging on their own, and a crowd that all sees the same screen and the same instructions can lose it entirely.

The origin and evolution of the term

The word was coined in a conversation. Jeff Howe and his Wired editor Mark Robinson landed on "crowdsourcing" together while developing a story, blending "crowd" and "outsourcing." Howe introduced it to a wide audience in "The Rise of Crowdsourcing," in the June 2006 issue of Wired [3]. The article looked at established companies that had already swapped contracted specialists for open calls to the public. Howe expanded the idea into a full book, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, in 2008 [4].

The word survived because the infrastructure under it kept growing. Cheap broadcast reach, payment platforms, and a large pool of people willing to take on short paid tasks arrived together. Short-term and contract work in the US rose from 10.7% to 15.8% of the workforce over the same decade [5], a trend the same authors later revised down to a modest one once survey and tax records were compared [6]. By the time ML teams needed labeled data at volume, both the distribution channel and the workforce already existed.

How crowdsourcing works: open calls and distribution

Strip away the specific use case and crowdsourcing always follows the same loop. A task goes out through an open call, usually via a crowdsourcing platform: a marketplace that handles distribution, submissions, and payment so the requester doesn't have to build any of it. Nobody is assigned, so anyone who sees the call decides for themselves whether to take it. What comes back is pooled and checked against a quality bar, and contributors get paid per accepted piece of work, or simply credited.

What varies is how small the pieces are. A logo-design contest and a thousand-item labeling batch run the same broadcast-contribute-aggregate loop, but design and research crowdsourcing collect a handful of full submissions. Labor-style crowdsourcing cuts the work into microtasks: units small enough to finish in seconds to minutes, such as labeling one image. That size decides whether a crowd can absorb the work at all, and it is the first thing to check when the work is data collection or annotation, meaning labeling data so a model can learn from it.

Crowdsourcing vs. outsourcing vs. insourcing

Three sourcing models get used interchangeably. They differ on two things: who does the work, and how it's paid.

The boundary between the last two is blurrier than the table makes it look. For document-heavy and data-heavy tasks, crowdsourcing works as a variant of outsourcing, since both push work off the payroll. The cleaner test is contractual: one named vendor on agreed terms, or an open call to anyone who wants the work.

The main types of crowdsourcing

There is no single standard taxonomy, which is worth knowing before you treat any list as final. Research most often sorts crowdsourcing four ways by what the crowd does: creating content, solving problems, rating, and processing tasks [7]. Crowdfunding sits outside that scheme altogether, and it is also what a lot of people expect the word to cover, so it earns a slot here. That makes five categories, each named for what the crowd hands over: content (crowd creation), judgment (crowd voting), money (crowdfunding), answers to defined technical problems (crowdsolving), or small paid units of work (microtask crowdsourcing, the labor model). None of them names an industry, so one program can sit in two categories at once.

Crowd creation, crowd voting, and crowdfunding

Crowd creation asks a distributed group to produce content or creative work directly. Wikipedia is the clearest example: volunteer contributors write and edit articles with no central authoring staff. In the best-known cases the contribution is unpaid. That changes the quality mechanism completely: reputation and peer editing do the work that payment and screening do elsewhere. Crowd voting uses the group to rank or judge instead: contest entries, feature prioritization, "most popular" rankings on a platform, where the group's combined preference is the decision.

Crowdfunding pools a lot of small contributions into one project budget. Kickstarter is the best-known version: a creator posts a funding goal publicly and collects pledges that are only charged if the goal is met. Equity crowdfunding, where backers receive a stake in the venture, is a separate category under its own securities rules [8].

For an ML team the practical read on all three is that they rarely produce training data. They produce decisions, artifacts, and budget.

Crowdsolving and microtask (labor) crowdsourcing

Crowdsolving routes a defined technical, scientific, or analytical problem to a distributed pool of solvers, usually through a competition. Biomedical and algorithmic challenges work this way. Open AI and data-science competitions have formalized the pattern far enough that how a challenge gets designed and scored is now a research topic of its own [9]. The requester is buying an answer here, and pays for the winning one.

Microtask crowdsourcing breaks a larger job into small, well-defined, paid units distributed to many contributors in parallel. Amazon Mechanical Turk is the canonical example. "Requesters" post small paid tasks — tagging an image, drawing a bounding box, moderating a piece of content — and "workers" complete them for a per-task fee [10]. This is the type that matters most to anyone training a machine learning model, because both halves of an AI data pipeline break down into it: one photograph collected, or one box drawn on a photograph you already have. It is also where the quality question stops being theoretical, since every accepted microtask becomes a row in someone's training set.

Crowdsourcing in AI and machine learning

Machine learning models need training data at volumes most teams can't produce in-house: raw examples, plus the labels the model learns to reproduce. A crowd can be pointed at either half. It can produce the raw material — photographs, audio, video, survey responses — or it can label material that already exists.

For the labeling half, work at scale usually splits three ways: an in-house team, a managed outsourced vendor, or crowdsourcing. The survey literature treats a crowd as the default when non-experts are plentiful and expert time isn't, and frames the choice as accuracy against scale [11]. Price usually drives the decision anyway, which is why the unit matters. What a model consumes is the accepted item, and price per item is not price per accepted item.

Crowdsourced data collection for AI datasets

Collection is the half where a crowd does something a dedicated team physically cannot: produce data from thousands of separate places, devices, and lives at once. What comes back is raw material built to a spec. Some data types are out of reach entirely: LiDAR, the laser scanning that returns a 3D point cloud, needs capture hardware, so no crowd produces it.

We don't recruit that crowd ourselves. We go to whoever has already gathered one, and the channel decides who does the day-to-day work. On a microtask platform all of it sits with whoever posts the task: writing it, checking what comes back, quality control, acceptance. Partnering with the admin of an existing community hands distribution to them. Going through a vendor hands over distribution and the checking too, and leaves you defining the task and accepting the result. That is also where you lose direct contact with individual respondents.

One number governs the economics of all three channels. Across our collection projects, roughly half of what arrives from a crowd platform is unusable. Going back to respondents to correct it rescues about a tenth of that half, or roughly 5% of intake. The bulk of the effort isn't collection at all. It is selecting what can be kept from everything that arrives.

Crowdsourced data labeling and annotation

Labeling narrows the crowd's advantage. Annotation decomposes into microtasks as neatly as collection does: bounding boxes around objects, sentiment tags on a passage of text, a transcript of thirty seconds of audio, a moderation flag on a message. Volume still scales by adding contributors instead of by extending anyone's hours.

What doesn't scale is agreement. Thousands of contributors working from one set of instructions, with no shared training, disagree on ambiguous cases far more than a small team does. Research on noisy crowd labels shows what that costs downstream: models trained on unreviewed crowd annotations inherit the crowd's labeling errors unless a correction step is built into the pipeline [12]. A human-in-the-loop review layer closes the gap: a checkpoint where a person, not an algorithm, checks or fixes the output before it reaches the training set. Agreement is also worth measuring rather than assuming. Inter-annotator agreement is the standard instrument, Cohen's kappa for two annotators and Krippendorff's alpha for more, and a low figure indicts the instruction before it indicts the crowd.

Crowd annotation is technically possible; we don't conduct it. What rules it out is the ambiguous middle of a dataset: the cases where the instruction doesn't quite decide, which are also the cases a model most needs decided consistently. Thousands of people working without shared training resolve those inconsistently by construction, and no amount of overlap turns inconsistent judgment into a stable label. That is why annotation work here goes to trained teams with team leads instead. Specialist work widens the gap further: for example, city-street LiDAR point clouds go to engineers with CAD and robotics backgrounds, because classifying individual points depends on how the laser beam behaves and what blocks it.

Crowdsourcing platforms for AI training data

Amazon Mechanical Turk is the platform most AI teams have heard of first, and it is no longer a place to start. Amazon has stopped accepting new Mechanical Turk customers and tells existing ones to migrate to a third-party service [10]. New work goes to the other general-purpose marketplaces: Microworkers, running since 2009, describes itself as a crowdsourcing platform for micro jobs spanning tagging, annotation, categorization, moderation, transcription and survey work [13]. What separates one marketplace from another is rarely the interface. It is geographic coverage, the real number of active contributors behind the registered number, and how deeply the platform has profiled its own audience.

That last one reshaped the category. The early model was one big undifferentiated pool with tasks thrown into it, closed by whoever happened to be online. Platforms then started scoring their audiences, recording languages, professional skills and location, so a task can be aimed instead of announced. It matters as soon as a dataset carries sampling quotas — so many speakers per language, so many per age band — because that requirement is answerable only on a platform that already knows which of its contributors are which.

Contributors are paid by the platform, not by the requester, who pays the task price plus the platform's commission. One thing to settle before comparing two quotes: whether the number you are given is per submission or per accepted item. On a platform you pay per submission and absorb the rejects yourself; the same job quoted by a vendor should be per accepted item, with the rejects on their side of the line. 

On the platforms we run collection through, the task price is calculated rather than quoted. Price is also a schedule lever: the same task at lower rate fills slowly and at higher rate fills fast, so when a collection deadline moves, price is usually what moves with it.

Benefits and challenges of crowdsourcing

Crowdsourcing trades control for scale. You get low cost, speed, a wide contributor pool, and a workforce that flexes with demand. The bill arrives as uneven quality, bias creeping in through task wording or through who volunteers, and unsettled ownership when the terms get written after the work. Which side wins depends less on the crowd than on how much judgment each single item needs.

Key benefits for organizations

Cost leads most decisions, and the mechanism is simple: paying per accepted submission lowers the cost of each extra unit of work. Speed comes from working in parallel. Contributors work at the same time, so a job that would tie up one team for weeks can finish in days, and because the same infrastructure serves many requesters at once, the workforce grows and shrinks with demand without permanent headcount. A large open pool also covers more demographic and geographic variation than a small in-house team, which matters for how well a dataset represents the people it will be used on.

One clarification, because two ideas get merged here. Demographic breadth improves dataset representativeness. The wisdom-of-the-crowd effect [1] rests on something else: judgments made independently of one another. A pool can be demographically broad while every member reads the same ambiguous instruction the same wrong way.

Quality control, bias, and risk management

Quality control is the most cited failure point in the crowdsourcing literature. Two themes recur: inconsistent output from contributors who were never trained for the specific task, and bias entering through self-selection or task wording [14]. A third problem sits beside them: unclear intellectual property ownership when contributor agreements don't address it up front.

Two mechanics carry the first two, and both are worth understanding before you rely on them. Overlap sends the same item to several contributors and treats convergence as a signal. Three matching answers out of five is a common working threshold, though it is a choice rather than a standard, and it holds only as far as the assumption that the contributors erred independently. 

Gold standards mix expert-labeled items into the batch, measure how far a contributor drifts from them, and apply that figure to everything else they touched. That estimates quality without proving it, since a contributor can clear the gold items and still be wrong on the odd cases. Layered review is what turns either estimate into a decision, and on our projects that means annotator, then reviewer, then QA audit, with a separate quality control department revalidating the previous day's data every day.

Consent settles the use-rights question, and the gate sits before the task begins. A respondent cannot start work without explicitly confirming participation, and our respondents sign consent forms wherever the data calls for one: a selfie, a voice recording, any biometric attribute. Consent under GDPR and CCPA establishes lawful use of the recording. Ownership of the result is a separate clause in the contributor agreement, and informal crowd recruiting is where both go missing.

Real-world examples of crowdsourcing

Wikipedia, Waze and Mechanical Turk cover three of the five categories between them. What holds Wikipedia together at that scale is not the editing but the rulebook: public standards for sourcing and for settling disputes, enforced by the community on itself.

Waze applies the same open call to data. Drivers running the app send in real-time location and incident reports, and together they produce traffic and hazard information no single provider could collect alone. The literature calls this participatory sensing: everyday mobile devices formed into a network so that ordinary users gather and share local knowledge [15]. It is a form of crowd creation where the contribution is a sensor reading.

Mechanical Turk closes the loop back to the AI section, and ImageNet shows why it mattered. The challenge dataset behind a decade of advances in object recognition was labeled by Mechanical Turk workers checking candidate images against a category definition, with several people labeling each image independently and an image counting as positive only on a convincing majority [16]. The threshold was not a constant. It moved with how confusable the category was, roughly five workers to settle a Burmese cat and fewer for a cat, and the team wrote an algorithm to set it per category [16]. Worth remembering the next time a vendor quotes one overlap number for a whole dataset.

Conclusion 

In one sentence: crowdsourcing gets ideas, labor, or content from an open, self-selecting group instead of a named contractor. For an AI data pipeline it divides along a clear line. Collection is where a spread-out crowd reaches places and people at a scale nobody can hire. Labeling is where the same crowd goes up against a trained team and loses on quality. Price the option against in-house and managed-vendor sourcing before committing. This is a build-or-buy decision more than a crowdsourcing-or-nothing one.

Crowdsourcing fits when
• you need raw data produced across many places, devices, or demographics at once
• a careful non-specialist can complete each item correctly from written instructions and examples
• an error on a single item is cheap to find and cheap to correct

Reconsider when
• correct output depends on training the contributor doesn't have: clinical, legal, 3D geometry
• the modality needs capture hardware rather than a phone, as LiDAR does
• IP and consent terms aren't settled before the first submission arrives
• selection overhead would erase the per-item savings

Always plan for
• on collection projects, roughly half of what arrives being rejected, with the selection work budgeted alongside the collection rather than after it
• a review layer with its own metrics and its own owner
• examples and counter-examples inside the task itself, alongside the written specification

A quote should itemise
• intake volume and accepted volume as separate lines, with the acceptance criteria they are measured against
• selection and review labor, priced apart from collection or labeling
• how many recontact rounds are included before a rejected item is written off
• consent administration for any modality that needs it
• the platform's commission, where a platform sits in the chain