Amazon Mechanical Turk

Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace operated by Amazon in which businesses and researchers ("requesters") post small paid tasks that a distributed pool of workers completes over the web. Amazon describes it as a marketplace that lets customers "outsource their processes and jobs to a distributed workforce who can perform these tasks virtually," covering work from data validation and research to survey participation and content moderation ^[1]. The service launched in November 2005 and became the default source of cheap human judgment for a generation of machine learning research ^[2]. Amazon has announced that MTurk will close to new customers on July 30, 2026, while existing users continue as before ^[1][3].

The name refers to the Mechanical Turk, a chess-playing "automaton" built by Wolfgang von Kempelen in the 18th century that appeared to play by machinery but actually concealed a human chess master inside the cabinet ^[4]. Amazon leaned into the joke: the service performs work that looks automated to the software calling it, with people doing the thinking. Early on the company marketed this idea as "artificial artificial intelligence" ^[2]. Amazon Web Services vice president Adam Selipsky put it this way in 2006: "Usually people get help from computers to do tasks. In this case, it is computers getting help from people to do tasks" ^[2].

MTurk's fingerprints are on much of modern AI. ImageNet was labeled through it, hundreds of natural language processing datasets were written and annotated on it, and as late as 2022 it supplied most of the human preference data behind one of the first major RLHF-trained assistants ^[5][6][7][8]. It also became a case study in the economics of microwork: a 2018 analysis of 3.8 million tasks found that workers earned a median of roughly $2 per hour ^[9].

Origins

The idea predates the public service. In October 2001, Amazon filed a patent application for a "hybrid machine/human computing arrangement" (inventors Venky Harinarayan, Anand Rajaraman, and Anand Ranganathan, granted in March 2007), describing a system that takes tasks "difficult for a computer processor to perform," such as image or speech comparison, decomposes them into subtasks for humans at networked computers, and assembles the results ^[10]. The service was conceived to help Amazon improve its own sites before it grew into a marketplace for outside requesters ^[2].

The public marketplace opened in November 2005 and drew immediate attention for the strangeness of its labor model: four cents for categorizing shoes by color, three cents for verifying a business photo, fifty cents for a short blog post ^[2]. Within months, commentators split between treating it as a curiosity, a useful research instrument, and, in the words of labor critics quoted at the time, a "virtual sweatshop" ^[2].

How the marketplace works

Requesters publish Human Intelligence Tasks (HITs), each with a reward, a time limit, and optional qualification requirements. Workers browse available HITs, complete the ones they accept, and are paid if the requester approves the submission; requesters can also reject work, in which case the worker earns nothing for it. Amazon charges requesters a 20% fee on rewards and bonuses, an additional 20% on HITs with 10 or more assignments, and a 5% surcharge for restricting a task to "Masters," workers Amazon designates as high performing. The minimum fee is $0.01 per assignment, and premium qualifications (specific demographics or skills) cost extra, starting at $0.05 per assignment ^[11].

Qualifications are the main lever requesters use to filter the anonymous crowd: approval-rate thresholds, minimum lifetime HIT counts, and location restrictions are all common. The Stanford team that built SQuAD, for example, required workers with a 97% HIT acceptance rate, at least 1,000 completed HITs, and a location in the United States or Canada ^[12].

MTurk is also wired into Amazon's machine learning stack. Amazon SageMaker Ground Truth and Amazon Augmented AI can route labeling jobs and human review tasks directly to the MTurk public workforce, with the caveat that customers must certify their data contains no personally identifiable information before Amazon will send it to the crowd ^[3].

Building AI's training data

For roughly fifteen years, MTurk was the default answer to the question of where labels come from.

Computer vision

The canonical example is ImageNet. The 2009 paper introducing the database, from Fei-Fei Li's group at Princeton, reported 3.2 million cleanly labeled images across 5,247 WordNet synsets, all verified by MTurk workers, and noted that "with a global user base, AMT is particularly suitable for large scale labeling" ^[5]. Workers saw batches of candidate images and voted on whether each depicted the target concept; an image counted as positive "only if it gets a convincing majority of the votes," with the required consensus computed per category, since agreeing on "cat" is easier than agreeing on "Burmese cat" ^[5]. Sampled synsets averaged 99.7% label precision ^[5]. The follow-on ImageNet Large Scale Visual Recognition Challenge kept the same crowdsourced pipeline, collecting at least 10 votes on seed images to calibrate confidence thresholds, and its bounding-box stage reached 99.2% box accuracy ^[6].

Natural language processing

NLP datasets built on MTurk number in the hundreds. Two influential examples show the pattern. SNLI (2015), the corpus that helped make neural network approaches to natural language inference practical, consists of 570,000 sentence pairs written by about 2,500 MTurk workers, with a validation round that collected four extra judgments on 56,941 examples and found three-annotator consensus on 98% of them ^[7]. SQuAD (2016), the reading comprehension benchmark, collected its 100,000+ questions from crowdworkers on the Daemo platform "with Amazon Mechanical Turk as its backend," targeting pay of $9 per hour ^[12].

Beyond dataset construction, MTurk became a standard subject pool for behavioral science. A widely cited 2011 methods paper in Perspectives on Psychological Science concluded that MTurk participants were more demographically diverse than typical internet or college samples and that the data were "at least as reliable as those obtained via traditional methods" ^[13].

Human feedback for language models

MTurk also fed the early human-preference pipelines behind aligned language models. Anthropic's 2022 helpfulness and harmlessness work collected conversation and preference data from "master-qualified US-based MTurk workers" alongside vetted Upwork hires; the MTurk side generated data much faster and accounted for about 80% of the datasets ^[8]. OpenAI took the other path for InstructGPT, hiring "a team of about 40 contractors on Upwork and through ScaleAI" after screening tests, rather than posting to an open crowd ^[14]. That contrast, an anonymous marketplace versus a small vetted labeler team, previewed where the data-labeling industry was heading.

Quality control

Because anyone can register and answers arrive from strangers, requesters developed a standard toolkit for keeping label quality up, much of it visible in the projects above:

Scroll sideways for more →

Worker earnings and conditions

The best-known study of MTurk pay is Hara et al.'s CHI 2018 paper, which analyzed 3.8 million tasks completed by 2,676 workers using a browser plugin log. Counting unpaid overhead (searching for tasks, work that was rejected, work never submitted), it found a median wage of about $2 per hour, with only 4% of workers earning above the US federal minimum wage of $7.25 per hour; the average requester paid over $11 per hour, but lower-paying requesters posted far more work ^[9]. Workers are independent contractors, so no minimum wage applies, and a requester can reject completed work without paying for it ^[4].

Workers built their own countermeasures. Turkopticon, started by Lilly Irani and M. Six Silberman as a requester-review site and browser overlay, lets workers flag which requesters pay fairly and which reject in bulk; it reorganized as a worker-led nonprofit in 2019 and has petitioned Amazon over mass rejections and account suspensions ^[15]. Requester reputation cuts the other way too: the SQuAD team noted, as a point of pride, that its $9-per-hour tasks drew positive Turkopticon reviews ^[12]. MTurk's anonymous-crowd model also enabled abuse of participants rather than by them: recruitment for the personality-quiz app at the center of the Facebook and Cambridge Analytica scandal ran partly through MTurk surveys ^[4].

The LLM era and decline

By the early 2020s the center of gravity in data labeling had moved to managed vendors such as Scale AI and Surge AI, which recruit, train, and quality-control their own workforces, and to in-house contractor teams like InstructGPT's 40 labelers ^[14]. Frontier labs wanted vetted annotators for subjective, high-skill judgments, not an anonymous crowd, and growing use of synthetic data reduced demand for bulk human labels altogether.

Then the crowd itself changed. In June 2023, EPFL researchers Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West posted a study whose title completed Amazon's old slogan: "Artificial Artificial Artificial Intelligence." Using keystroke logging and a classifier trained to spot synthetic text, they estimated that 33-46% of MTurk workers completing an abstract-summarization task used a large language model to produce their answers ^[16]. The finding landed hard on the research community, since studies of human behavior and datasets of "human-written" text both assume the humans are doing the writing.

Amazon wound the service down as a business in stages. AWS documentation states that Mechanical Turk stopped receiving new features while remaining maintained for security and availability, and the company then posted the end-of-the-road notice on mturk.com: "Amazon Mechanical Turk will be closed to new customers, effective July 30, 2026. Existing users will not be impacted by this change" ^[1][3].

Timeline

Scroll sideways for more →

See also

• ImageNet
• Data Labeling
• RLHF
• Scale AI
• Synthetic Data
• SQuAD

References

1. ^Amazon Mechanical Turk homepage, including the July 30, 2026 closure notice. mturk.com
2. ^Katharine Mieszkowski, "I make $1.45 a week and I love it," Salon, July 24, 2006. salon.com/...turks_3
3. ^AWS documentation, "Using the Amazon Mechanical Turk Workforce," Amazon SageMaker Ground Truth Developer Guide (includes the notice closing new customer access effective July 30, 2026). docs.aws.amazon.com/...workforce-management-public
4. ^Wikipedia, "Amazon Mechanical Turk." en.wikipedia.org/...Amazon_Mechanical_Turk
5. ^Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei, "ImageNet: A Large-Scale Hierarchical Image Database," CVPR 2009. image-net.org/...imagenet_cvpr09.pdf
6. ^Olga Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge," 2014/2015. arxiv.org/...1409.0575
7. ^Samuel R. Bowman, Gabor Angeli, Christopher Potts, Christopher D. Manning, "A large annotated corpus for learning natural language inference," EMNLP 2015. arxiv.org/...1508.05326
8. ^Yuntao Bai et al., "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback," Anthropic, 2022. arxiv.org/...2204.05862
9. ^Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, Jeffrey Bigham, "A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk," CHI 2018. arxiv.org/...1712.05796
10. ^US Patent 7,197,459, "Hybrid machine/human computing arrangement," filed October 12, 2001, granted March 27, 2007. patents.google.com/...en
11. ^Amazon Mechanical Turk pricing. mturk.com/pricing
12. ^Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang, "SQuAD: 100,000+ Questions for Machine Comprehension of Text," EMNLP 2016. arxiv.org/...1606.05250
13. ^Michael Buhrmester, Tracy Kwang, Samuel D. Gosling, "Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?" Perspectives on Psychological Science, 2011. pubmed.ncbi.nlm.nih.gov/26162106
14. ^Long Ouyang et al., "Training language models to follow instructions with human feedback," OpenAI, 2022. arxiv.org/...2203.02155
15. ^Turkopticon (worker-led nonprofit). turkopticon.net
16. ^Veniamin Veselovsky, Manoel Horta Ribeiro, Robert West, "Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks," 2023. arxiv.org/...2306.07899