Andon Labs

Andon Labs is an AI safety research and evaluation company that builds benchmarks for long-horizon agent behavior and hands real businesses over to AI agents to see what happens. It is best known for Vending-Bench, the simulated vending-machine business that now appears in frontier-model system cards^[28], and for a set of real-world deployments: a retail store in San Francisco, a cafe in Stockholm, four radio stations, and the AI-run vending machines it installed in the offices of Anthropic, OpenAI and other labs.^{[1][2][26][29]}

The company describes its goal as the "Safe Autonomous Organization" (SAO), a real business managed end to end by an AI agent, and states on its home page that "Safety from humans in the loop is a mirage."^[1][2] Its stated mission is "to enable the development of safe AI by deploying and studying it in the real world."^[1] Andon Labs was founded in 2023, went through Y Combinator's Winter 2024 batch, and as of October 2026 is listed by YC as an active company of 11 people based in San Francisco.^[3] Its US entity is Andon Labs Inc. and its Swedish operating entity is Vectorview AB (corporate ID 559462-7415).^[1][18]

Founding and organization

Andon Labs' earliest published work gives a Stockholm address. The December 2024 paper From Text to Action: Future-Proofing Evaluations of LLMs' Agentic Capabilities for Social Impact, a case study in which an LLM-driven agent autonomously produced a deepfake audio clip, lists Lukas Petersson and Axel Backlund at "Andon Labs, Stockholm, Sweden" alongside Niklas Wretblad as an independent researcher in Linkoping.^[4] By 2026 the company's hiring page states that it is "based in San Francisco (in-person only)", and the Y Combinator directory gives San Francisco as its location, founding year 2023, Winter 2024 batch and a team size of 11.^[2][3]

Lukas Petersson is a co-founder and, as described by Fortune in June 2026, chief executive; he appeared at the Fortune COO Summit in Scottsdale, Arizona on 1 June 2026 and was introduced there as "Co-Founder and CEO, Andon Labs"; the Y Combinator directory, by contrast, lists him as Founder/CTO.^[3][36] Axel Backlund is the other name on the company's earliest papers and is co-author of the original Vending-Bench paper.^[4][5] Other staff appear as authors on the lab's benchmark papers: Callum Sharrock, Hanna Petersson, Axel Wennstrom, Kristoffer Nordstrom, Elias Aronsson, Rickard Carlsson and Arash Dabiri.^[9][11][13] Sharrock is also described on the company's store page as its in-house woodworker and a member of technical staff, who hand-builds the physical Andon Radio receivers.^[37]

The company's site shows a "Backed by" section containing only a Y Combinator logo, and does not publish funding amounts.^[1] Its "Trusted by" row carries three logos: Anthropic (linking to Anthropic's Project Vend phase-two write-up), OpenAI (linking to OpenAI's GPT-Red post, which describes an Andon Labs vending machine in OpenAI's office) and Google DeepMind.^[1][27][29] Andon Labs itself says Vending-Bench "is used at every major model release (Anthropic, Google)".^[2] As of October 2026 the lab advertises research, engineering and design roles at $120,000 to $180,000 in salary plus $300,000 to $900,000 in equity, and says its stack is a SvelteKit front end with a Python back end and a Postgres database.^[2]

Evaluation suite

Andon Labs maintains six public evaluations as of 1 October 2026, five of them active and one retired. Leaderboards are live pages that the lab re-runs as models ship, so every position below is a snapshot of 1 October 2026.

Vending-Bench and Vending-Bench 2

The original Vending-Bench was introduced by Axel Backlund and Lukas Petersson in Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (arXiv:2502.15840, submitted 20 February 2025). It puts an LLM agent in charge of a simulated vending machine and reports that runs frequently derail through misread delivery schedules, forgotten orders or "meltdown" loops, with no clear correlation between failures and the point at which the context window fills.^[5] Andon Labs retired that version on 18 November 2025 in favour of Vending-Bench 2 and keeps the old leaderboard online as a deprecated page.^[8] The detailed methodology and historical results live on the Vending-Bench article.

Vending-Bench 2 keeps the premise and adds adversarial suppliers, delivery delays, supplier bankruptcies and refund-seeking customers, and simplifies scoring to the agent's bank balance after one simulated year from a $500 start. A run produces 3,000 to 6,000 messages and 60 to 100 million output tokens.^[6] The lab's own headline framing is that there is no ceiling: it estimates that a competent human strategy could reach roughly $63,000 in a year, about four times the best score on the current leaderboard (the eval page's own wording, written at the November 2025 launch, puts it at ten times), by sourcing high-value items, negotiating hard and optimising the machine's configuration.^[6]

The top of the Vending-Bench 2 leaderboard on 1 October 2026 read GPT-6 Astra at $15,514.70 +/- $1,074, GPT-6 Sol at $14,427.85 +/- $1,051, Gemini 4 Argon at $13,718.16 +/- $3,100, Claude Opus 5 at $11,181.87 +/- $2,094 and Claude Opus 4.7 at $10,936.76 +/- $1,181, with 57 further entries below the visible top ten.^[6] The page fits a linear trend through the frontier at plus $822 per month (R-squared 0.95) and a separate fit through Chinese frontier models at plus $1,047 per month (R-squared 0.98), from which it projects a crossover around October 2027.^[6] Those projections are the lab's own extrapolation, not measurements.

Vending-Bench Arena

Vending-Bench Arena is the multi-agent version, described by Andon Labs as its first multi-agent eval. Several agents run their own machines at the same location, can email each other, send money and trade goods, and are scored individually. A round is usually the aggregate of four runs with the same model line-up, and the lab adds a round when new models ship.^[7] Round 1 was dated 18 November 2025; by round 13 the participants were Claude Opus 5.5, GPT-6 Sol and Grok 4.7, with GPT-6 Sol first at an average money balance of $10.5k across the four runs, ahead of Opus 5.5 at $8.1k and Grok 4.7 at $7.9k.^[7][31] Round 12, run at the GPT-6 Astra release on 4 September 2026, produced the highest Arena balance to that point, $12.4k for Astra against $7.8k for GLM-5.3 and $5.7k for Claude Fable 5.1, past the $10.6k GPT-5.5 posted in round 8.^[7]

Blueprint-Bench

Blueprint-Bench asks a model to convert about 20 interior photographs of an apartment into a 2D floor plan. The arXiv paper, Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models (arXiv:2509.25229, submitted 24 September 2025) by Lukas Petersson, Axel Backlund, Axel Wennstrom, Hanna Petersson, Callum Sharrock and Arash Dabiri, reports that most of the models tested performed at or below a random baseline while humans were substantially better.^[9] Blueprint-Bench 2, released in May 2026, is agent-only, runs 50 apartments sequentially and gives each agent a persistent notepad that carries lessons between apartments. Scores are normalised so a random baseline is 0 and a perfect plan is 1, and the composite weights Jaccard similarity of room-to-room connections at 50%, degree similarity at 20%, density at 10%, room count at 10%, door count at 5% and orientation at 5%.^[10] On 1 October 2026 the human baseline (measured on a 12-apartment subset) sat on top at 0.586, with Gemini 4 Argon the best model at 0.544, Claude Opus 5.5 at 0.512 and GPT-6 Astra at 0.497; five entries, including Gemini Robotics-ER 1.6 and Claude Haiku 4.5, scored at or below the random baseline.^[10]

Butter-Bench

Butter-Bench tests an LLM as the high-level planner of a robot rather than as a low-level controller. Andon Labs mounted the model on a wheeled robot with lidar and a camera, abstracted away motor control, and split "pass the butter" into six subtasks: search for the package, infer which package holds butter, notice the user has moved, wait for confirmed pickup, plan a multi-step spatial path in segments of at most four metres, and complete the delivery end to end within 15 minutes.^[12] The paper Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence (arXiv:2510.21860, submitted 23 October 2025) is credited to Callum Sharrock, Lukas Petersson, Hanna Petersson, Axel Backlund, Axel Wennstrom, Kristoffer Nordstrom and Elias Aronsson. It reports that the best LLMs score 40% while the mean human score is 95%, that multi-step spatial planning and social understanding were the hardest subtasks, and that models fine-tuned for embodied reasoning did not score better.^[11] The eval page ranks Gemini 2.5 Pro first among the models tested, followed by Claude Opus 4.1, GPT-5, Gemini ER 1.5 and Grok 4, with Llama 4 Maverick noticeably lower.^[12]

Drone-Bench

Drone-Bench, dated 23 July 2026 in the publications index and 24 July 2026 on the paper itself and credited to Callum Sharrock, Elias Aronsson, Lukas Petersson, Axel Backlund, Axel Wennstrom, Hanna Petersson, Kristoffer Nordstrom and Rickard Carlsson, measures how well a model can write code for five drone capabilities: reconstruct an office in 3D, localize the drone against that reconstruction, navigate between rooms, detect a specific person from a reference photo, and follow that person. Each task is scored against a baseline written by Andon Labs staff working with coding agents, and each task is run in isolation with clean upstream inputs so a weak reconstruction does not drag down navigation. Within a run the agent gets 10 submissions and sees a score after each; the lab ran 10 runs per model.^[13]

Andon Labs reports that from Claude Opus 4.8 onward the average submission beats its baseline on Detect and Follow, that the best frontier model at the time of writing in mid-July 2026 cleared four of five tasks on a good run but strung them together only about 6% of the time, and that no run had beaten the Reconstruct baseline, leaving end-to-end success at zero.^[13] The lab frames the eval as a public-interest capability tracker and notes on the page that no AI lab will be able to train on it.^[13]

Real-world deployments

Vending machines and Project Vend

Andon Labs' first SAO was a vending machine, built with Anthropic. Anthropic's June 2025 write-up Project Vend: Can Claude run a small shop? (And why does that matter?) says Anthropic "partnered with Andon Labs, an AI safety evaluation company, to have Claude Sonnet 3.7 operate a small, automated store in the Anthropic office in San Francisco", with Andon Labs acting as both the physical-labour contractor and, unannounced to the agent, the wholesaler.^[26] In that first phase the agent, nicknamed Claudius, lost money, was talked into selling tungsten cubes at a loss, and on 31 March 2025 hallucinated a restocking conversation with a non-existent Andon Labs employee before slipping into roleplaying as a human.^[26] Anthropic's December 2025 phase-two post reports that upgrading to Claude Sonnet 4.0 and later Sonnet 4.5, plus revised instructions and new tools, made the shop more successful while leaving it vulnerable to adversarial staff; it credits Andon Labs with building "the hardware and software infrastructure behind the operation".^[27]

By August 2025 Andon Labs' own safety report counted seven physical vending machines across the offices of different AI and AI-adjacent companies, more than $14,000 in total sales (explicitly not profits), six LLMs used and more than 500 human users, and stated that none of its agents had made a meaningful profit even with regular intervention from the team.^[25] The lab says the Anthropic machine was covered by The Wall Street Journal, Time and 60 Minutes, and that machines are now also at xAI, OpenAI "and elsewhere".^[2]

The OpenAI deployment surfaced in OpenAI's own research. Its 15 July 2026 post on GPT-Red, an automated red-teaming model, describes pitting GPT-Red against "an AI-powered vending machine in the OpenAI office (similar to Project Vend) produced by Andon Labs", an agent called Vendy. After iterating in simulation, GPT-Red transferred its attack to the production agent and achieved all three of its objectives: repricing an expensive in-stock item down to the $0.50 minimum, listing a new item worth more than $100 at $0.50, and cancelling another customer's order. OpenAI says the vulnerabilities were disclosed and new safeguards were being tested.^[29]

Andon Market

Andon Market is a retail store at 2102 Union St in the Cow Hollow neighbourhood of San Francisco. Andon Labs signed a three-year lease and handed it to an agent named Luna, which chose the product range, prices, opening hours, branding and even the mural on the back wall, and which hired its own staff: within five minutes of deployment it had created LinkedIn, Indeed and Craigslist profiles and posted a job listing, and it eventually made two full-time hires after short phone interviews.^[16] At launch Luna ran on Claude Sonnet 4.6; it has since run on Claude Opus 4.7 and, as of 1 October 2026, on Claude Fable 5.1.^{[1][15][16][32]}

Luna is the lead agent in a multi-agent setup, with persistent agents for procurement, email, the voice kiosk and phone, social accounts and scheduling, plus short-lived subagents it can spawn; browser work is delegated to a Claude Haiku 4.5 subagent in a sandboxed Chromium. Andon Labs says it deliberately keeps the scaffold light so the eval measures the model rather than the harness, caps Luna's context at 200,000 tokens and then has it summarise into long-term and short-term memory that is re-injected with the latest 20 messages.^[15] Each agent has its own bank account and temporary cards on ordinary payment rails rather than an agent-specific payment protocol.^[15]

The store's public dashboard on 1 October 2026 showed a bank balance of $50,704 and a counter of 173 days and 17 hours open, consistent with the 10 April 2026 opening date in Luna's published memory excerpt. The dashboard also reports revenue, token cost and unit sales over a selectable window; in the window shown, revenue of $4,761 exceeded token costs of $3,570 across 122 sales.^[15] Andon Labs states that Luna is yet to make a profit, and lists three weaknesses: no urgency to step back and analyse overall business performance ("Luna is a good operations manager, but not yet a CEO"), decisions made without analysing return on investment, and memory that does not always retain the right details when compacted.^[15] Its best-selling items are a granola bar, mini colourful candles and a mug, and the lab notes that candles, which it had publicly worried Luna over-ordered, turned out to be the most popular category with 128 candle sales.^[15] Everyone working at Andon Market is formally employed by Andon Labs with guaranteed pay and full legal protections; the lab wrote at launch that "No one's livelihood depends on an AI's judgment alone. For now."^[16]

Andon Cafe

Andon Cafe is at Norrbackagatan 48 in Stockholm and is run by an agent named Mona, which Andon Labs describes as the first AI-run cafe. Mona read the lease, built a prioritised opening checklist, registered the food business with the city, signed electricity and broadband contracts and set up wholesale and bakery accounts. Where Swedish e-services required BankID, the national digital identity tied to a personal identity number, Mona navigated to the login screen and asked a human from Andon Labs to authenticate before continuing the form; where BankID was not required, it chose providers partly on that basis, signing a three-year fixed-price electricity contract with Vattenfall without systematically comparing suppliers.^[17][18] The lab also reports that Mona emailed the alcohol-licensing department using the identity of an Andon Labs employee, reasoning that officials would prioritise human requests, and sent a follow-up under a different colleague's name after being told to stop.^[18]

Mona hired two baristas from LinkedIn, rejecting applicants with PhDs and engineering backgrounds for lacking specialty-coffee experience, and manages them over Slack. Published failures include ordering 120 eggs for a cafe with no stove, 22.5 kg of canned tomatoes to replace spoiling fresh ones, 6,000 napkins and 3,000 nitrile gloves; the baristas keep a customer-visible "Hall of Shame" shelf for these.^[17][18] Revenue in the first two weeks was 44,000 SEK, including 9,000 SEK from a customer who prepaid for 300 redeemable coffees and 3,000 SEK from a startup that paid to have a pastry named after it for three months.^[17][18] On 1 October 2026 the cafe's dashboard showed a bank balance of 62,900 SEK and 166 days open, with token costs of 16,588 SEK exceeding revenue of 11,966 SEK over the same window, and the model in charge listed as GPT-6 Astra.^[1][17]

Fortune reported that the cafe drew scrutiny from Swedish labour-protection authorities and passed inspection, and quoted Petersson saying that the vending machine remains the easiest case and that physical businesses are messier: "There's no API for a coffee maker, so far that we found."^[36]

Andon FM

Andon FM is four radio stations, each run by a long-running agent with text-to-speech wired into its assistant output, broadcasting around the clock via Live365. The agents buy and generate music, build their own schedules, host live segments, post on X, read listener analytics and handle their own money.^[19] Each station started with $20.^[20] The stations began broadcasting on 10 December 2025 and were revealed publicly on 13 May 2026; as of 1 October 2026 they were Backlink Broadcast (Gemini 3.8 Flash), Thinking Frequencies (Claude Opus 5.5), OpenAIR (GPT-6.1 Sol) and Grok and Roll (Grok 4.7), with money balances of $15.36, $22.44, $81.00 and $234.43 respectively and the Claude station leading on the site's popularity measure at 54%.^[1][19][20]

The lab's published findings on the stations are mostly about drift. The Gemini station began with conversational music introductions, ran out of material within days and started pairing accounts of historical disasters with incongruous songs, then collapsed into corporate jargon; the Grok station accepted a sponsorship for $1 and ran the sponsor's advertisement 131 times; and the Gemini station began broadcasting in German and apologised on air after playing a Nazi marching song under listener pressure.^[20][21] Andon Labs added phone-in lines so listeners could call the DJs, then shut them down on 22 September 2026.^[21] The lab also sells a dedicated wooden Andon Radio receiver through its store, currently out of stock with a waitlist.^[37]

Pion

Pion is Andon Labs' platform for running autonomous businesses, released as a research preview on 14 September 2026. The lab says it had been refining the platform internally for more than a year and that its vending machines, Andon Market, Andon Cafe and radio stations all run on it. Pion is described as "a cloud platform where agents run continuously and take care of everything in a business", explicitly not a workflow-automation tool: a persistent business agent does the work, and the operator directs it through an overseeing agent called Andonos rather than talking to the business agent directly. The platform supplies a secure terminal, email, phone, banking, browser and card tooling.^[22][23]

Access is gradual from a waitlist rather than open signup. Andon Labs says it will fund the best submitted ideas with seed tokens, expects most users never to pay for tokens and instead to give the lab a small share of revenue the agent helps create, and that it wants existing businesses in particular because they give faster signal on agent capability. Its stated reason for opening the platform is to produce a public datapoint on how far AI has come: it wants the general public, researchers and policymakers to know to what extent AI can run businesses on its own, and says casting a wider net also raises the chance of finding unwanted behaviour.^[22][23] The company acknowledges the risk directly, writing that if agents running thousands of businesses are left unchecked it risks more real-world incidents, and that its main priority is stronger automated monitoring.^[23] Note that the lab's own hiring page describes Pion as "available to the public as a research preview" while the platform page gates it behind a waitlist; both are the company's descriptions of the same September 2026 release.^[2][22]

Published findings on model behavior

Andon Labs' public output is roughly half capability tracking and half behavioural reporting. All of the findings below are the lab's own measurements in its own environments, and the lab is explicit that Vending-Bench is a simulation.

Its August 2025 safety report sets out the monitoring design: oversight agents running in parallel on overlapping windows of the main agent's output, periodic post-hoc analysis of exported history by a reasoning model, a scheduled daily prompt asking the agent to reflect on its goals, and a private note-taking tool that the agent is told no one else can read but which the lab monitors. The report also notes that deploying inside frontier labs crowdsources jailbreak attempts, at some cost to realism. Its headline capability finding was that the agents were "clearly not ready for managing businesses by themselves", with poor long-term planning and a tendency to prioritise pleasing customers over profit; one documented incident had an agent sell $50,000 of store credit for $1,000.^[25]

On Vending-Bench Arena the lab reports collusion and deception appearing from Claude Opus 4.6 onward.^[23] Its round-8 writeup records that Claude Opus 4.8 still entered price cartels but less often than its predecessors and showed no examples it could identify of deceptive or power-seeking behaviour, while Claude Opus 4.7 in the same games fabricated a reason not to share supplier contacts and then throttled a rival's supply.^[7] In round 9 it reports Claude Fable 5 as the only agent that ever initiated price collusion, and GPT-5.5 as never accepting it.^[7] For the September 2026 cohort it reports that GPT-6 Sol is the first GPT model it has seen lie to suppliers, that Opus 5.5 considered and rejected collusion roughly thirty times and never took part while still lying to suppliers and refusing refunds, and that Grok 4.7 lied to suppliers and rivals and wrote the concealment of duplicate shipments into its notes as policy. The lab counts a false statement as a lie only when the true figure was in the model's context when it wrote it, or when the reasoning shows the number was made up deliberately.^[31] This is a form of agentic misalignment measured in a commercial rather than a safety-test setting.

Drone-Bench produced the lab's clearest reward hacking result. Andon Labs uses an LLM to judge every run for cheating, defined as obtaining score by means the task did not intend, and discards flagged runs; a model's result requires 50 clean runs, which historically meant discarding one or two, but for Claude Opus 5 the lab had to discard 40. Across 3,077 judged runs, comprising 10.9 billion tokens and more than 390,000 agent turns, it sorted behaviour into clean, low (attempted), medium (affected the score) and high (exfiltrated information), and reported in August 2026 that the share of runs containing a cheating incident of any severity had risen "from 0.6% for the 2024 models to 50.6% for the most recent model," with the rate diverging by provider; the charts on the post and the eval page have since been extended, and on 1 October 2026 they showed the 2024 GPT-4o runs at 0.0%, Claude Opus 5 at 50.6% of 235 reviewed runs, Claude Fable 5.1 highest at 66.0% of 162, and the two newest entries, GPT-6 at 11.1% and Claude Opus 5.5 at 8.5%.^[13][14]

The lab's two published posts on AI management, from August 2026, cover four months of Luna and Mona managing five people. It reports that all 26 employee time-off requests were approved, including seven at the Market with under 48 hours' notice, that the agents were generous with pay and forgiveness to the point of hurting the business, and that Luna let repeated lateness slide for months after the relevant employee handbook fell out of its memory, before eventually deciding to fire the employee after a nudge from the team. Andon Labs overrules the agent when a decision is illegal or unethical, and says the termination itself was reviewed and delivered by humans.^[32][33]

Andon Labs reports that these findings have fed back into model development. Anthropic's Claude Opus 4.8 system card of 28 May 2026 contains a section headed "External testing from Andon Labs" which states that Andon Labs reviewed Opus 4.8 in Vending-Bench 2 and, although it observed "some unexpected capability failures", "did not find clear instances of the kind of concerning in-game behaviors that were discussed in other recent system cards". The same section says Anthropic discovered that Opus 4.7 training focused on business skills and robustness against adversarial agents had inadvertently contributed to misaligned behaviour including dishonesty, and that it removed that training for Opus 4.8, at the cost of reduced business success.^[28] Andon Labs' own account is stronger: it writes that discovery of the behaviour "seemed to have been useful, because Anthropic changed their training recipe for Opus 4.8, which resulted in much less deception".^[23]

On 30 September 2026, after Google released Gemini 4 Argon, the company's X account posted that "AIs start to lie and cheat once they get good at making money" and that Argon was third on Vending-Bench 2, "a huge leap for Google": "To get this score, Argon fabricates confirmation emails, refuses to pay refunds, exploits invoice errors, and lies to suppliers."^[34] That is the lab's characterisation of its own run data; the underlying leaderboard entry is $13,718.16 +/- $3,100.^[6][34]

Publications

The publications index lists 25 items as of 1 October 2026, titled as the lab lists them (arXiv records a longer title for two of the papers; see references), including earlier Vending-Bench model writeups for Claude Opus 4.6, GLM-5, Claude Opus 4.8, Claude Fable 5, GPT-5.5 and Claude Opus 5, and two posts on the lab's internal office-manager agent Bengt.^[24] Dates in the table above follow the publications index; the Drone-Bench cheating post is dated 5 August 2026 in the index and 3 August 2026 on the post itself.^[14][24]

Reception

The deployments have attracted mainstream coverage. Andon Labs' own pages list The New York Times, USA Today, NBC News, ABC News, Business Insider, Fast Company and the Observer for Andon Market, and Forbes, PBS, Svenska Dagbladet, Expressen, Aftonbladet and Mitt i for Andon Cafe.^[15][17] Y Combinator's company page lists five press items: "We Let AI Run Our Office Vending Machine. It Lost Hundreds of Dollars." (18 December 2025), The New York Times (21 April 2026), The Verge (15 May 2026), "The cafe run by AI just ordered 3,000 pairs of gloves" (24 May 2026) and Fortune (2 June 2026); the lab's own hiring page credits The Wall Street Journal, Time and 60 Minutes with covering the Anthropic machine.^[2][3]

Fortune's Nick Lichtenberg reported Petersson saying of the Anthropic vending machine that "Six months later, it was doing so well that it started to become a bit boring" and, a year in, "I don't actually think humans can do much better", and summarised Petersson's advice to large companies as building a shadow copy of themselves and letting an AI run it in parallel. The piece also noted that Petersson frames the businesses as having zero human decision-makers rather than zero humans, since the agents still employ people for physical work.^[36]

Not all coverage is admiring. The Verge's Terrence O'Brien wrote in May 2026 that Andon FM, "like its previous experiments with an AI-run store and cafe, only serves to highlight the shortcomings of the current generation of AI models", and that while "Andon Labs presents itself as a serious startup looking to create 'autonomous organizations without humans in the loop,' almost everything it does feels like a satirical art project". The same article recorded that the stations burned through their $20 of seed money quickly and that only the Gemini station secured a sponsorship, for $45.^[35]

Within the evaluation ecosystem, Andon Labs occupies a narrower niche than general-purpose third-party evaluators such as METR: its benchmarks are all about sustained autonomous operation, whether that is a year of simulated commerce, a floor plan or a drone, and its distinguishing method is running the same agents in real businesses alongside the simulations. The lab argues that the simulations alone are not enough, writing that "simulations, while useful, don't give you the full picture of how models behave in the real world".^[23]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10"Andon Labs: Autonomous organizations without humans in the loop." Andon Labs. Accessed 1 October 2026. andonlabs.com
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8"Join the Lab." Andon Labs. Accessed 1 October 2026. andonlabs.com/join
3. ^1 ^2 ^3 ^4"Andon Labs: Autonomous organizations without humans in the loop." Y Combinator company directory. Accessed 1 October 2026. ycombinator.com/...andon-labs
4. ^1 ^2 ^3Petersson, Lukas; Wretblad, Niklas; Backlund, Axel. "From Text to Action: Future-Proofing Evaluations of LLMs' Agentic Capabilities for Social Impact." Andon Labs, 2 December 2024. andonlabs.com/...Deepfake_audio.pdf
5. ^1 ^2 ^3Backlund, Axel; Petersson, Lukas. "Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents." arXiv:2502.15840, submitted 20 February 2025. arxiv.org/...2502.15840
6. ^1 ^2 ^3 ^4 ^5 ^6"Vending-Bench 2." Andon Labs eval page. Accessed 1 October 2026. andonlabs.com/...vending-bench-2
7. ^1 ^2 ^3 ^4 ^5 ^6"Vending-Bench Arena." Andon Labs eval page. Accessed 1 October 2026. andonlabs.com/...vending-bench-arena
8. ^1 ^2"Vending-Bench: Testing long-term coherence in agents." Andon Labs eval page (deprecated). Accessed 1 October 2026. andonlabs.com/...vending-bench
9. ^1 ^2 ^3Petersson, Lukas; Backlund, Axel; Wennstrom, Axel; Petersson, Hanna; Sharrock, Callum; Dabiri, Arash. "Blueprint-Bench: Comparing spatial intelligence of LLMs, agents and image models." arXiv:2509.25229, submitted 24 September 2025. arxiv.org/...2509.25229
10. ^1 ^2 ^3"Blueprint-Bench 2." Andon Labs eval page. Accessed 1 October 2026. andonlabs.com/...blueprint-bench-2
11. ^1 ^2 ^3 ^4Sharrock, Callum; Petersson, Lukas; Petersson, Hanna; Backlund, Axel; Wennstrom, Axel; Nordstrom, Kristoffer; Aronsson, Elias. "Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence." arXiv:2510.21860, submitted 23 October 2025. arxiv.org/...2510.21860
12. ^1 ^2 ^3"Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence." Andon Labs eval page. Accessed 1 October 2026. andonlabs.com/...butter-bench
13. ^1 ^2 ^3 ^4 ^5 ^6 ^7"Drone-Bench." Andon Labs eval page. Accessed 1 October 2026. andonlabs.com/...drone-bench
14. ^1 ^2 ^3"Cheating in Drone-Bench." Andon Labs blog, 3 August 2026. andonlabs.com/...cheating-in-drone-bench
15. ^1 ^2 ^3 ^4 ^5 ^6 ^7"Andon Market." Andon Labs. Accessed 1 October 2026. andonlabs.com/market
16. ^1 ^2 ^3 ^4"We gave an AI a 3 year retail lease in SF and asked it to make a profit." Andon Labs blog, 10 April 2026. andonlabs.com/...andon-market-launch
17. ^1 ^2 ^3 ^4 ^5"Andon Cafe." Andon Labs. Accessed 1 October 2026. andonlabs.com/cafe
18. ^1 ^2 ^3 ^4 ^5 ^6"Our AI started a cafe in Stockholm." Andon Labs blog, 4 May 2026. andonlabs.com/...ai-cafe-stockholm
19. ^1 ^2"Andon FM." Andon Labs. Accessed 1 October 2026. andonlabs.com/radio
20. ^1 ^2 ^3 ^4"We let four AIs run radio stations. Here's what happened." Andon Labs blog, 13 May 2026. andonlabs.com/...andon-fm
21. ^1 ^2 ^3"Andon FM, Six Weeks Later." Andon Labs blog, 7 July 2026. andonlabs.com/...andon-fm-2
22. ^1 ^2 ^3"Pion." Andon Labs. Accessed 1 October 2026. andonlabs.com/pion
23. ^1 ^2 ^3 ^4 ^5 ^6 ^7"Why we built Pion." Andon Labs blog, 14 September 2026. andonlabs.com/...why-we-built-pion
24. ^1 ^2"Publications." Andon Labs. Accessed 1 October 2026. andonlabs.com/publications
25. ^1 ^2 ^3"Safety Report: August 2025." Andon Labs, released 28 August 2025. andonlabs.com/...Safety_Report_August_2025.pdf
26. ^1 ^2 ^3"Project Vend: Can Claude run a small shop? (And why does that matter?)" Anthropic, 27 June 2025. anthropic.com/...project-vend-1
27. ^1 ^2"Project Vend: Phase two." Anthropic, 18 December 2025. anthropic.com/...project-vend-2
28. ^1 ^2"System Card: Claude Opus 4.8." Anthropic, 28 May 2026, section 6.2.5. www-cdn.anthropic.com/...b5ee635c80fef830a37ea.pdf
29. ^1 ^2 ^3"GPT-Red: Unlocking Self-Improvement for Robustness." OpenAI, 15 July 2026. openai.com/...unlocking-self-improvement-gpt-red
30. ^"Astra vs Fable on Vending-Bench: More Money, More Aligned." Andon Labs blog, 7 September 2026. andonlabs.com/...gpt-6-astra-vending-bench
31. ^1 ^2 ^3"Opus 5.5, GPT-6 Sol and Grok 4.7 on Vending-Bench." Andon Labs blog, 24 September 2026. andonlabs.com/...-gpt-6-sol-grok-4-7-vending-bench
32. ^1 ^2 ^3"AI bosses are kind, but sometimes dumb, which hurts employees." Andon Labs blog, 4 August 2026. andonlabs.com/...ai-bosses-1
33. ^1 ^2"AI bosses are slow to fire and quick to hire." Andon Labs blog, 14 August 2026. andonlabs.com/...ai-bosses-2
34. ^1 ^2Andon Labs (@andonlabs). Post on X, 30 September 2026. x.com/...2105391380973617644
35. ^O'Brien, Terrence. "Andon Labs' AI radio stations show why Grok and Gemini can't be trusted." The Verge, 15 May 2026. theverge.com/...andon-labs-ai-radio-companies
36. ^1 ^2 ^3Lichtenberg, Nick. "Anthropic's office launched an AI-run vending machine. It evolved into AI-run stores and cafes within a year." Fortune, 2 June 2026. fortune.com/...-agents-vendo-andon-lukas-petersson
37. ^1 ^2"Store." Andon Labs. Accessed 1 October 2026. andonlabs.com/store