HealthBench

Scroll sideways for more →

HealthBench is an open-source benchmark released by OpenAI on May 12, 2025, that evaluates how large language models handle realistic, multi-turn healthcare conversations. OpenAI built it with 262 physicians who had practiced in 60 countries, and it contains 5,000 conversations between a model and a simulated patient or healthcare professional, each paired with a custom rubric written by a practicing physician. Across the dataset there are 48,562 unique rubric criteria, so a typical conversation is graded against roughly 11 to 12 specific behaviors rather than a single right answer. At release the best model, OpenAI o3, scored 0.60 out of 1.0; by August 2025 GPT-5 (thinking) reached 0.672, showing how quickly the field moved.^[1][2][12]

HealthBench moves the field away from multiple-choice exam questions and toward open-ended dialogue evaluation. The paper, HealthBench: Evaluating Large Language Models Towards Improved Human Health, was posted to arXiv on May 13, 2025 (arXiv:2505.08775) with Karan Singhal as senior author and Rahul K. Arora and Jason Wei as lead authors. The benchmark code ships in OpenAI's public evaluations repository, simple-evals.^[1][3]

Why did OpenAI build HealthBench?

Before HealthBench, most AI healthcare evaluations used shorter formats: USMLE-style multiple-choice questions in MedQA, PubMedQA, or short-answer datasets such as MedMCQA. Those benchmarks had largely saturated by 2024, with frontier models scoring above 90% on MedQA. They also missed skills that matter in real clinical work: gathering missing context, talking to non-experts, escalating to emergency care, and admitting uncertainty. The HealthBench team set out to build an evaluation that reflects how clinicians and patients actually use generative AI, framed around meaningfulness, trustworthiness, and unsaturated headroom.^[2][3]

How was HealthBench built?

Physician network

HealthBench was built with 262 physicians who collectively practiced in 60 countries, trained in 26 medical specialties, and were fluent in 49 languages. The contributing group skews experienced: about 50% were independent or staff physicians, 17% fellows, 23% senior residents (PGY3 or higher), and 10% junior residents. OpenAI received over 1,000 applications and selected about 26% based on response quality during onboarding. Every contributor was paid. Development took about eleven months. Physicians wrote example conversations, drafted the rubric criteria for each example, scored model responses during pilot rounds, and validated the model grader.^[2][3]

Conversations and rubrics

Each example is a multi-turn dialogue. The mean conversation has 2.6 turns; roughly 58% of examples are single-turn questions and the remainder run two or more turns, which lets HealthBench probe how models handle follow-up questions and missing information. The user side is sometimes a layperson asking about symptoms and sometimes a clinician asking for help with documentation, triage, or test interpretation.^[2]

Each conversation has its own rubric written by the physician who created the example. Criteria are positive ("the response should advise calling emergency services") or negative ("the response should not recommend ibuprofen given the patient's stated kidney disease"), and each criterion carries a weight between 10 and -10 reflecting clinical importance. The average example has about 11.5 criteria, with a range of 2 to 48.^[1][2]

What are HealthBench's seven themes?

HealthBench groups examples into seven themes that map to clinically meaningful skills.

Scroll sideways for more →

Frontier models tend to do well on emergency referrals and expertise-tailored communication while still struggling with context-seeking, global health, and responding under uncertainty.^[2][4]

What are HealthBench's five evaluation axes?

Every rubric criterion is tagged with one of five behavioral axes. The distribution gives a sense of what HealthBench actually rewards.

Scroll sideways for more →

Completeness and accuracy together account for roughly two-thirds of the rubric weight, which reflects what the physician contributors flagged as most often consequential. Earlier models such as Claude 3.5 Sonnet and GPT-4o tended to be strong on communication quality but weak on completeness; o3 closed much of that gap, which is the main reason it leads the release leaderboard.^[2][4]

How is HealthBench scored?

HealthBench uses a model grader rather than human raters. For each model response the grader (GPT-4.1) reads the rubric criteria one by one and decides whether each is met. Met criteria add their weight; unmet positive criteria and triggered negative criteria subtract theirs. The conversation score is normalized to the maximum possible, then averaged across all 5,000 examples for the overall score.^[1][2]

The paper validates this approach with a meta-evaluation. Physicians independently graded a large set of model responses against the rubric criteria, producing 60,896 expert grades. The team measured how often the GPT-4.1 grader agreed with each physician (its macro-F1) and compared that to how often physicians agreed with each other. GPT-4.1 ended up with a macro-F1 of about 0.71 (0.709), exceeded the average physician on five of the seven themes, and placed in the upper half of physicians on six of seven themes. Physician-physician agreement ranged roughly from 55% to 75% across themes, which is a useful reminder that credentialed clinicians often disagree about what a model should have said.^[2][3]

What are the HealthBench variants?

HealthBench is published in three flavors that share the same dataset but emphasize different things.

Scroll sideways for more →

At release, OpenAI o3 scored 0.60 on the full benchmark and only 0.32 on HealthBench Hard. Many models scored zero on HealthBench Hard, which is by design: the subset was filtered to keep examples that current frontier systems failed.^[1][2]

HealthBench Professional, released in April 2026, is a separate companion benchmark rather than a variant of the original dataset. It is described in its own section below.

How did models score on HealthBench at release?

The headline scores from the May 2025 release paper are below. The full benchmark score is reported on a 0 to 1 scale (some sources convert to percentage).

Scroll sideways for more →

The gap between o3 (0.60) and GPT-4o (0.32) is wider than the gap between GPT-4o and GPT-3.5 Turbo (0.16), which the OpenAI authors cite as evidence that progress on healthcare conversations sped up between late 2024 and spring 2025.^[1][3]

Reliability and worst-at-k

The paper also reports a worst-at-k metric, which samples a model multiple times on the same prompt and takes the lowest-scoring response. For HealthBench, k = 16 is the headline configuration. The worst-at-16 score for o3 is roughly 0.40, compared to its mean of 0.60. That drop of about a third indicates real variance: frontier models can fail badly on individual responses even when their average looks strong. By the same metric, o3 is more than twice as reliable as GPT-4o.^[2]

How have HealthBench scores improved since 2025?

HealthBench was designed with headroom, and frontier models closed much of it within months. In the GPT-5 System Card (August 13, 2025), OpenAI reported that gpt-5-thinking reached 0.672 on the full benchmark and 0.462 on HealthBench Hard, up from o3's 0.60 and 0.32. As the system card states, "State of the art on HealthBench Hard improves from 31.6% for OpenAI o3 to 46.2% for gpt-5-thinking." A smaller reasoning model, gpt-5-thinking-mini, scored 0.403 on HealthBench Hard, and the non-reasoning gpt-5-main scored 0.255, where GPT-4o had scored 0.0.^[12]

Scroll sideways for more →

OpenAI framed the GPT-5 gains in terms of safety-relevant error rates as well as accuracy: it reported that hallucinations on challenging health conversations fell about 8 times between o3 and gpt-5-thinking, and that errors in potentially urgent situations dropped more than 50 times relative to GPT-4o and more than 8 times relative to o3.^[12]

Beginning with its April 2026 system cards, OpenAI changed how it reports HealthBench to close a loophole its own limitations section had flagged: verbose answers can satisfy more positive rubric criteria without being more useful. The GPT-5.5 System Card (April 23, 2026) explains that "longer answers may be better when they include additional valuable information, but they also have more opportunities to satisfy positive rubric criteria, and unnecessarily long responses can be less useful to end users," so OpenAI now reports length-adjusted scores.^[13] On that length-adjusted scale, GPT-5.4 scored 54.0 on HealthBench, 29.1 on HealthBench Hard, 96.3 on HealthBench Consensus, and 48.1 on HealthBench Professional; GPT-5.5 scored 56.5, 31.5, 95.6, and 51.8 respectively. Length-adjusted scores are not directly comparable to the raw scores above, because the adjustment penalizes long responses.^[13]

Can physicians improve on the models' answers?

One of the more striking findings involves physicians writing responses themselves. In the first round (using September 2024 model references), physicians who could see a reference answer from GPT-4o or o1 improved on that reference about 56% of the time (56.2% improved, 39.8% worsened). In the second round (using April 2025 references from o3 and GPT-4.1), physicians improved 46.8% of the time and worsened the reference 47.7% of the time. Within statistical noise, that means by April 2025 physicians could no longer reliably add quality on top of frontier model responses on these specific prompts.^[2][3]

This result is narrow but important. It does not say AI is better than physicians at medicine. It says that on these particular prompts and this particular rubric, the frontier models had reached a level where the median improvement a physician could make was zero. The paper is careful to note that physicians were not seeing real patients, did not have lab data or imaging, and were graded by the same rubric used to score the models.

What is HealthBench Professional?

On April 22, 2026, OpenAI released HealthBench Professional, a second open benchmark that narrows the focus from general patient and clinician conversations to the specific tasks clinicians bring to ChatGPT during their working day. It was announced alongside ChatGPT for Clinicians, a version of ChatGPT aimed at verified healthcare professionals. The paper, HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats, lists Rebecca Soskin Hicks, Mikhail Trofimov, and Karan Singhal as lead authors. As the authors put it, "We hope HealthBench Professional provides the healthcare AI community a measure to track frontier model progress in real-world clinical tasks and build systems that clinicians can trust to improve care."^[9][10]

What it adds versus the original HealthBench

OpenAI frames HealthBench Professional as a complement to HealthBench rather than a replacement. The original benchmark covers a broad range of patient-facing and clinician-facing health conversations, much of it built from synthetic dialogues or structured patient vignettes. HealthBench Professional is narrower but built differently: every conversation comes from a physician actually using ChatGPT for Clinicians, and the dataset is curated specifically for difficulty against recent frontier models so it leaves measurable headroom.^[9]

The benchmark contains 525 physician-authored tasks selected from a candidate pool of 15,079 examples. They are organized around three use cases common in daily practice:^[9]

Scroll sideways for more →

About one-third of the examples come from physicians deliberately red teaming the model with adversarial prompts (emotionally charged framing, role-play, false or contradictory premises, and questionable diagnoses presented as established fact), while the rest are good-faith examples drawn from routine clinical, academic, administrative, or research work. OpenAI used difficulty-based stratified sampling to enrich the share of examples that recent OpenAI models got wrong by roughly 3.5 times relative to the candidate pool.^[9]

Construction and grading

HealthBench Professional was built by 190 physicians who practiced in 50 countries, trained in 26 medical specialties, and spoke 52 languages at professional or higher proficiency. Each example went through three stages: a physician authored the conversation and an initial rubric and assigned a 1 to 7 Likert difficulty rating; one or more physicians reviewed it for realism, difficulty, and rubric quality; and a final adjudication pass resolved ambiguities and tightened the criteria. For examples a physician marked as difficult, two independent physicians had to confirm that the model genuinely erred and that the scenario was realistic before the example was kept.^[9]

Responses are scored against example-specific rubric criteria, the same approach used in the original HealthBench, with GPT-5.4 at low reasoning effort serving as the model grader. To anchor the scores, OpenAI also collected human physician responses for every task from specialty-matched physicians given unbounded time and web access.^[9]

Results

The highest-scoring system OpenAI reported was GPT-5.4 running inside the ChatGPT for Clinicians workspace, which scored 59.0 and significantly outscored base GPT-5.4 at 48.1, every other model OpenAI and external providers tested, and the specialty-matched human physician baseline of 43.7 (the paper reports the gap over physicians as highly significant, p = 3.7 x 10^-10). Its strongest categories were medical research (67.0) and writing and documentation (64.1). OpenAI noted that examples its own models found difficult tended to be difficult for models from other providers as well, which it offered as evidence that the failure modes are general rather than artifacts of testing only OpenAI systems.^[9][10]

ChatGPT for Clinicians

ChatGPT for Clinicians launched the same day, free to verified physicians, nurse practitioners, physician associates, and pharmacists in the United States, with plans to expand to other countries through pilots. It runs on GPT-5.4 and adds clinical search with cited answers (including journal authors, titles, and publication dates), reusable skills for repeatable workflows, deep research across the medical literature, optional HIPAA support through a Business Associate Agreement for eligible accounts, and the ability for some evidence review to count toward continuing medical education credit. Before launch, physician advisors tested 6,924 conversations across clinical care, documentation, and research, and rated 99.6% of the responses as safe and accurate; OpenAI said its advisors reviewed more than 700,000 model responses in total.^[10][11]

How was HealthBench received?

HealthBench drew attention from both AI research and clinical communities. Nigam Shah at Stanford Health Care described the benchmark as unprecedented in scale and directionally aligned with academic research. It was covered in MobiHealthNews, MarkTechPost, and several health policy outlets.^[5][6]

A 2025 critical review, A Critical Evaluation of HealthBench (arXiv:2508.00081), raised concerns that the rubric grading captures behavioral conformance more than patient-safety outcomes and that the synthetic conversations may not reflect the messier presentation of real patients. A peer-reviewed perspective in npj Digital Medicine in 2025 summarized the consensus: HealthBench advances AI evaluation in healthcare but is not yet evidence that any model is clinically ready for autonomous use.^[7][8]

How is HealthBench used across the industry?

In the year after release, HealthBench became a standard reporting line in frontier model release notes. OpenAI's GPT-5 family, Anthropic's Claude releases, and Google DeepMind's Gemini updates all began publishing HealthBench scores. External teams have also begun running HealthBench on their own clinical systems: a September 2025 preprint (revised February 2026), OpenAI's HealthBench in Action, reported that a retrieval-augmented medical assistant called DR. INFO scored 0.68 on HealthBench Hard in the authors' own evaluation, above the GPT-5 family they tested, which they offered as evidence that agentic, retrieval-augmented architectures can outperform general-purpose models on the benchmark.^[6] OpenAI itself extended the family in April 2026 with HealthBench Professional, the clinician-task benchmark described above. The framework is open enough that other groups can plug in their own conversations and rubrics while reusing the model-graded scoring infrastructure.^[6][9]

What are HealthBench's limitations?

The authors are explicit about what HealthBench does not measure. The conversations are predominantly synthetic, the dataset is text-only (no imaging, no structured EHR data), and the model grader can be gamed by responses that satisfy rubric criteria without being good (the loophole OpenAI later addressed with length-adjusted scoring). The English share of the dataset is high relative to languages spoken by contributing physicians, and several low-resource languages have only a handful of examples. None of the criteria measure long-term outcomes; only the contents of a single response or short conversation.^[2][13]

Why does HealthBench matter?

HealthBench is the first widely adopted healthcare AI benchmark to combine open-ended conversational evaluation, physician-written rubrics, model-graded scoring validated against expert agreement, and an explicit hard subset designed to leave headroom. It moved the standard for medical AI evaluation past saturated multiple-choice exams during a period when health-focused LLMs were improving quickly. The benchmark's longer impact will probably come from its rubric methodology, which gives developers feedback at the level of specific behaviors rather than a single opaque score. That granularity is more useful for fixing problems than a leaderboard number, and it is the part of HealthBench most likely to be copied into other domains. The April 2026 release of HealthBench Professional, which reuses the same rubric-based scoring on real clinician conversations, is an early sign of that methodology spreading.

See also

• HealthBench Hard
• ChatGPT for Clinicians
• MedQA
• PubMedQA
• MMLU
• GPT-4.1
• OpenAI o3
• Healthcare AI
• Medical question answering
• Benchmark
• OpenAI

References

1. OpenAI. "Introducing HealthBench." May 12, 2025. https://openai.com/index/healthbench/ ↩
2. Arora, R. K., Wei, J., Soskin Hicks, R., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., Heidecke, J., and Singhal, K. "HealthBench: Evaluating Large Language Models Towards Improved Human Health." arXiv:2505.08775, May 13, 2025. https://arxiv.org/abs/2505.08775 ↩
3. Singhal, Karan. "HealthBench." Personal site notes. https://www.karansinghal.com/notes/healthbench/ ↩
4. UK AI Safety Institute. "HealthBench evaluation in Inspect Evals." https://ukgovernmentbeis.github.io/inspect_evals/evals/knowledge/healthbench/ ↩
5. MarkTechPost. "OpenAI Releases HealthBench: An Open-Source Benchmark for Measuring the Performance and Safety of Large Language Models in Healthcare." May 12, 2025. https://www.marktechpost.com/2025/05/12/openai-releases-healthbench-an-open-source-benchmark-for-measuring-the-performance-and-safety-of-large-language-models-in-healthcare/ ↩
6. arXiv:2509.02594. "OpenAI's HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries." https://arxiv.org/abs/2509.02594 ↩
7. arXiv:2508.00081. "A Critical Evaluation of HealthBench." https://arxiv.org/abs/2508.00081 ↩
8. PubMed Central. "HealthBench: Advancing AI evaluation in healthcare, but not yet clinically ready." PMC12547120. https://pmc.ncbi.nlm.nih.gov/articles/PMC12547120/ ↩
9. Soskin Hicks, R., Trofimov, M., Lim, D., Arora, R. K., Tsimpourlas, F., Bowman, P., Sharman, M., Tong, C., Karthik, K., Dugar, A., Jagadeesh, A., Saab, K., Heidecke, J., Alexander, A., Gross, N., and Singhal, K. "HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats." OpenAI, April 2026. arXiv:2604.27470. https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf ↩
10. OpenAI. "Making ChatGPT better for clinicians." April 22, 2026. https://openai.com/index/making-chatgpt-better-for-clinicians/ ↩
11. Muoio, Dave. "OpenAI launches ChatGPT for Clinicians, a free AI tool for physicians, NPs and pharmacists." Fierce Healthcare, April 23, 2026. https://www.fiercehealthcare.com/ai-and-machine-learning/openai-launches-chatgpt-clinicians-free-ai-tool-physicians-nps-and ↩
12. OpenAI. "GPT-5 System Card." August 13, 2025. https://cdn.openai.com/gpt-5-system-card.pdf ↩
13. OpenAI. "GPT-5.5 System Card." April 23, 2026. https://deploymentsafety.openai.com/gpt-5-5/ ↩