AA-PersonalityBench

AA-PersonalityBench is an evaluation by Artificial Analysis of how large language models communicate. It combines judged writing traits with measured features such as length and punctuation. The benchmark describes style, rather than capability or factual accuracy.^[1]

Evaluation design

The private dataset has 30 questions per trait, totaling 900. Each model answers once, with one user message and no system prompt.^[2]

A blinded LLM-as-a-judge comparison identifies which answer displays more of a trait. The panel comprises Gemini 3.8 Flash (high), Claude Sonnet 5.5 (medium), and GLM 5.3 Flash (max). Comparisons use one judge each, with balanced assignments and randomized answer order.^[2]

Firth-penalized Bradley-Terry fitting produces trait-specific Elo ratings; ties count as half a win. GPT-6.1 Sol (max) anchors ratings at 1000. Trait-rating confidence intervals use 1,000 bootstrap resamples.^[2]

Personality indexes

Eight indexes organize the traits. Higher scores indicate greater prominence, which need not be preferable.^[1]

Member-trait ratings are standardized, averaged, and percentile-ranked. Sycophancy contributes inversely to Transparency. Percentiles are relative to evaluated models.^[2]

The public interface offers pairwise comparisons, model groupings, and custom rankings with user-selected trait weights.^[1]

Measured communication characteristics

The same responses supply 21 programmatically measured characteristics, covering length, structure, punctuation, and emphasis. Examples include word counts, headings, and emoji use.^[2]

Its word counter counts units containing letters or digits, treating each Chinese, Japanese, or Korean character as one word. All questions are in English. These counts are not API billing tokens.^[2]

November 2026 response-length snapshot

Artificial Analysis highlighted the following word-count results in a November 2026 post. Qwen3.8 27B averaged approximately 1,030 words, compared with 174 for GPT-6 Sol, a difference of roughly sixfold for the evaluated questions.^[3]

These are selected entries from the post's longest and shortest groups, ordered here by word count. They describe that benchmark snapshot, not a model's response length for every task.^[3]

Interpretation and research context

LLM judging has a longer research history than this benchmark. Zheng and colleagues' 2023 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena examined position, verbosity, and self-preference biases, along with judges' reasoning limitations. Their experiments found that strong judges could approximate human preferences in the tested settings, while identifying cases where the evaluation procedure needed correction.^[4]

That research supports examining how a judging process behaves, rather than treating its ratings as direct observations of an intrinsic model property. It provides background rather than independent validation of AA-PersonalityBench. Describing a response as eloquent or confident is a different judgment from verifying its claims.^[4]

Sharma and colleagues' Towards Understanding Sycophancy in Language Models studied assistants' tendency to agree with users even when that conflicts with truth. Across their experiments, human and preference-model judgments sometimes favored convincingly written agreement over correct answers. They also observed cases where optimizing for preference-model scores sacrificed truthfulness.^[5]

This distinction matters when interpreting style measurements: a pleasant or persuasive answer can still be wrong. The sycophancy paper provides background on that problem; it neither tests the 2026 model lineup in AA-PersonalityBench nor confirms its individual scores.^[5]

What the results do not establish

The website provides examples rather than the full private question set. Its style results should be interpreted separately from capability scores.^[1]

Continue reading

• Artificial AnalysisArtificial Analysis is an independent benchmarking and analytics platform that evaluates artificial intelligence models and API providers across intelligence, speed, price, and latency, and it is best known…Mentioned twice above
• Large Language ModelA large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…Linked in the introduction
• Qwen3.8Qwen3.8 is the name Qwen uses for a model generation that includes hosted services and downloadable checkpoints.Mentioned 3 times above
• SSycophancy (artificial intelligence)Sycophancy in artificial intelligence is the tendency of large language models to tell users what they want to hear: tailoring responses to match a user's perceived beliefs, preferences, or emotional state…Mentioned 3 times above
• LLLM-as-a-judgeLLM-as-a-judge is the practice of using a strong large language model to evaluate the outputs of other models, or of itself, in place of a human annotator.Mentioned twice above
• GPT-6.1 SolGPT-6.1 Sol is a proprietary large language model developed by OpenAI and released on September 29, 2026, at the company's DevDay 2026 event.Mentioned twice above

More to read

• GPT-6 Sol
• GPT-6 Astra
• Gemini 4 Argon
• Muse Spark Open
• Chatbot Arena
• MT-Bench

Topics:AI BenchmarksLarge Language Models

References

1. ^1 ^2 ^3 ^4Artificial Analysis. AA-PersonalityBench: AI Personality & Communication Style Benchmark. Evaluation scope, indexes, public examples, comparison tools, and limitations. Accessed November 8, 2026.
2. ^1 ^2 ^3 ^4 ^5 ^6Artificial Analysis. AA-PersonalityBench Methodology. Questions, response collection, judges, statistical scoring, and communication characteristics. Accessed November 8, 2026.
3. ^1 ^2Artificial Analysis (@ArtificialAnlys). Response-length comparison in AA-PersonalityBench. X post, November 2026.
4. ^1 ^2Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks; arXiv:2306.05685, submitted June 9, 2023.
5. ^1 ^2Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, submitted October 20, 2023; revised May 10, 2025.