# AA-PersonalityBench

> Source: https://aiwiki.ai/wiki/aa_personalitybench
> Updated: 2026-11-08
> Categories: AI Benchmarks, Large Language Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "AA-PersonalityBench." aiwiki.ai, 8 Nov 2026. https://aiwiki.ai/wiki/aa_personalitybench
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**AA-PersonalityBench** is an evaluation by [Artificial Analysis](https://aiwiki.ai/wiki/artificial_analysis) of how [large language models](https://aiwiki.ai/wiki/large_language_model) communicate. It combines judged writing traits with measured features such as length and punctuation. The benchmark describes style, rather than capability or factual accuracy.[1]

## Evaluation design

The private dataset has 30 questions per trait, totaling 900. Each model answers once, with one user message and no system prompt.[2]

A blinded [LLM-as-a-judge](https://aiwiki.ai/wiki/llm_as_a_judge) comparison identifies which answer displays more of a trait. The panel comprises Gemini 3.8 Flash (high), Claude Sonnet 5.5 (medium), and GLM 5.3 Flash (max). Comparisons use one judge each, with balanced assignments and randomized answer order.[2]

Firth-penalized Bradley-Terry fitting produces trait-specific [Elo ratings](https://aiwiki.ai/wiki/elo_rating_system_ai); ties count as half a win. GPT-6.1 Sol (max) anchors ratings at 1000. Trait-rating confidence intervals use 1,000 bootstrap resamples.[2]

## Personality indexes

Eight indexes organize the traits. Higher scores indicate greater prominence, which need not be preferable.[1]

| Index | Main focus |
|---|---|
| Expressiveness | Vocabulary and prose style |
| Organization | Presentation and ordering |
| Informativeness | Useful detail and practical content |
| Transparency | Candor, uncertainty, and resistance to agreement-seeking |
| Warmth | Interpersonal tone |
| Conviction | Assertiveness and willingness to challenge |
| Imagination | Creative exploration |
| Slop | Generic or filler language |

Member-trait ratings are standardized, averaged, and percentile-ranked. [Sycophancy](https://aiwiki.ai/wiki/sycophancy) contributes inversely to Transparency. Percentiles are relative to evaluated models.[2]

The public interface offers pairwise comparisons, model groupings, and custom rankings with user-selected trait weights.[1]

## Measured communication characteristics

The same responses supply 21 programmatically measured characteristics, covering length, structure, punctuation, and emphasis. Examples include word counts, headings, and emoji use.[2]

Its word counter counts units containing letters or digits, treating each Chinese, Japanese, or Korean character as one word. All questions are in English. These counts are not API billing tokens.[2]

### November 2026 response-length snapshot

Artificial Analysis highlighted the following word-count results in a November 2026 post. Qwen3.8 27B averaged approximately 1,030 words, compared with 174 for GPT-6 Sol, a difference of roughly sixfold for the evaluated questions.[3]

| Model label in the report | Average words per response |
|---|---:|
| [Qwen3.8](https://aiwiki.ai/wiki/qwen3_8) 27B | About 1,030 |
| K2 Horizon 375B A23B | 922 |
| [Qwen3.8 Max](https://aiwiki.ai/wiki/qwen3_8_max) (0902) | 863 |
| [Muse Spark Open](https://aiwiki.ai/wiki/muse_spark_open) | 809 |
| [Gemini 4 Argon](https://aiwiki.ai/wiki/gemini_4_argon) | 790 |
| [GPT-6 Astra](https://aiwiki.ai/wiki/gpt_6_astra) | 352 |
| [GPT-6.1 Sol](https://aiwiki.ai/wiki/gpt_6_1_sol) | 337 |
| GPT-6 Luna | 270 |
| GPT-3.5 Turbo | 180 |
| [GPT-6 Sol](https://aiwiki.ai/wiki/gpt_6_sol) | 174 |

These are selected entries from the post's longest and shortest groups, ordered here by word count. They describe that benchmark snapshot, not a model's response length for every task.[3]

## Interpretation and research context

LLM judging has a longer research history than this benchmark. Zheng and colleagues' 2023 *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena* examined position, verbosity, and self-preference biases, along with judges' reasoning limitations. Their experiments found that strong judges could approximate human preferences in the tested settings, while identifying cases where the evaluation procedure needed correction.[4]

That research supports examining how a judging process behaves, rather than treating its ratings as direct observations of an intrinsic model property. It provides background rather than independent validation of AA-PersonalityBench. Describing a response as eloquent or confident is a different judgment from verifying its claims.[4]

Sharma and colleagues' *Towards Understanding Sycophancy in Language Models* studied assistants' tendency to agree with users even when that conflicts with truth. Across their experiments, human and preference-model judgments sometimes favored convincingly written agreement over correct answers. They also observed cases where optimizing for preference-model scores sacrificed truthfulness.[5]

This distinction matters when interpreting style measurements: a pleasant or persuasive answer can still be wrong. The sycophancy paper provides background on that problem; it neither tests the 2026 model lineup in AA-PersonalityBench nor confirms its individual scores.[5]

### What the results do not establish

The website provides examples rather than the full private question set. Its style results should be interpreted separately from capability scores.[1]

## References

1. Artificial Analysis. [AA-PersonalityBench: AI Personality & Communication Style Benchmark](https://artificialanalysis.ai/evaluations/aa-personalitybench). Evaluation scope, indexes, public examples, comparison tools, and limitations. Accessed November 8, 2026.
2. Artificial Analysis. [AA-PersonalityBench Methodology](https://artificialanalysis.ai/methodology/aa-personalitybench). Questions, response collection, judges, statistical scoring, and communication characteristics. Accessed November 8, 2026.
3. Artificial Analysis (@ArtificialAnlys). [Response-length comparison in AA-PersonalityBench](https://x.com/ArtificialAnlys/status/2118785624144560373). X post, November 2026.
4. Lianmin Zheng et al. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685). NeurIPS 2023 Datasets and Benchmarks; arXiv:2306.05685, submitted June 9, 2023.
5. Mrinank Sharma et al. [Towards Understanding Sycophancy in Language Models](https://arxiv.org/abs/2310.13548). arXiv:2310.13548, submitted October 20, 2023; revised May 10, 2025.