Chinese LLM Benchmark: What Changed After More Testing

The first version of this benchmark tried to answer a simple question: which language model produced the strongest graded Chinese draft under one fixed setup? More experiments made that question less useful. The answer changes with the HSK level, whether the model is planning or drafting, the prompt, the story concept, and the review method.

This page now preserves the research as a dated sequence instead of presenting one permanent scoreboard. It covers model outputs used in drafting experiments, not authorship. HSKStory's editors develop, revise, and approve the stories that readers see. The full editorial and production process is described in How I Make Graded Chinese Stories.

What This Benchmark Can and Cannot Tell You

Each result on this page belongs to a particular experiment. A score is useful only with its date, HSK level, sample size, prompt and model role, reviewer protocol, rubric, and known limitations.

Within one controlled screen, the scores can rank the arms tested. They cannot establish that the same ranking will hold at another HSK level or for another genre. Raw scores from different rubrics are not compared here. The March six-criterion score, the July planning score, and the July story-craft score measure different things.

Vocabulary diagnostics are narrower still. They can show how often above-level words occur or how many distinct above-level words appear. They cannot tell us whether a plot works or whether the Chinese sounds natural. Manual reading also remains necessary: in the July HSK 7 prose screen, every unedited model output needed changes.

Round 1: Seven Models, One Historical Setup

The original round ran in March 2026. Seven model routes attempted a multi-chapter story task at HSK 1, 3, 5, and 7. The archived table reports scores for six routes at all four levels. The seventh, Kimi K2.5, appears in the failure notes because planning text leaked into its output; it has no comparable row in the score table.

The condensed archive proves that writing quality was scored from one to five on six criteria, for 30 points total. It does not retain the criterion names or a per-call manifest that establishes reviewer identity, blind status, or reviewer count.

Under that setup, Kimi K2 had the highest average and DeepSeek V3.2 was close behind. Qwen 3.5 Plus followed the vocabulary constraint tightly but received the lowest prose score. The table contains 24 model-level scores, but the condensed archive is not a per-call sample receipt. It is not a general ranking of Chinese language models or a score that can be compared with the July rubrics.

The round also produced useful failure observations. Kimi K2.5 leaked English planning text, while separate GLM-4.7-Flash and Step-3.5-Flash trials leaked analysis or fell into repetition. Those failures matter for those exact model versions and roles. They do not support the broader claim that reasoning models cannot help with fiction.

The Metric Correction: Types Are Not Running Tokens

The benchmark history used several vocabulary percentages, and the old article did not keep their denominators clear enough.

The current validator distinguishes three measures:

• Running-token error rate is above-level word occurrences / all running-word occurrences. If an above-level word appears five times, all five appearances count.
• Token coverage is 1 - running-token error rate. It reports the share of running-word occurrences that are within level.
• Unique-type error rate is distinct above-level words / all distinct words. A word used five times counts once.

These measures answer different questions. Running tokens approximate how often a reader meets unfamiliar vocabulary. Unique types describe the breadth of the story's above-level vocabulary. Neither measures prose quality.

Where an old summary preserved a percentage but not its denominator, this page treats it only as a historical number from that setup. It does not relabel it or compare it with later unique-type or token-coverage results.

What the Vocabulary-Control Round Added

On April 4, 2026, E9 and E9c tested nine model routes on one deliberately hard HSK 1 cooking concept. Each route received the prompt in use at the time and a 1,000-character target. This was one output per route, so it was a screen, not a replicated model ranking.

The concept required words such as 锅, 炒, 厨房, 盐, and 油, all above HSK 1. Every route therefore had a high unique-type error rate. GLM also ignored the requested concept, StepFun leaked English, and some outputs missed the requested structure. Those defects are part of the result, not details to average away.

A follow-up on April 10 measured run-to-run variance on a different HSK 1 school concept. DeepSeek V3.2 and Qwen 3.5 Plus each ran five times. Their standard deviations were 4.6 and 4.9 percentage points, with 12.4- and 12.8-point ranges. The 3.3-point difference between their means was not significant. That made single-run prompt and model differences much harder to trust.

Later HSK 1 tests also corrected the vocabulary-list conclusion. The original setup implemented a broad negative constraint by injecting the full cumulative HSK allowlist and requiring the draft to stay inside it. At HSK 7, that list was about 50,000 characters, or roughly 25,000 tokens. A later full 300-word allowlist also failed to improve the result. Neither test showed that every vocabulary list is harmful. On April 30, a small positive list of 25 target words reduced average unique-type error from 29.2% to 26.2% across five concepts with DeepSeek V4 Pro, though some concepts got worse and outputs became about 50% longer. The conclusion needs to stay narrow: the broad negative constraint failed, while small positive target lists can help in a specific low-level setup.

Why the Original Reviewer Was Removed

A related single-reviewer system later served as an automated production selector. Claude Sonnet reviewed two draft candidates, and the pipeline selected the lowest vocabulary error among candidates that cleared its threshold.

In May 2026, a controlled HSK 1 test ran that selector on five concepts. Both candidates always cleared the review threshold, so the review changed the best-of-two choice zero times out of five. The selector was removed from production because it did not affect the decision in that test.

That removed selector is not the same system as the later research review. The production selector was one automated reviewer inside a selection rule. The July research used two independent reviewers from different model families, anonymous packets, scores frozen before identity reveal, integrity checks, and later manual editorial reads. It was designed to compare experimental arms, not to approve stories for publication.

Later Blind Reviews Changed the Model Picture

The July HSK 7 work separated planning from prose.

First, nine configurations each produced a beat sheet and solution outline for the same vampire-mystery fixture. That Phase 0 screen made 18 calls. Two cross-family reviewers scored anonymous planning packets on six criteria, with the scores frozen before model identities were revealed. GPT-5.6 with medium reasoning was the only configuration both reviewers placed in their top two, and both gave it 27/30. A second-sample consistency draw then confirmed gpt_medium and fable_adaptive as the two planning finalists. For that fixture, GPT-5.6 was the dependable planner. The screen did not test downstream prose.

The later prose screen held that winning plan, the full-solution projection, the draft prompt, and diagnostics fixed. Seven model routes produced 13 HSK 7 samples, including one reused Kimi incumbent sample. Two cross-family reviewers blind-scored each story on a nine-criterion, 45-point rubric.

GPT-5.6 won the blind craft scoring, but the manual read changed the interpretation of that win. It was strongest on coherent, fair-play plot execution, while its language register was too low for the HSK 7 brief. Fable 5 had the strongest prose and register, but only one sample, a large reviewer split, weaker plot coherence, and the highest cost in the screen.

No raw output from any route was ready to publish as-is. Every sample needed editorial changes for register, continuity, reveal structure, or language control. Reasoning behavior was therefore model- and role-specific: GPT-5.6 was a dependable planner and blind craft winner in this setup, but it did not solve advanced HSK 7 prose by itself.

Outline Architecture: More Detail Helps Until It Does Not

The July A/B/C screen tested one HSK 7 planning question while holding the master plan and Kimi K2.6 draft route fixed. Each arm produced one five-chapter story:

• A received a brief with cast, setting, and chapter summaries.
• B received the full causal solution outline, including chronology, rules, clues, explanations, and ending.
• C received B plus a per-chapter state card.

All 15 chapter calls passed. Two cross-family reviewers scored the anonymous stories on the same nine-criterion story rubric and froze their reviews before the reveal.

Both reviewers ranked B above C above A. The causal solution outline gave the draft route enough information to build a coherent investigation. The extra state card improved some objective measures, including length control and top-tier vocabulary share, but made the resolution more expository. It did not prevent more continuity errors than B in this one run.

This result also shows why the old 22/30 finding was not a universal ceiling. The old and new experiments used different HSK levels, model roles, prompts, rubrics, and reviewer protocols. Their raw scores cannot be compared. The later screen supports a specific conclusion: for this HSK 7 mystery and draft route, a causal solution outline beat a brief, while the extra state card hurt craft.

Current Routing, Checked July 29, 2026

The current production configuration routes early-draft generation by level:

This table is a dated routing snapshot, not a claim that one route is the best Chinese creative-writing model. Planning can use a different tool from early-draft generation, and editors remain responsible for the story. Cost, provider reliability, length control, vocabulary, plot coherence, and register can all change the route.

For the complete division of work between editors and the production system, see How I Make Graded Chinese Stories.

Findings That Held Up — and Findings I Retracted

What held up is that model choice cannot be separated from job, level, concept, prompt, and evaluation method. The dates are part of the evidence.

Historical Tables and Experiment Limits

The table most responsible for the old ceiling claim came from a March 2026 architecture test. Kimi K2 handled the same HSK 5 story concept in each arm. The archive records one 30-point score per arm but does not retain a per-call manifest that establishes reviewer protocol or replication.

These approaches did not beat the baseline in that experiment. Calling 22 a permanent ceiling went beyond the evidence.

The main limits across the research are:

• The condensed March model comparison preserves scores at four selected HSK levels but not a per-call reviewer or sample manifest. Its model versions later changed or disappeared.
• The April nine-route vocabulary screen used one hard HSK 1 cooking concept and one output per route. Several outputs also had structural or language failures.
• The April variance test covered two models, one HSK 1 school concept, and five runs per model. It measures noise in that setup, not every model's variance.
• The July A/B/C architecture screen had one story per arm, all at HSK 7 and all using the same plan and Kimi K2.6 draft route.
• The July prose screen held one HSK 7 mystery architecture fixed. Fable had one sample; most other routes had two. Some exploratory routes used proxies, and cost attribution was incomplete.
• The HSK 4 GPT-5.6 routing decision came from one head-to-head story. It remains a production choice under review, not a settled general result.

The benchmark will keep changing as models, routes, prompts, and editorial needs change. The stable way to read it is experiment by experiment, with the setup and limitation attached.

Related guides:

• How I Make Graded Chinese Stories
• How to Add Accurate Pinyin to Chinese Text
• Running Chinese Text-to-Speech on a GPU
• Extensive Reading in Chinese: The Research-Backed Method