KoLLA v2 benchmark · Follow-up experiments

How much does the prompt matter?

The main benchmark uses one short instruction to ask each model to correct Korean sentences. I wanted to see how much that choice affects the results, and whether asking a model to correct its own answer helps. These two experiments use the same learner sentences.

Practical recommendations See the main benchmark
  1. +13.4 pts

    More specific instructions helped

    GPT-5.6 Luna scored 55.8% with the leaderboard prompt and 69.2% with its best prompt in this test.

  2. 171 : 8

    Repeated corrections made things worse

    Repeatedly sending GPT-5.6 Luna its own answer lost 171 fully correct answers and fixed 8 wrong ones.

  3. 33.4%

    The model still made a larger difference

    The best score from either 3B model was below even the lowest score from a larger model in these runs (51.8%).

Models
2 hosted, 3 open weights, 2 small local
Runs
49 complete, 14 stopped early
Dataset
KoLLA v2
Run on
September 14 and September 19, 2026

Experiment 1 · Comparing prompts

Does a more specific instruction help?

The main benchmark asks each model to “correct the Korean sentence.” For this experiment, I compared that short instruction with three alternatives: one that says to fix only actual errors, a more detailed brief, and an instruction written in Korean.

Each prompt was tested on the same 1,418 learner sentences, once in the message alongside the sentence and once as a system prompt, the separate instruction field used by an API. That makes eight runs per model. Some runs did not finish; their coverage is shown below.

  1. Short Leaderboard prompt

    Correct the Korean sentence. Reply with the corrected sentence only.

  2. Short, real errors only

    Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.

  3. Detailed brief

    You are correcting Korean sentences written by learners of Korean. Correct grammatical, lexical, spacing, spelling, and word-order errors while preserving the learner's intended meaning as closely as possible. Make only the changes necessary to produce a grammatically correct and natural Korean sentence. Do not rewrite the sentence merely to improve style. If the sentence is already acceptable, return it unchanged. Return only the corrected Korean sentence, with no explanation or additional text.

  4. Written in Korean

    아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라. 단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라.

    “Fix only the wrong parts of the Korean sentence below and output the corrected sentence on one line. If it is already grammatically correct, output the original as it is.”

Overall quality by prompt Each dot shows one prompt's score. The line connects the lowest and highest scores for that model. Higher is better.
  • Short
  • Short, real errors only
  • Detailed brief
  • Written in Korean

GPT-5.6 Luna Hosted service

+13.4 pts +13.7 pts best vs. Short

Gemini 3.5 Flash Lite Hosted service

+7.7 pts +7.2 pts best vs. Short

Gemma 4 26B A4B Open weights, rented GPU

+9.6 pts +9.7 pts best vs. Short

Muse Glimmer Open weights, rented GPU88–99% of sentences scored

+7.7 pts +6.3 pts best vs. Short

Kanana 2 3B Instruct 3B, on your computer

+6.2 pts +19.1 pts best vs. Short

Ministral 3 3B 3B, on your computer

+1.3 pts +1.2 pts best vs. Short

Show as a table
Model Short In the message Short As system prompt Short, real errors only In the message Short, real errors only As system prompt Detailed brief In the message Detailed brief As system prompt Written in Korean In the message Written in Korean As system prompt
GPT-5.6 Luna 55.8%54.4%66.3%66.3%64.0%62.3%69.2%68.1%
Gemini 3.5 Flash Lite 55.8%58.1%63.5%64.6%62.7%62.0%63.5%65.3%
Gemma 4 26B A4B 53.0%51.8%62.6%61.5%62.2%61.5%61.9%61.1%
Muse Glimmer 56.9%58.3%62.0%64.1%62.2%63.1%64.6%64.6%
Kanana 2 3B Instruct 27.2%12.5%32.7%18.4%33.4%22.3%30.9%31.6%
Ministral 3 3B 27.1%27.8%28.5%29.0%27.7%27.6%22.0%25.3%
Qwen3.8 27B partial: 114–521 of 1,418 62.7%57.2%61.5%64.4%63.7%64.2%64.2%63.8%

Muse Glimmer (88–99% of sentences scored): the missing sentences timed out, came back without an answer, or were never sent before the run stopped. The score covers the rest.

Qwen3.8 27B is only in the table: charts need every run to score at least 85% of the sentences.

The larger models benefited from clearer instructions

The short prompt from the leaderboard was the worst of the four for every larger model. With the instruction in the message, their best scores were +7.7 to +13.4 points higher. Each model did best with either the short “real errors only” prompt or the Korean prompt. The detailed brief did not improve on those simpler alternatives.

The Korean prompt was within 0.7 points of the best on each larger model. That makes it a useful starting point, though this does not isolate the effect of language: its wording differs from the English prompts too.

The two 3B models remained well behind

The smaller models also responded to the wording, but there was no clear best prompt for both. Across all prompts and placements, their highest score was 33.4%, compared with 51.8% for the lowest-scoring run of a larger model. For these two models, changing the prompt was not enough to make them competitive.

What changes when the prompt changes? For the larger models, clearer instructions mainly helped them leave correct sentences alone. The share of mistakes they caught changed less. Select a model to compare both measures, with the instruction in the message.

Correct sentences left alone among scored sentences both human annotators left unchanged

Short 60.6%
Short, real errors only 75.9%
Detailed brief 72.7%
Written in Korean 77.5%

Mistakes caught of the edits the human annotators made

Short 69.0%
Short, real errors only 72.5%
Detailed brief 73.6%
Written in Korean 73.4%

Three sentences the human annotators left unchanged, and how GPT-5.6 Luna responded

  • What the learner wrote 대학교에서 일이 어렵습니까?

    Short prompt 대학교에서 공부하는것이어렵습니까?

    Every other prompt left it unchanged

  • What the learner wrote 이 사람은 여자 친구입니다.

    Short prompt 이 사람은 제여자 친구입니다.

    Every other prompt left it unchanged

  • What the learner wrote 음식을 만든 후에 드라마를 보고 음식을 먹습니다.

    Short prompt 음식을 만든 후에 드라마를 보면서먹습니다.

    Every other prompt left it unchanged

Does it matter where the instruction goes? The change in overall quality when the instruction moves from the message into the system prompt. Values to the right mean a higher score as a system prompt; values to the left mean a lower one.

GPT-5.6 Luna Hosted service

Short −1.4
Short, real errors only +0.0
Detailed brief −1.6
Written in Korean −1.1

Gemini 3.5 Flash Lite Hosted service

Short +2.4
Short, real errors only +1.1
Detailed brief −0.6
Written in Korean +1.9

Gemma 4 26B A4B Open weights, rented GPU

Short −1.2
Short, real errors only −1.1
Detailed brief −0.7
Written in Korean −0.8

Muse Glimmer Open weights, rented GPU88–99% of sentences scored

Short +1.5
Short, real errors only +2.1
Detailed brief +0.9
Written in Korean +0.0

Kanana 2 3B Instruct 3B, on your computer

Short −14.7
Short, real errors only −14.3
Detailed brief −11.1
Written in Korean +0.7

Ministral 3 3B 3B, on your computer

Short +0.7
Short, real errors only +0.5
Detailed brief −0.2
Written in Korean +3.2

For the larger models, the difference was at most 2.4 points, with no consistent advantage for either placement. Comparable runs varied by up to 1.8 points, so I would be cautious about reading much into these small differences. Kanana 2 3B Instruct was more sensitive to placement, losing up to 14.7 points as a system prompt.

Experiment 2 · Repeated corrections

Does another correction improve the answer?

After a model corrects a sentence, it seems reasonable to ask whether it missed anything. I tested one version of that: sending its answer back as a new sentence to correct. Each request started a fresh chat, so the model only saw the previous answer, not the learner's original sentence.

I repeated this until the answer came back unchanged or the model had received 10 requests in total. Every request used the “Short, real errors only” prompt. This tests repeated correction of the output; it does not tell us whether a follow-up question in an ongoing conversation would help.

  1. 1Send the learner's sentence
  2. 2Model returns a correction
  3. 3Send that correction to a fresh chat
  4. 4Stop when the answer comes back unchanged, or after 10 requests
Overall quality after each extra request Each point uses the latest answer for every sentence. Once a model returned a sentence unchanged, that answer was kept for the remaining points.
  • GPT-5.6 Luna 66.9% → 55.2%
  • Gemini 3.5 Flash Lite 63.1% → 55.9%
  • Gemma 4 26B A4B 62.8% → 51.4%
  • Kanana 2 3B Instruct 28.4% → 22.1%
  • Ministral 3 3B 28.6% → 22.7%

Requests per sentence

Every model shown here finished with a lower score than it started with. The largest drop for the larger models came on the second request: GPT-5.6 Luna lost 9.6 points at that step. The two 3B models continued to lose quality over later requests.

Left out for scoring fewer than 85% of the sentences: Muse Glimmer (375 of 1,418 sentences), Qwen3.8 27B (86 of 1,418 sentences).

How often did another correction help? The red bars count fully correct first answers that were no longer fully correct at the end. The green bars count the reverse: answers that became fully correct after more requests.

GPT-5.6 Luna

171 8 21 : 1

Gemini 3.5 Flash Lite

114 16 7 : 1

Gemma 4 26B A4B

234 7 33 : 1

Kanana 2 3B Instruct

71 23 3 : 1

Ministral 3 3B

68 25 3 : 1

37% → 4% Share of fully correct answers before and after revision, among the 486 sentences that GPT-5.6 Luna changed after its first answer.

31% → 6% Share of fully correct answers before and after revision, among the 402 sentences that Gemini 3.5 Flash Lite changed after its first answer.

37% → 3% Share of fully correct answers before and after revision, among the 666 sentences that Gemma 4 26B A4B changed after its first answer.

How long did the models keep making changes? Share of the 1,418 sentences. The larger models used 2.1 to 2.7 requests on average. For both 3B models, most sentences reached the 10-request limit.
  • 1
  • 2
  • 3–4
  • 5–9
  • 10 (limit)

GPT-5.6 Luna

2.2× requests

Gemini 3.5 Flash Lite

2.1× requests

Gemma 4 26B A4B

2.7× requests

Kanana 2 3B Instruct

8.2× requests

Ministral 3 3B

8.2× requests

Each request means another model call. The two 3B models averaged more than 8 requests per sentence and were still changing 975 (Kanana 2 3B Instruct) and 993 (Ministral 3 3B) sentences at the limit. More calls did not produce better corrections here.

How an answer changes over repeated requests

What the learner wrote 시드니에서 서울까지 걸립었습니다.

Human corrections 시드니에서 서울까지 걸렸습니다. / 시드니에서 서울까지 아홉 시간 걸렸습니다.

  1. 1

    시드니에서 서울까지 걸렸습니다.

    ✓ Matches a human correction
  2. 2

    시드니에서 서울까지 가는데시간이얼마나걸렸습니다.

    ✕ No longer matches
  3. 3

    시드니에서 서울까지 가는 데 시간이 얼마나 걸렸습니까?

    ✕ No longer matches
  4. 4

    시드니에서 서울까지 가는 데 시간이 얼마나 걸렸습니까?

    Unchanged, loop stops

Practical recommendations

What I would use

For checking a sentence, I would start with either of these two prompts. The Korean version had the highest score on GPT-5.6 Luna and Muse Glimmer and was within 0.7 points of the best on every other larger model shown above. The short English version was at most 2.9 points behind it. That makes it a reasonable choice if you prefer to read the instruction in English.

Paste your sentence after the prompt. Both ask for a corrected sentence without an explanation, which is what this benchmark measures. I did not test how well they explain a correction or help you understand it.

  • Ask the model to correct actual errors.
  • The Korean prompt also says to leave correct sentences unchanged.
  • I would avoid repeatedly submitting the answer for another correction; that reduced quality in this test.
  • For the two 3B models tested here, a different prompt did not close the gap to the larger models.

Written in Korean

아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라. 단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라.

Short, real errors only

Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.

How the test works

The sentences and scoring

I used the same 1,418 learner sentences from KoLLA v2 as in the main benchmark, with the same human corrections and M2 scorer. “Overall quality” is the F0.5 score: it balances how many mistakes the model catches with how often its changes are useful, giving more weight to avoiding unnecessary changes.

Seven models, with some incomplete runs

The selection covers two hosted models, three open-weight models on a rented GPU and two 3B models on a laptop. The charts require at least 85% of sentences to have a scored answer in each run. This leaves Qwen3.8 27B and Muse Glimmer out of some comparisons; their coverage is listed alongside the results.

Each condition ran once. Comparable runs differed by up to 1.8 points, so I would read small gaps as ties. There is no statistical significance analysis here.

What the repeated-correction test covers

Each extra request started a fresh chat with the previous answer as its input. The model could not compare that answer with the learner's original sentence. Asking a follow-up question in the same conversation may behave differently; I did not test that here.

A valid rewrite can still lose points

The scorer checks changes against the human corrections. A different rewrite can be valid Korean and still lose points if those changes are not in the references. For a learner, unnecessary rewrites can make it harder to see which part was actually wrong, but a lower score does not mean every change was bad Korean.

Main benchmark and method Benchmark code KoLLA v2 publication

Understanding a correction

A corrected sentence is useful, but it can leave you wondering why something changed. Elephant's sentence analysis explains corrections word by word to help you work through your own Korean sentences.

Try sentence analysis Back to the main benchmark