How Much Does the Prompt Matter When AI Corrects You?

TL;DR: Tell it to only correct actual errors (in Korean or English), and only ask once unless you know the correction is wrong.

In my last post, I created a benchmark that tests AI models on correcting errors of Korean learners. I used one short prompt for all of them on purpose, because that is what a learner likely types and is similar to the prompt used in the original KoLLA v2 paper:

Correct the Korean sentence. Reply with the corrected sentence only.

However, Kanana 2 significantly underperformed compared to the original paper: F0.5 of 0.366 in my benchmark vs 0.59 in the paper. There were a few justifications for this result: I used a highly quantized version and a different prompt (content and language).

First, I added more models to the benchmark and published the results for more experiments.

Review of the Test Setup

The dataset is the KoLLA v2, it contains sentences by Korean learners writing an essay. The essay is split up into sentences. Each sentence has annotations from two human correctors.

A model’s task is to produce the corrected version of the sentence. The result is then scored via the “edit graph”, i.e. which edits were necessary to come to this correction. This edit graph is compared to the graph produced by the human correctors.

The models are prompted via OpenRouter or via LM-Studio running on my computer.

A huge limitation of running the local models is my graphics card, an AMD 7900 XTX - the most VRAM AMD was willing to sell to consumers. It’s limiting in speed, execution and how many I have.

Vast.ai for new Local Models

To parallelize runs, get faster results, and try larger self-hostable models, I extended the benchmark runner to allow offloading to vast.ai. The script simply takes a limitation, e.g. VRAM >= 20GB, and then:

• rents the cheapest machine with a graphics card fulfilling the limitation
• sends over the benchmark code
• executes the benchmark for all models the current specification says
• uploads the result to an S3 bucket
• frees the machine

This worked surprisingly well and I decided to run a few more and larger models. Command A+ by Cohere, a giant 219B model. Unquantized versions of models I’ve tried before, especially Muse Glimmer and Kanana 2 30B. More niche models I simply ignored before, like Nemotron and Mi:dm by Korea’s KT.

The most surprising result for me: Muse is still on top and the heavily quantized versions can keep up with the unquantized version.

Kanana 2, however, improved massively when running the unquantized version: F0.5 of 0.536, reasonably close to the originally reported 0.59, so the difference may be explained by the prompt used.

Four Prompts

To get the answer to the prompt dependency, I created a new experiment: Running the same model with different prompts in different ways.

• Short: “Correct the following korean sentence. Only correct actual errors.” The original benchmark prompt.
• Short, real errors only: “Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.”
• Detailed brief: A paragraph explaining who wrote the sentences, what kinds of errors to fix, not to rewrite for style, and to return correct sentences unchanged.
• Written in Korean: 아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라. 단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라. Taken from the original KoLLA v2 paper.

To get a very fast picture of the effects, I ran it through the fastest models, hosted and self-hosted, and the top local contender Muse.

For every larger model, the original short prompt came in last, by a significant margin. GPT-5.6 Luna went from 55.8% with the base prompt to 69.2% with the Korean prompt.

The simple ‘real errors only’ addition had nearly the same effect. The reason for the improvements is what you might expect from the addition, but it is the case for all improved prompts: The models mostly stopped “fixing” things that weren’t wrong.

Given the large changes, it’s reasonable to expect Kanana 2 unquantized would perform similarly, getting to the 0.59 value.

Lastly, Ministral really does not like the Korean prompt.

Correcting the Correction

With a completely different architecture, Grammarly suggests iterating on corrections. To see if it works with LLMs as well, I take the model’s corrected sentence, send it back as a new sentence to correct. Repeat until the model stops changing it (or reaches an upper limit of 10 rounds to prevent infinite loops).

It made every model worse. The biggest drop came right on the second request: GPT-5.6 Luna went from 66.9% to 57.3%, and ended at 55.2%.

The problem becomes obvious if you count what happened to individual sentences:

The same over-”fixing” issue we fought with the prompt change. Once the model sees only its own corrected sentence, it doesn’t know what the learner originally meant. And it still really, really wants to correct something.

An important caveat: Every extra request was a fresh chat. Asking “are you sure?” in the same conversation, where the model still sees your original sentence, may behave differently. I did not test that.

Conclusion

Don’t overthink how to get corrections when using AI. The extended simple prompt:

Correct the following korean sentence. Only correct actual errors. Reply with only the corrected sentence.

Performed well. If you feel fancy, copy-paste the Korean request:

아래 한국어 문장에서 틀린 부분만 수정하고, 교정 문장 한 줄만 출력하라. 단, 문장이 이미 문법적으로 올바르면 원문을 그대로 출력하라.

Without any evidence, I would also recommend keeping more context. Were you translating something? Add the original task. Ask for an explanation of the errors you made.

I’ll keep updating the hosted models of the benchmark, and, when I have time, the self-hosted ones as well.

For now, I am quite busy with v0.18 of elephant, which will bring improvements to grammar detection and handling. I’ll write about it soon, with the announcement of the release.

David