Which AI Should You Use to Check Your Korean?
What I learned from testing AI models on Korean learner sentences.
TL;DR: Probably DeepSeek web or a local Muse Glimmer, and add an “only correct real errors” part when prompting.
Recently, I released v0.17 of Elephant.
It contained many quality improvements for vocabulary handling, sentence analysis, and more.
However, how do I decide what an improvement is? What even is an improvement? I wanted - no, I needed something systematic to make decisions.
Benchmarking
By now, I assume most people have seen one benchmark or another of AI models. There are a lot.
However, a short recap: A benchmark is a dataset and an execution framework for that dataset on whatever you want to test. Frameworks also include the scoring, condensing the result down to numbers. In most cases even a single number.
A standardised test from school, but for computers.
They may write a whole essay as a response to “What is 0.1 + 0.2?”, but in the end they only got a C-.
PapersWithCode used to be a nice collection of standardised benchmarks for all kinds of tasks. It was my favorite resource for natural language processing (NLP) and its many, many subtasks. (You can get a glance of what it was like here.)
My interest is very, very specific though: Korean, especially for learners of the language.
KoLLA v2
There are specific datasets for Korean. Shout-out to Songys and their AwesomeKorean_Data repository listing many datasets. Some of them are unfortunately difficult to use.
Luckily, I stumbled upon KoLLA v2, a dataset of Korean learner writing. It is a recent release focusing on exactly my niche:
This paper introduces KoLLA v2.0, the first Korean learner corpus to incorporate both multi-reference GEC annotations and rubric-based essay scoring.
It split 100 essays into sentences and provided two human corrections for each. Exactly what I needed for at least one of the questions I wanted to answer.
The Question(s)
Which AI model is best at grading Korean? (And at what cost?)
That’s a tricky question. With or without reasoning enabled for the models? How detailed should the prompt be? Should it try to instruct the model against overcorrecting? What language should the prompt use?
I wanted it to be close to what users of Elephant likely would do. Therefore, I used a simple instruction for the models and permitted anything the default settings would allow:
Correct the Korean sentence. Reply with the corrected sentence only.
I ran all sentences through OpenRouter (which has its own caveats) and LM Studio. The result was then put through the same scoring mechanism that the paper used - although it was reimplemented in Rust.
You can find all the code on GitHub.
Result overview
I used the same metric for the evaluation as the original paper: F0.5. F0.5 is a precision-favoring metric. Precision can be thought of as how often the edits the model made are correct. Recall, the other metric going into F0.5, is how many of the errors were actually fixed by the model.
The best model, with a strong lead, is Grok 4.6. It is followed by DeepSeek V4.1 Flash - which is possibly accessible in their chat, though this is difficult to confirm.
To compare with local models, I also tortured my poor 7900 XTX for many hours by running the benchmark against LM Studio with various models. The really small and fast models (Ministral 3 3B and Gemma 4 E4B) unfortunately underperformed my hopes. The larger ones in the 30B range performed much better than expected (with one exception): Meta’s Muse Glimmer could keep up well with paid models, falling right in the middle of the pack.
However, why did Kanana 2 underperform so badly?
Limitations
The original paper got a much higher result for Kakao Corp.’s Kanana 2: An F0.5 score of 0.59, which would beat Muse Glimmer.
This and other things are huge indicators of the limitations of a single benchmark. The original paper used a fully Korean prompt, even translating it into slightly different instructions, and all the models I ran locally are heavily quantized. All of these relate to what I wanted to test: An average learner is unlikely to ask the LLM in Korean to get corrections. A model run locally is unlikely to be unquantized. I believe my 24 GB of VRAM is already on the high end for most. So, while the settings make sense, they have to be considered when thinking about the benchmark results.
Lastly, Grammarly, at least somewhat knowledgeable in error correction, argued not only against sequence-to-sequence for error correction but also for something important besides that: Iterate on corrections. My test did not do that, but I’ll plan to add it as a comparison mode for future experiments.
Where to Next?
I am currently working on building a dataset for grammar analysis. The goal is not only to improve grammar analysis, but also to show that it improved.
As new models are released, I also want to keep the benchmark somewhat up to date and add some features for comparison, like an iterative approach for a few selected models or a Korean prompt for Kanana 2 (give it a fairer chance!).
That is a lot of work though.
David