deterministic · real Vim · no judge model
Can your model
speak Vim?
Give a model a start buffer, a target buffer, and one instruction. It must return the keystrokes that get there—then we replay them in a real, headless Vim.
instruction
“Change the text inside the quotes to goodbye.”
start
message = "hello world"model keystrokes
: f"ci"goodbye<Esc>:wq<CR>PASSexact match · efficiency 1.00
01 / RESULTS
Single-turn leaderboard
Seed set · 30 tasks · exact byte comparison
Higher is better, except runtime.
| Rank | Model | Pass rate | Passed | Efficiency | Partial | P95 latency | Tokens / pass | Cost / pass | Runtime |
|---|---|---|---|---|---|---|---|---|---|
| Loading benchmark results… | |||||||||
Efficiency compares candidate keystroke length with the verified reference solution. Partial credit is a buffer similarity score on failures. P95 is the 95th-percentile provider-request latency. Tokens and cost are normalized by successful tasks; older runs without telemetry display “—”.
02 / WHY VIM?
A small editor test
with a bigger signal.
Vim is not the point. Reliable tool use is.
Vim turns editing into a compact language of actions: move, select, change, repeat, save. To solve a task, an LLM must translate intent into an exact sequence, track state, and stop with the right result. Those are the same habits a useful coding agent needs when it edits your files, calls tools, or carries out a multi-step instruction.
This makes vimbench a focused test of execution reliability, not a claim that Vim knowledge equals intelligence. Every answer is run, every final buffer is checked, and short correct solutions are rewarded. There is no judge model deciding whether an answer “looks right.”
Exactness
A high pass rate means the model more often turns a clear request into the exact requested state—fewer almost-right edits for you to repair.
Efficiency
Efficient solutions use fewer actions. For everyday agents, that can mean shorter traces, less opportunity for drift, and lower tool overhead.
Recovery signal
Partial credit shows how close failures get. It helps distinguish a near miss from a model that leaves the document badly mangled.
03 / METHOD
Small loop. Hard signal.
Every score comes from executable behavior,
not another model’s opinion.
Prompt
The model sees a start buffer, target buffer, natural-language instruction, and keystroke notation.
Replay
Its answer is translated to bytes and replayed in clean, headless Vim with shell access disabled.
Compare
The saved buffer is compared byte-for-byte with the target. Correct answers are scored for keystroke efficiency.
model output→keystroke bytes→vim -u NONE→exact diff
Read the full methodology
04 / RUN IT
Your model.
Your machine.
Real Vim.
Quick startPython 3 + Vim
$ git clone https://github.com/supercobra/vimbench.git
$ cd vimbench
$ python3 run.py --verify
$ python3 llm_runner.py --provider mock --max-tasks 5