Mechanical
Line deletion, swaps, joins, copies, case changes, and basic movement.
open protocol · executable scoring · reproducible tasks
How vimbench
measures skill.
vimbench tests whether a language model can turn a precise editing request into a correct Vim action sequence. The result is executed in real Vim and judged against an exact target buffer.
The benchmark is deliberately narrow. It does not measure every dimension of an LLM. It isolates a useful one: reliable translation from intent to stateful tool use.
01 / PROTOCOL
From request
to exact diff.
The model never edits the target directly. It emits keystrokes, and the harness does the rest.
-
1
Present the task
The prompt contains a natural-language instruction, the starting buffer, the desired target buffer, and the supported keystroke notation. Few-shot examples establish the response format.
-
2
Parse the answer
The runner extracts the candidate sequence, accepting plain text or fenced code. Named tokens such as
<Esc>,<CR>, and<C-V>are converted into their byte values. -
3
Replay in real Vim
The harness writes the start buffer to a temporary file and feeds the bytes to headless Vim. A safety suffix exits and saves even if the model forgets, so the resulting state can always be scored.
-
4
Compare and score
The saved file is compared byte-for-byte with the target. Exact matches pass; failures receive a similarity score; correct answers also receive a keystroke-efficiency score.
Execution pathsingle turn
model response
│
▼
notation_to_bytes()
│
▼
vim -u NONE -N -n -s keys buf.txt
│
▼
final buffer == target buffer02 / TASKS
Verified before
models see them.
Each seed task has a known reference sequence that must produce the exact target through the same harness.
Instruction
“Change the text inside the quotes to goodbye.”
Start buffer
message = "hello world"Target buffer
message = "goodbye"Reference
f"ci"goodbye<Esc>:wq<CR>Vim grammar
Motions, text objects, macros, visual block operations, and composed commands.
Ex commands
Substitution, global operations, ranges, sorting, and pattern-based edits.
Composed edits
Multi-part transformations such as CSV changes, extraction, and cleanup.
Fresh-task track. Programmatic templates can generate additional parameterized tasks. A generated task is kept only when its reference sequence replays to an exact match, reducing dependence on a fixed, potentially memorized set.
03 / SCORING
Three numbers.
Different questions.
Read pass rate first. Efficiency and partial credit explain what happened around it.
Pass rate
exact matches / tasks
The primary reliability measure. A pass requires the final buffer to match the target byte-for-byte, including whitespace and newlines.
Efficiency
min(1, reference keys / model keys)
Calculated for passed tasks. It rewards concise action sequences without granting extra credit for beating the reference.
Partial credit
buffer similarity, 0…1
A diagnostic score for the final buffer. It distinguishes a near miss from a destructive failure, but does not turn a failure into a pass.
P95 latency
95th percentile request time
Shows tail latency: 95% of model requests finished this quickly or faster. It better reflects occasional slow waits than an average alone. Total wall-clock runtime is reported separately.
OPERATIONAL METRICS
Tokens and cost belong beside the score.
When an API reports usage, vimbench should record input, output, cached, and reasoning tokens separately, plus the billed cost. These numbers explain how expensive a model was to reach its accuracy.
total tokens / passed tasks
Tokens per successful task is the most useful comparison: it does not reward a cheap model for failing quickly. Cost per successful task is the corresponding real-world measure.
Compare with care: providers use different tokenizers, cache rules, and reasoning-token reporting. Missing hidden tokens must be shown as unavailable—not treated as zero.
04 / INTERPRETATION
What it means
when choosing.
The leaderboard is evidence about exact editing behavior—not a universal ranking of model quality.
- Following precise transformation instructions
- Producing executable tool syntax
- Tracking state through multi-action edits
- Finishing with an exact requested result
- Doing the above with relatively few actions
- Factual accuracy and research quality
- Long-form writing or conversation
- Image, audio, or multimodal understanding
- Repository-scale software architecture
- Price, privacy, latency, and context limits
A practical way to use the table
- Start with pass rate. If you want an agent to make dependable edits, exact success is the most relevant column.
- Use efficiency as a tie-breaker. Among similarly accurate models, concise tool use can reduce drift and overhead.
- Inspect partial credit. When pass rates are close, it reveals whether misses tend to be recoverable or destructive.
- Then apply your constraints. Compare price, speed, privacy, availability, context window, and results on a sample of your own work.
05 / CONTROLS
Less environment.
Less ambiguity.
The harness removes common sources of variation so model behavior dominates the result.
Clean Vim
-u NONE -N -n disables user configuration and swapfiles while using nocompatible mode.
No shell shortcut
set shell=/bin/false blocks shell commands. The benchmark measures Vim editing, not escape hatches.
Fixed timeout
Each task has a 10-second execution limit so runaway macros and hung sessions become consistent failures.
Exact fixtures
Start and target buffers are explicit strings, and every reference answer is replay-verified through real Vim.
Shared task set
Models in a leaderboard run receive the same selected tasks, preserving head-to-head comparability.
Open implementation
The tasks, harness, scorer, runner, and result format are public and can be audited or run locally.
06 / LIMITS
Know what is
outside the frame.
Narrow domain
Vim competence is a proxy for compact, stateful tool use. It is not a complete coding, reasoning, or intelligence evaluation.
Reference is not proven optimal
Efficiency compares against a verified reference solution, not a mathematically guaranteed shortest possible sequence.
Environment scope
The task language intentionally targets a portable subset of Vim. Plugin knowledge and advanced full-Vim features are outside the current benchmark.
Sample size and contamination
The 30-task seed set is small enough to interpret but cannot cover every edit. Generated tasks help, but do not eliminate all benchmark familiarity risk.
Provider settings matter
Prompt format, model version, reasoning effort, temperature, and API behavior can affect results. Compare runs with matching settings.
REPRODUCE IT
Trust the run,
not the claim.
Verify every reference task on your machine, inspect the source, then run the same suite against your model.
Reference verificationPython 3 + Vim
$ git clone https://github.com/supercobra/vimbench.git
$ cd vimbench
$ python3 run.py --verify
# 30/30 verified