Measuring model's ability to naturally recover from errors.
Emoji-Bench presents models with 100 tasks, each a logical expression to solve step-by-step. We inject an incorrect step into a pre-filled model response, then prompt the model to "Please continue." The better a model recovers from the error to reach the correct answer, the higher its score.
How well can a model recover from its error?
Every problem are emoji-based and procedurally generated. This ensures questions fall outside LLM's training data while keeping the answers verifiable.
Emojis act as neutral, out-of-distribution variables with no formal logic priors. This allows for fair comparison: the derivations themselves are within reach of any current frontier model, so the benchmark can isolate self-correction from raw capability.
Each system and expression are sampled from a difficulty knob, with a validator that ensures problems are solvable with an unique solution. Synthetic generation makes samples cheap, and because the logic is fully specified, every answer is verifiable by construction.
After giving a model a derivation task, we inject an incorrect step in the model’s prefilled response. We then only prompt “Please continue”, and score whether the model can naturally reach the correct answer despite the error.
The user message contains a procedurally generated emoji formal system, an expression to derive, and a strict step-by-step format.
The assistant transcript ends on a deliberately injected error. The model is told nothing is wrong.
The prompt is exactly Please continue. — no instruction to review or inspect previous steps.
The extracted final output must match ground truth, and not the wrong derivation implied by the prefilled error.
You are given the Sylk Structure with the following operation table:
| ⊕ | 🪈 | 🪵 | 🥟 |
|---|---|---|---|
| 🪈 | 🥟 | 🪵 | 🥟 |
| 🪵 | 🪈 | 🪵 | 🥟 |
| 🥟 | 🪈 | 🪈 | 🪵 |
Simplify the following expression:
((🪈 ⊕ 🥟) ⊕ (🪵 ⊕ (🥟 ⊕ 🪵)))
I’ll simplify step by step.
Please continue.
Model thinking
For feedback, questions, collaboration - or anything in between.