Omar Badran
I train open-weight language models to reason with my own reinforcement-learning trainer. One person, one rented GPU at a time, every number published beside Meta's own model.
The trainer
The only reward is a program's verdict on the final answer: right or wrong. No judge model, no human labels, nothing that scores the reasoning.
- Credit where the outcome was decided. GRPO scores whole answers and gives every token the same credit. My trainer grows a tree of the model's own answers, so it can see where an answer went right or wrong, and learns from the difference between branches.
- Hard problems still teach. When every first answer is wrong, it keeps searching the model's own text for a right ending instead of throwing the problem away.
- It remembers problems. Across rounds a mastered problem is retired, and the next rounds prefer problems that still teach.
- From a raw base. No supervised step needed: no example answer is ever shown.
Tree-shaped RL trainers have been published (TreeRL, TreePO, 2025). I do not claim the idea of a tree; I claim these results and their cost. The weights are open; the trainer's code is private.
Results
Llama-3.1-8B-Aleph-2 beside Meta's Llama 3.1 8B Instruct: one greedy answer a problem, no examples in the prompt, the same problems and grader for both.
| test | my warmup | Aleph-2 | Meta's Instruct |
|---|---|---|---|
| tool calls (JSON) | – | 99.5 | 82.4 |
| extraction (JSON) | – | 99.0 | 90.0 |
| GSM8K, grade-school math | 52.3 | 76.0 | 85.8 |
| MBPP, Python tasks | 27.4 | 42.0 | 54.0 |
| HumanEval, Python functions | 35.4 | 40.2 | 70.1 |
Tool calls and extraction are tests made for this project, from the same programs as its training tasks. The full nine-test table, including where it falls behind, is on the model page. The run behind Aleph-2 cost 26.61 USD.
- From the raw Llama 3.1 8B base, no warmup: GSM8K 67.2 held out after 84 rounds, about an hour of one GPU, 5.66 USD.
- On Llama 3.2 3B, adding code to the mix lifted MBPP by 14.6 and HumanEval by 26.8 points with math kept.
Next
A fair test of my trainer against stock GRPO at the same compute, then a long run from the raw base to a full table beside Meta's Instruct. If you can sponsor GPU time, or you are hiring for RL post-training, write to me.