Treepai g1: A 518M-Parameter Language Model Trained From Scratch on 10 Billion Tokens With One GPU
Jose Franco · Treepai, Inc. · October 2026
Treepai g1 is our second model. It carries the training recipe of g0 to a model 4.7 times larger with four times the data. The report describes how it was built, how it compares with open models of its size, how it did on a first fine-tuned task, and where it falls short.
Read the report (PDF, 24 pages)
You may download this report and share it unchanged, with credit to Treepai, Inc. All other rights are reserved.
Main results
- Trained from scratch on 10 billion tokens in 87 hours on one consumer GPU. It averages 49.6% on five core benchmarks, up from 38.9% for g0, and is ahead of GPT-2 medium (44.9%) and Pythia-410M (47.0%), the established open models of its size.
- Against a Pythia-410M checkpoint at a similar 10.5 billion training tokens, it leads on every core benchmark (49.6% against 37.6% on average).
- Fine-tuned for a first task, sorting customer-support messages, it is level with Qwen2.5-0.5B fine-tuned the same way: 95.2% and 95.6% of categories right. Open instruction-tuned models of its size, given the task by prompt alone, got 42.5% at best.
- A 7% slice of filtered public-domain fiction in the training text raised story-level reading (LAMBADA) from 21.8% for g0 to 42.3%.
- The first attempt at the run became unstable. We measured the cause, a missing weight decay on the matrices trained with the Muon optimizer, and a small weight decay fixed it.
Where it falls short
- Qwen2.5-0.5B (54.4%) and SmolLM2-360M (58.3%), the strongest open models of its size, are ahead on the core benchmarks. Both were trained with hundreds of times more data.
- The training text is narrow. On Wikipedia text g1 predicts less well than every finished model of its size.
- In a same-engine speed test (llama.cpp, 8-bit) it was the slowest of the four models compared. 8-bit quantization cost nothing measurable.
- The model was trained once, and the first 13% of training had no weight decay on the Muon matrices, so it is not a single-recipe run.
- The lead over prompted models on the sorting task comes from fine-tuning, not from g1’s base model. Fine-tuning was tested on one task, with one run per model, on tests drawn mostly from the datasets used for training.
- g1 is a base model with no instruction tuning or safety training. It is not suitable for any user-facing use, and its weights are not released.
How we measured
- One evaluation harness for every model. The reference models were scored again with it, not copied from papers.
- The benchmark list, the harness and the comparison models were fixed before g1 was trained, and g1 was scored once, on its final checkpoint.
- Before training, we removed every training document that shared a 13-word sequence with an item of any of the nine benchmarks.
- We compare g1 with open models of similar size, 355M to 494M parameters. Smaller models appear only as a labelled reference group.
- For the sorting task, both models were fine-tuned on the same data with the same settings, and the pass marks were written down before training.
The full method is in Sections 4 to 6 and 9 of the report, and its limits are in Section 10.
Cite as
Franco, J. (2026). Treepai g1: A 518M-Parameter Language Model Trained From Scratch on 10 Billion Tokens With One GPU. Treepai, Inc. https://treepai.com/research/treepai-g1/
@techreport{franco2026treepaig1,
title = {Treepai g1: A 518M-Parameter Language Model
Trained From Scratch on 10 Billion Tokens
With One GPU},
author = {Franco, Jose},
institution = {Treepai, Inc.},
year = {2026},
month = oct,
url = {https://treepai.com/research/treepai-g1/}
}
The model’s intended use, limits and known risks are on its model card.
Questions about our research: research@treepai.com.