We have published our second research report. It describes Treepai g1, a 518M-parameter language model we trained from scratch on 10 billion tokens in 87 hours on one consumer GPU.
What we found
On five core benchmarks g1 averages 49.6%, up from 38.9% for g0. It is ahead of GPT-2 medium (44.9%) and Pythia-410M (47.0%), the established open models of its size. Against a Pythia-410M checkpoint at a similar 10.5 billion training tokens, it leads on every core benchmark (49.6% against 37.6% on average).
It is not the best model of its size. Qwen2.5-0.5B (54.4%) and SmolLM2-360M (58.3%), trained with hundreds of times more data, are ahead. g1’s training text is narrow, and in a same-engine speed test it was the slowest of the four models we compared.
A first job for the model
A base model of this size is fine-tuned for one job, so we tested one: sorting customer-support messages into categories and handing off the ones a support desk should not answer. Fine-tuned g1 is level with Qwen2.5-0.5B fine-tuned the same way, with 95.2% and 95.6% of categories right. Its gap on general text did not carry into the task.
Open instruction-tuned models of its size, given the same task by prompt alone, got 42.5% of categories right at best. That lead comes from fine-tuning, not from g1’s base model. It is one task, with one run per model.
What’s next
g1 is a research model, and its weights are not released. Next we want to know whether broader training text closes the general gaps, whether a smaller or reshaped model can do the same job at lower serving cost, and whether the fine-tuning result holds on a second task.
Read the report. Questions: research@treepai.com.