Enterprise AI
Quasar Shows Early Benchmark Gains in Bittensor SN24 Decentralized Pretraining
The SN24 subnet reports consistent lifts on standard evaluations after two weeks of decentralized pretraining with limited compute, providing data points on scalable model development via distributed resources.
Quasar is a long-context foundation model and decentralized evaluation subnet on Bittensor SN24.
Quasar has demonstrated early benchmark gains during its decentralized pretraining run on Bittensor. The gains appeared after two weeks of operation with limited compute resources. The project has shared details of these shifts through official channels.
What benchmark improvements has the Quasar subnet achieved so far?
The MMLU score improved by 0.61 points. The starting score was 68.40 percent. The ending score was 69.01 percent. This change is attributed to the training update from the project. The improvement is part of a series of positive shifts on other evaluations. The project has shared these results through its official communication channels. The changes are small but consistent across the board.
MMLU-Pro showed an improvement of 0.52 points. GPQA showed an improvement of 0.41 points. ARC Easy showed an improvement of 3.86 points. These changes occurred in the same two week period. The project has shared these results through its official communication channels. The training is still in its early stages according to the update.
Quasar is just getting started.QuasarModels
The current training run is only around 10 percent complete. The run has operated with relatively low compute up to this point. Additional compute support is expected to allow the training to scale further. The results are projected to keep improving as the run advances. The project has indicated that the benchmark movement is already clear despite the early stage.
| Benchmark | Initial Score | Current Score | Change |
|---|---|---|---|
| MMLU | 68.40% | 69.01% | +0.61 |
| MMLU-Pro | Not reported | Not reported | +0.52 |
| GPQA | Not reported | Not reported | +0.41 |
| ARC Easy | Not reported | Not reported | +3.86 |
How does the decentralized training system on the Quasar subnet operate?
The production training system coordinates independent miners for the pretraining tasks. The production training system coordinates validators for the pretraining tasks. The production training system coordinates a subnet-operated orchestrator for the pretraining tasks. This setup is described in the subnet documentation. The coordination enables real training work in a decentralized manner.
This coordination allows real training work to be assigned. This coordination allows real training work to be verified. This coordination allows real training work to be merged. This coordination allows real training work to be converted into subnet weights. The goal is to enable real training work in a decentralized manner. The system supports continuing pretraining of the Quasar-Preview model.
- Assign training work to independent miners
- Verify the work using validators
- Merge the results from the participants
- Convert the merged results into subnet weights
What is the design and hosting of the Quasar-Preview model?
Quasar-Preview is the first model in a planned series of Quasar models. The model was trained on more than 1 trillion tokens. The model was trained on less than 1.5 trillion tokens. The model is hosted on Hugging Face under the silx-ai account. The model is designed for research. The model is designed for distillation. The model is designed for SN24 training. The model is not yet the final Quasar model.
What scale does the Quasar pretraining effort target?
The goal is to complete more than 10 trillion additional pretraining tokens on Quasar Preview. The run is intended to be the largest decentralized model-training run that can be made practical. The run is intended to be the largest decentralized model-training run that can be made verifiable. The run is intended to be the largest decentralized model-training run that can be made repeatable. The effort focuses on continuing pretraining of the Quasar-Preview model through decentralized means.
What implications exist for enterprise AI strategies from this decentralized approach?
Enterprise decision makers may consider decentralized subnets for aspects of their AI implementation plans. The distributed nature of the training can support access to large scale pretraining without full ownership of compute resources. The verifiable process described in the subnet supports accountability for training outcomes. The early benchmark gains suggest that consistent improvements are possible even with limited initial resources.
Enterprises focused on data sovereignty may examine how decentralized systems handle model training data. The approach provides a mechanism for scaling training as more compute support becomes available. The consistent lifts on the evaluations indicate the potential for ongoing progress. The use of Bittensor for this purpose allows for a community driven model development process.
What reactions from AI industry stakeholders are expected regarding the Quasar results?
Stakeholders in the AI industry may observe the consistent lifts across multiple evaluations. The lifts appeared after a short period of two weeks. The project has noted that the training is still early in its progress. The expectation is that adding more compute will lead to further scaling of the training.
The results are projected to continue improving based on the current trajectory. The early results from the Quasar subnet indicate that decentralized pretraining can deliver measurable improvements even in the initial phases of a run. The movement occurred despite limited initial compute.
What are the anticipated next developments for the Quasar project?
The training run will continue with the addition of more compute support. The benchmark results are expected to keep improving as the run scales. Additional models in the Quasar series are planned as the research advances. The subnet will aim to achieve its target of more than 10 trillion additional tokens.
The overall effort will seek to maintain the practical, verifiable, and repeatable qualities of the decentralized training. The project aims to complete more than 10 trillion additional pretraining tokens on Quasar Preview through the largest decentralized model-training run we can make practical, verifiable, and repeatable.
Frequently asked
What benchmark saw the largest reported improvement in the initial Quasar training?
ARC Easy saw an improvement of 3.86 points after two weeks of training.
What is the target number of additional pretraining tokens for the Quasar project?
The goal is to complete more than 10 trillion additional pretraining tokens on Quasar Preview.