Frontier Models
AMD Instella-MoE-16B-A3B-Think Tops Fully Open Models on AIME25 at 73.40
The 16B parameter Mixture-of-Experts model, trained entirely from scratch on AMD Instinct GPUs, releases every stage of its pipeline and posts the highest average score among fully open systems.
Instella-MoE-16B-A3B-Think is a decoder-only Mixture-of-Experts language model with 16 billion total parameters and 2.8 billion active parameters per token.
AMD released the complete set of weights, configuration files, data mixtures, and inference code for the model family under ResearchRAIL for weights and MIT for training code. The release covers every checkpoint from the initial pre-training run through the final RL Think stage.
What architecture defines the Instella-MoE model family?
The decoder-only design uses a Mixture-of-Experts structure with two shared experts and six routed experts selected from a pool of 64. This configuration keeps active parameters at 2.8 billion per token while the total parameter count reaches 16 billion. The architecture supports efficient scaling during both training and inference on AMD hardware.
Training occurred entirely on AMD Instinct MI300X and MI325X GPUs through the ROCm software stack. AMD developed and applied the Primus and Miles frameworks to manage the end-to-end process from data ingestion to final alignment.
What training stages were executed and released?
The pipeline began with pre-training on 7.1 trillion tokens. Subsequent stages included mid-training to refine capabilities, long-context extension reaching 64K tokens, supervised fine-tuning, direct preference optimization, and a final RL Think stage focused on reasoning.
- Pre-training on 7.1 trillion tokens
- Mid-training refinement
- Long-context extension to 64K tokens
- Supervised fine-tuning
- Direct preference optimization
- RL Think stage for reasoning improvement
Each stage produced separate model checkpoints that AMD made available. Researchers can therefore reproduce or extend any portion of the training sequence using the released code and data mixtures.
Which benchmarks show the strongest results for Instella-MoE-16B-A3B-Think?
The Think variant posted the highest average score of 73.22 across evaluated post-training benchmarks among all fully open models. It outperformed OLMo3-7B-Think, which recorded 71.97 on the same aggregate metric.
Specific leadership appeared on AIME25 at 73.40 and on LiveCodeBench at 54.30. The model also recorded 82.50 on AGIEval. These results position the release ahead of other fully open systems on the reported metrics.
| Model | Average Score | AIME25 | LiveCodeBench | AGIEval |
|---|---|---|---|---|
| Instella-MoE-16B-A3B-Think | 73.22 | 73.40 | 54.30 | 82.50 |
| OLMo3-7B-Think | 71.97 | Not reported | Not reported | Not reported |
How does the open release affect market and stakeholder dynamics?
The complete artifact release allows academic and commercial teams to inspect, modify, and build upon every training decision. This transparency contrasts with many frontier models that withhold intermediate checkpoints and data details.
Hardware vendors gain visibility into performance on AMD Instinct accelerators. Software developers can optimize inference paths using the released code under the MIT license for the training components.
What reactions accompanied the Instella-MoE announcement?
AMD is excited to introduce Instella-MoE, a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters.AMD, Company announcement
The announcement emphasized the end-to-end training on AMD GPUs and the decision to release every stage of the pipeline. Observers noted the rarity of full multi-stage artifact availability for a model of this scale.
What developments are expected next from the Instella series?
AMD indicated continued work on the Instella line with potential expansions in model size and capability. The open licensing framework supports community contributions that could accelerate follow-on releases.
Further long-context and reasoning improvements remain possible given the released RL stage as a starting point. Stakeholders can now experiment with custom data mixtures on the provided base models.
Frequently asked
What licenses cover the Instella-MoE weights and code?
Weights are released under ResearchRAIL while training code uses the MIT license.
Which hardware was used for the entire training run?
Training ran end-to-end on AMD Instinct MI300X and MI325X GPUs with the ROCm stack.
Sources
- AMD ROCm Blogs — Instella-MoE-16B-A3B-Think achieves the highest average score of 73.22 among all fully open models on post-training benchmarks and leads on AGIEval at 82.50 and AIME25 at 73.40.
- AMD-AGI — Instella-MoE is a state-of-the-art fully open Mixture-of-Experts language model trained end-to-end from pre-training to RL on AMD Instinct MI300X and MI325X GPUs using the Primus framework.
- Hugging Face — The Instella-MoE model checkpoint release spans every stage of the model training pipeline, including pre-training, mid-training, long-context extension, SFT, DPO and RL.