The Shenzhen He Tao Academy, in collaboration with Harbin Institute of Technology (Shenzhen), the Shenzhen Institute of Big Data, and Huawei teams, recently announced the completion of full-parameter post-training for the 1.6 trillion-parameter model DeepSeek-V4-Pro. This was achieved on a thousand-card cluster (Shenzhen Smart City AI Computing Platform) powered by Huawei’s Ascend 910C chips, spanning over 1,500 steps with zero interruptions and zero errors. The participating teams stated this marks the first time a third-party organization globally has completed training exploration for a model of this parameter scale on a domestic computing platform, confirming that domestic AI chips can support world-class ultra-large parameter model training. In terms of key metrics, the Model Flops Utilization (MFU) exceeded 30%, and the efficiency of core training operators increased by 14%, both meeting industrial-grade operational standards.
For a long time, domestic computing power in the large model field has primarily handled inference and minor fine-tuning tasks. However, the computing scale and cross-chip communication volume required for full-parameter post-training far exceed those of inference scenarios — especially as DeepSeek-V4-Pro adopts a Mixture of Experts (MoE) architecture, where inter-expert communication during post-training is roughly tens of times that of ordinary models. The team overcame three core challenges: First, they designed a precise distributed deployment scheme, allocating the 1.6 trillion parameters like a “jigsaw puzzle” across the thousand-card cluster; second, they optimized the MoE scheduling strategy to eliminate load imbalance between experts and cross-card communication bottlenecks; third, they established a fully visible and controllable monitoring system to ensure stable operation over long training steps. Compared to using a ready-made model for inference, full-parameter post-training requires the model to “self-reflect and dynamically adjust parameters,” drastically increasing computation and communication volume. This project also included a talent cultivation mission, with 42 students deeply involved in the engineering practice throughout.