Huawei Researchers Develop Pangu-Σ: A Large Language Model With Sparse Architecture And 1.085 Trillion Parameters

Huawei Researchers develop Pangu-Σ: A Large Language Model with Sparse Architecture and 1.085 trillion parameters. They say the performance of language models scales up with compute budget and model parameters. The researchers say the effectiveness of big language models depends on how to scale the system performance with a restricted computing budget.