Application of Reconfigurable OCS Technology for Pre-training Large Language Models

ZHU Chen; ZHOU Xu; WANG Peilong

doi:10.13756/j.gtxyj.2024.240049

您当前的位置：

首页 >

文章列表页 >

Application of Reconfigurable OCS Technology for Pre-training Large Language Models

更新时间：2024-10-08

- Application of Reconfigurable OCS Technology for Pre-training Large Language Models
- Study on Optical Communications Issue 5, Pages: 24004901-24004910(2024)
- 作者机构：
  
  百度在线网络技术有限公司　系统部，北京　100085
- 作者简介：
- 基金信息：
- DOI：10.13756/j.gtxyj.2024.240049
  CLC： TN929
- Published：10 October 2024，
  
  Received：01 March 2024，
  
  Revised：12 April 2024，
- 稿件说明：
移动端阅览
朱宸，周谞，王佩龙. 可重构OCS技术在大模型预训练中的应用[J].光通信研究，2024(5)：240049.

Zhu C, Zhou X, Wang P L. Application of Reconfigurable OCS Technology for Pre-training Large Language Models[J]. Study on Optical Communications, 2024(5): 240049.
朱宸，周谞，王佩龙. 可重构OCS技术在大模型预训练中的应用[J].光通信研究，2024(5)：240049. DOI： 10.13756/j.gtxyj.2024.240049.

Zhu C, Zhou X, Wang P L. Application of Reconfigurable OCS Technology for Pre-training Large Language Models[J]. Study on Optical Communications, 2024(5): 240049. DOI： 10.13756/j.gtxyj.2024.240049.

摘要

【目的】

相比于电子分组交换机（EPS），全光电路交换（OCS）在时延、功耗、成本和稳定性等各个方面都体现出了优势，文章通过分析大模型预训练中的并行切分策略、集合通信需求、流量模式和现今的网络架构，讨论了基于OCS在训练组网中的可行的应用方式，以在训练任务中充分利用OCS的优势。

【方法】

文章提出在故障快速恢复中采用多个小端口OCS进行网络设备冗余保护的机制，可在机顶（ToR）交换机故障时快速切换不中断训练任务。此外，文章还提出OCS只为数据并行（DP）服务，且仅在任务开始前进行配置。

【结果】

文章提出了多种可行的光电组网架构，以及在不同AllReduce算法下的具体配置，采用包括集合通信算法和架构设计联合优化的方式达到更优的带宽利用率。

【结论】

只要充分结合训练任务的流量模型，OCS可以很好地融入现有EPS网络架构，从成本、低功耗、低时延以及高稳定性等各方面对大模型预训练进行优化。

Abstract

【Objective】

Compared to Electronic Packet Switching (EPS)

Optical Circuit Switching (OCS) demonstrates advantages in latency

power consumption

cost

and stability. This study aims to explore feasible applications of OCS in the networking of training tasks by analyzing parallel partitioning strategies

collective communication requirements

traffic patterns

and current network architectures in large model pretraining

in order to fully leverage the benefits of OCS.

【Methods】

We propose a mechanism for network device redundancy protection using multiple small-port OCS devices

enabling rapid switching without interrupting training tasks in the event of Top-of-Rack (ToR) switch failures. Additionally

we advocate for the exclusive service of OCS to data parallelism

requiring configuration only at the start of the task.

【Results】

We present several feasible opto-electronic networking architectures and specific configurations under different AllReduce algorithms

including joint optimization of collective communication algorithms and architectural design to achieve optimal bandwidth.

【Conclusion】

By adequately integrating the traffic models of training tasks

OCS can seamlessly blend into existing EPS network architectures and optimize the large model pretraining from multiple perspectives

including cost

low power consumption

reduced latency

and enhanced stability.

关键词

全光电路交换可重构光电混合网络架构大模型预训练集合通信并行训练

Keywords

OCSreconfigurableopto-electro hybrid network architectlarge language models pre-trainingcollective communicationparallel training

references

Boser B E, Guyon I M, Vapnik V N. A Training Algorithm for Optimal Margin Classifiers[C]//Proceedings of the Fifth Annual Workshop on Computational Learning Theory. Pittsburgh, Pennsylvania, USA: ACM, 1992: 144-152.

Krizhevsky A, Sutskever I, Hinton G E. Im-ageNet Classification with Deep Convolutional Neural Networks[J]. Communications of the ACM, 2017, 60(6): 84-90.

LeCun Y, Bengio Y, Hinton G. Deep Learning[J]. Nature, 2015, 521: 436-444.

Cortes C, Vapnik V. Support-vector Networks[J]. Machine Learning, 1995, 20(3): 273-297.

Quinlan J R. Induction of Decision Trees[J]. Machine Learning, 1986, 1(1): 81-106.

Anderson J A, Rosenfeld E. Neurocomputing, Volume 1: Foundations of Research[M]. Cambridge, MA: The MIT Press, 1988.

Brown T B, Mann B, Ryder N, et al. Language Models are Few-shot Learners[DB/OL]. (2020-07-22) [2024-03-01]. http://arxiv.org/abs/2005.14165http://arxiv.org/abs/2005.14165.

Guo C, Wu H, Deng Z, et al. RDMA over Commodity Ethernet at Scale[C]//Proceedings of the 2016 ACM SIGCOMM Conference. Florianopolis, Brazil: ACM, 2016: 202-215.

Tang Y, Yuan T, Liu B, et al. Effective *-flow Schedule for Optical Circuit Switching based Data Center Networks: a Comprehensive Survey[J]. Computer Networks, 2021, 197: 108321.

Urata R, Liu H, Yasumura K, et al. Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale[DB/OL]. (2020-08-22) [2024-03-01]. https://arxiv.org/abs/2208.10041https://arxiv.org/abs/2208.10041.

Jouppi N P, Kurian G, Li S, et al. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings[DB/OL]. (2023-04-04) [2024-03-01]. https://arxiv.org/abs/2304.01433https://arxiv.org/abs/2304.01433.

Banerjee D, Mukherjee B. Wavelength-routed Optical Networks: Linear Formulation, Resource Budgeting Tradeoffs, and a Reconfiguration Study[J]. IEEE/ACM Transactions on Networking, 2002, 8(5): 598-607.

Chiu A L, Choudhury G, Clapp G, et al. Architectures and Protocols for Capacity Efficient, Highly Dynamic and Highly Resilient Core Networks[J]. Journal of Optical Communications and Networking, 2012, 4(1): 1-14.

Qiao C, Yoo M. Optical Burst Switching (OBS) - A New Paradigm for an Optical Internet[J]. Journal of High Speed Networks, 1999, 8(1): 69-84.

Narayanan D, Shoeybi M, Casper J, et al. Efficient Large-scale Language Model Training on GPU Clusters Using Megatron-LM[DB/OL]. (2021-08-23) [2024-03-01]. https://arxiv.org/abs/2104.04473https://arxiv.org/abs/2104.04473.

Harris J M, Lindquist R, Rhee J, et al. Optical Switching[M]. Boston, MA: Springer, 2008.

Huang Q. Optical Switching in Next Generation Data Centers[M]. Cham: Springer, 2018.

Frisken S, Baxter G, Abakoumov D, et al. Flexible and Grid-less Wavelength Selective Switch Using LCOS Technology[C]//Optical Fiber Communication Conference/National Fiber Optic Engineers Conference 2011. Los Angeles, California, USA: OSA, 2011: OTuM3.

Doerr C R. Silicon Photonic Integration in Telecommunications[J]. Frontiers in Optics, 2015, 3: 00037.

Furukawa H. Petabit-class Optical Networks and Switching Technologies [C]//OSA Advanced Photonics Congress 2021. Washington, DC, USA: OSA Technical Digest, 2021: NeW1C.

Hill K O, Malo B, Bilodeau F, et al. Bragg Gratings Fabricated in Monomode Photosensitive Optical Fiber by UV Exposure through a Phase Mask[J]. Applied Physics Letters, 1993, 62(10): 1035-1037.

Zheng X, Kaman V, Yuan S, et al. Three-dimensional MEMS Photonic Cross-connect Switch Design and Performance[J]. IEEE Journal of Selected Topics in Quantum Electronics, 2003, 9(2): 571-578.

Kewitsch A S. Large Scale, All-fiber Optical Cross-connect Switches for Automated Patch-panels[J]. Journal of Lightwave Technology, 2009, 27(15): 3107-3115.

Barua B, Nayim M H, Bhuiyan M A M. A 2×2 MEMS Optical Switch based on Piezoelectric Actuation[C]//2006 International Conference on Electrical and Computer Engineering. Dhaka, Bangladesh: IEEE, 2006: 52-55.

Sohma S, Watanabe T, Shibata T, et al. Compact and Low Power Consumption 16 /Spl Times/ 16 Optical Matrix Switch with Silica-based PLC Technology[C]//OFC/NFOEC Technical Digest Optical Fiber Communication Conference. Anaheim, CA, USA: IEEE, 2005: 193036.

Patronas G, Syrivelis D, Bakopoulos P, et al. Software-defined, Programmable L1 Dataplane: Demonstration of Fabric Hardware Resilience Using Optical Switches[C]//2023 Optical Fiber Communications Conference and Exhibition (OFC). San Diego, CA, USA: IEEE, 2023: 10116070.

Gibiansky A. Bringing HPC Techiques to Deep Learning[DB/OL]. (2017-02-21)[2024-03-01].https://andrew.gibiansky.com/blog/machine-learning/baidu-allreduce/https://andrew.gibiansky.com/blog/machine-learning/baidu-allreduce/.

Jeaugey S. Masively Scale Your Dep Leaning Traning with NCCL 2.4[DB/OL]. (2019-02-04)[2024-03-01].https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/.

Thakur R, Rabenseifner R, Gropp W. Optimization of Collective Communication Operations in MPICH[J]. The International Journal of High Performance Computing Applications, 2005, 19(1): 49-66.

Wang W, Ghobadi M, Shakeri K, et.al. Optimized Network Architectures for Large Language Model Training with Billions ofParameters[DB/OL]. (2023-10-21)[2024-03-01].https://arxiv.org/pdf2307.12169v2https://arxiv.org/pdf2307.12169v2.

Views

下载量

CSCD

CNKI被引量

Alert me when the article has been cited

提交

Tools

Publicity Resources

Microwave Photonic Filter Combined with Stimulated Brillouin Scattering

Related Author

GUO Li-jun

TAN Zhong-wei

Related Institution

Key Laboratory of All Optical Network and Advanced Telecommunication Network, Ministry of Education, Institute of Lightwave Technology, Beijng Jiaotong University

Postal code：430205
Technical support is provided by Beijing Founder electronics co., LTD 鄂ICP备2022010197 鄂公网安备 42011202002092号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：103142 Visits Today：66

⁰