基于强化学习的光网络自适应高效率RWA算法（特邀）

刘兆洋; 潘必韬

doi:10.13756/j.gtxyj.2024.240024

您当前的位置：

首页 >

文章列表页 >

基于强化学习的光网络自适应高效率RWA算法（特邀）

专题：数据中心内光交换 | 更新时间：2024-10-08

- 基于强化学习的光网络自适应高效率RWA算法（特邀）
- A Reinforcement Learning based Adaptive and Efficient RWA in All Optical Networks
- 光通信研究 2024年第5期页码：24002401-24002408
- 作者机构：
  
  北京邮电大学　信息与通信工程学院，北京　100876
- 作者简介：
  
  刘兆洋（2000－），女，河北沧州人。硕士，主要研究方向为光通信系统与网络。
  潘必韬，副研究员。E-mail：bitao.pan@bupt.edu.cn
- 基金信息：
  
  国家自然科学基金资助项目(6220010519)
- DOI：10.13756/j.gtxyj.2024.240024
  中图分类号： TN929
- 纸质出版日期：2024-10-10，
  
  收稿日期：2024-02-03，
  
  修回日期：2024-03-01，
- 稿件说明：
移动端阅览
刘兆洋，潘必韬. 基于强化学习的光网络自适应高效率RWA算法[J]. 光通信研究，2024(5)：240024.

Liu Z Y, Pan B T. A Reinforcement Learning based Adaptive and Efficient RWA in All Optical Networks[J]. Study on Optical Communications, 2024(5)：240024.
刘兆洋，潘必韬. 基于强化学习的光网络自适应高效率RWA算法[J]. 光通信研究，2024(5)：240024. DOI： 10.13756/j.gtxyj.2024.240024.

Liu Z Y, Pan B T. A Reinforcement Learning based Adaptive and Efficient RWA in All Optical Networks[J]. Study on Optical Communications, 2024(5)：240024. DOI： 10.13756/j.gtxyj.2024.240024.

摘要

【目的】

在光网络中，目前基于深度强化学习（DRL）的路由与波长分配（RWA）算法大多依赖于K最短路径（KSP）路由算法提前计算备选路径，这种方法缺乏灵活性和动态性，当网络拓扑变化时需要为所有节点对重新进行KSP计算。为了解决上述问题，文章提出了基于DRL的自适应高效率（ADE）-RWA框架。

【方法】

文章提出的ADE-RWA框架的关键点及创新之处在于训练过程中，DRL的智能体在动作选择时从当前节点出发选择最优单步连接，而不是预先计算的完整路径，从而可以动态实时依据网络状态的变化做出路由策略的改变，以适应链路故障等网络拓扑改变的情况。ADE-RWA框架的另一关键点在于DRL训练过程不仅训练其决策神经网络，并且每次成功分配一条路径时，都将其以路由表（LUT）的形式存储。在DRL训练收敛后，网络路径的LUT信息也将完整。此时使用LUT直接查找可用路径，可有效降低计算成本，提高RWA算法效率。此外，LUT查找与DRL训练能够实现动态切换。文章所提算法通过动态感知网络状态的变化，当网络故障引起拓扑变化时，切换到DRL阶段继续进行训练，保证了LUT的动态更新。

【结果】

实验结果表明，与KSP-首次拟合（FF）和路由调制和频谱分配的深度学习（DeepRMSA）算法相比，文章所提ADE-RWA框架的阻塞率分别降低了36%和30%；且当发生链路故障时，ADE-RWA框架可以迅速适应网络拓扑的变化。

【结论】

文章提出了基于DRL的ADE-RWA框架，实现了动态光网络拓扑下的自适应RWA，并降低了计算成本。

Abstract

【Objective】

Recent research efforts on Routing and Wavelength Assignment (RWA) for all optical networks are focused on Deep Reinforcement Learning (DRL) based algorithms. The DRL based RWA algorithms are mostly rely on the K Shortest Paths (KSP) routing to calculate candidate paths in advance

hence the DRL agent can choose possible actions from the precomputed paths. These KSP based models lack of flexibility and dynamicity

since they need to re-calculate the KSP for all the node pairs once the topology changes occur. To address this issue

this paper proposes an Adaptive and Efficient(ADE)-RWA algorithm based on DRL.

【Methods】

The key points and innovations of the ADE-RWA lie in that during the training process

the DRL agent takes actions in a step-by-step way instead of selecting from the precomputed K complete paths. Therefore

the routing strategies are dynamically adjustable in training even under the case of topology changes. It is because that the actions are open for the agent to take without concerning the limitations of the K fixed paths. Moreover

the ADE-RWA records the successfully assigned routes during the training in a LookUp Table (LUT). The algorithm turns to LUT checking for finding the available routes once the DRL training is converged

since at that time the LUT has acquired enough information for the RWA from the DRL training. The LUT based routing can effectively reduce the computational costs and improve the efficiency of RWA. In addition

the DRL training phase and LUT routing phase are real-time switchable. The algorithm turns to the DRL training phase when a link failure caused topology change occurs

and turns back to LUT checking when the model training is converged again.

【Results】

Experimental results show that compared with KSP-First Fit(FF)and Deep Reinforcement Learning Framework for Routing

Modulation and Spectrum Assignment (DeepRMSA)

the blocking probability of ADE-RWA is reduced by 36% and 30% respectively. When a link failure occurs

the algorithm can quickly adapt to the changes in network topology.

【Conclusion】

The proposed DRL based RWA framework ADE-RWA can achieve adaptive routing and wavelength allocation under dynamic network conditions with low computational cost.

关键词

波长路由光网络路由与波长分配深度强化学习数字孪生

Keywords

wavelength-routed optical networkRWADRLdigital twin

references

Ji Y F, Wang H X, Cui J B, et al. All-optical Signal Processing Technologies in Flexible Optical Networks[J]. Photonic Network Communications, 2019, 38(1): 14-36.

Chu X W, Li B. Dynamic Routing and Wavelength Assignment in the Presence of Wavelength Conversion for All-optical Networks[J]. IEEE/ACM Transactions on Networking, 2005, 13(3): 704-715.

Zhu Z Q, Liu X H, Wang Y X, et al. Impairment- and Splitting-aware Cloud-ready Multicast Provisioning in Elastic Optical Networks[J]. IEEE/ACM Transactions on Networking, 2017, 25(2): 1220-1234.

Zhu Z, Lu W, Zhang L, et al. Dynamic Service Provisioning in Elastic Optical Networks with Hybrid Single-/Multi-path Routing[J]. Journal of Lightwave Technology, 2013, 31(1): 15-22.

Vincent R J, Ives D J, Savory S J. Scalable Capacity Estimation for Nonlinear Elastic All-optical Core Networks[J]. Journal of Lightwave Technology, 2019, 37(21): 5380-5391.

Xu L F, Huang Y C, Xue Y, et al. Deep Reinforcement Learning-based Routing and Spectrum Assignment of EONs by Exploiting GCN and RNN for Feature Extraction[J]. Journal of Lightwave Technology, 2022, 40(15): 4945-4955.

Zhang Y J, Xin J J, Li X, et al. Overview on Routing and Resource Allocation based Machine Learning in Optical Networks[J]. Optical Fiber Technology, 2020, 60: 102355.

Suárez-Varela J, Mestres A, Yu J L, et al. Routing in Optical Transport Networks with Deep Reinforcement Learning[J]. Journal of Optical Communications and Networking, 2019, 11(11): 547-558.

周刚, 吴树霖, 张江龙, 等. 基于拓扑链路识别的光网络流量数据合成算法[J]. 光通信研究, 2022(1): 31-36.

Zhou G, Wu S L, Zhang J L, et al. Traffic Data Synthesis Algorithm for Optical Network based on Topological Link Recognition[J]. Study on Optical Communications, 2022(1): 31-36.

Chen X L, Li B J, Proietti R, et al. DeepRMSA: a Deep Reinforcement Learning Framework for Routing, Modulation and Spectrum Assignment in Elastic Optical Networks[J]. Journal of Lightwave Technology, 2019, 37(16): 4155-4163.

Quang H T, Houidi O, Errea-Moreno J, et al. MAGC-RSA: Multi-agent Graph Convolutional Reinforcement Learning for Distributed Routing and Spectrum Assignment in Elastic Optical Networks[C]//2022 European Conference on Optical Communication (ECOC). Basel, Switzerland. IEEE: 2022: 9979335.

Shimoda M, Tanaka T. Mask RSA: End-to-end Reinforcement Learning-based Routing and Spectrum Assignment in Elastic Optical Networks[C]//2021 European Conference on Optical Communication (ECOC). Bordeaux, France: IEEE, 2021: 9606169.

Ericsson. AI: Enhancing Customer Experience in a Complex 5G World[EB/OL]. (2021-07-30) [2024-02-01]. https://www.ericsson.com/49ced5/assets/local/reports-papers/mobility-report/documents/2021/ai_enhancing-customer-experience.pdfhttps://www.ericsson.com/49ced5/assets/local/reports-papers/mobility-report/documents/2021/ai_enhancing-customer-experience.pdf.

Dai Y Y, Zhang K, Maharjan S, et al. Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks[J]. IEEE Transactions on Industrial Informatics, 2021, 17(7): 4968-4977.

Janz C, You Y, Hemmati M, et al. Digital Twin for the Optical Network: Key Technologies and Enabled Automation Applications[C]//NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium. Budapest, Hungary: IEEE, 2022: 9789844.

Nevin J W, Nallaperuma S, Shevchenko N A, et al. Techniques for Applying Reinforcement Learning to Routing and Wavelength Assignment Problems in Optical Fiber Communication Networks[J]. Journal of Optical Communications and Networking, 2022, 14(9): 733-748.

Mnih V, Badia P A, Mirza M, et al. Asynchronous Methods for Deep Reinforcement Learning[EB/OL]. (2016-06-16) [2024-02-01]. https://arxiv.org/abs/1602.01783https://arxiv.org/abs/1602.01783.

Kingma D P, Ba J. Adam: a Method for Stochastic Optimization[EB/OL]. (2017-01-30) [2024-02-01]. http://arxiv.org/abs/1412.6980http://arxiv.org/abs/1412.6980.

Ebrahimzadeh A, Rahbar A G, Alizadeh B. Request Differentiation in Dynamic Light-path Establishment for WDM Routed all Optical Networks of Data Centers[J]. Optical Fiber Technology, 2015, 21: 73-80.

浏览量

下载量

CSCD

CNKI被引量

文章被引用时，请邮件提醒。

提交

工具集

关联资源

信息化前景和网络演进探讨

一种基于改进遗传算法的波长路由算法

一种基于改进蚁群算法的光网络动态RWA机制

波长路由光网络中RWA算法的设计分析

基于T-MPLS的路由与波长分配问题研究